Optimize test run time with autoresearch - #357
Open
jhugman wants to merge 6 commits into
Open
Conversation
jhugman
changed the title
jhugman
force-pushed
the
jhugman/testing/optimize
branch
3 times, most recently
from
March 20, 2026 21:59
e111470 to
b975d0b
Compare
…TMPDIR Replace Metro bundler with esbuild for TypeScript test bundles. Remove cargo_metadata dependency and derive target directory and fixture paths directly from CARGO_TARGET_TMPDIR. Add shared caching and file-sync utilities to lib.rs.
jhugman
force-pushed
the
jhugman/testing/optimize
branch
from
March 21, 2026 18:48
b975d0b to
7d60149
Compare
…iled headers - Cache codegen output (keyed on cdylib hash + ubrn binary mtime) - Run C++ and TypeScript compilation in parallel via std::thread::scope - Cache cmake configure step (skip when inputs unchanged) - Add precompiled headers for jsi/jsi.h and jsi/jsi-inl.h - Use -O0 for test C++ extension builds
- Share a single CARGO_TARGET_DIR across all wasm fixture builds - Cache codegen output (same pattern as JSI) - Limit cargo --jobs to max(1, num_cpus/3) to avoid CPU oversubscription
Replace cargo test with cargo nextest run across all CI jobs. Nextest runs each test in its own process with better parallelism and output.
jhugman
force-pushed
the
jhugman/testing/optimize
branch
from
March 21, 2026 18:50
7d60149 to
75f173c
Compare
jhugman
requested review from
Johennes and
zzorba
and removed request for
Johennes
March 23, 2026 13:18
zzorba
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixture test cold starts dropped from ~24min to ~20min (JSI) and ~37min to ~18min (WASM). Warm runs take 1-2min. Changing a template and re-running
cargo testnow works without manual steps.How
The test infrastructure gained three things: caching, parallelism, and correct change detection.
Caching. A three-layer content-hash cache sits between source changes and rebuilds. The codegen layer hashes the cdylib and ubrn binary mtime — if neither changed, binding generation is skipped. A write-if-changed sync preserves file mtimes so downstream tools (ninja, cargo, tsc) see no change. A build-level stamp hashes the generated output and skips compilation when content is identical. Each layer short-circuits independently.
Parallelism. JSI C++ and TypeScript compilation now run concurrently. WASM fixture builds share a single cargo target directory so dependencies compile once. Cargo's
--jobsflag is capped atnum_cpus/3to prevent thrashing when nextest runs multiple builds.Change detection.
ensure_ubrn_binary()always invokescargo build(a 1-2s no-op when nothing changed). This replaced a stale-binary guard that silently skipped rebuilds after template edits.What changed, file by file
ubrn_fixture_testing/src/lib.rs— Caching utilities (content hashing, cache stamps, write-if-changed sync), shared target dir management, always-rebuild ubrn binary.ubrn_fixture_testing/src/jsi.rs— Codegen caching, parallel C++/TS compilation with separate cache stamps, cmake configure caching.ubrn_fixture_testing/src/wasm.rs— Codegen caching, sharedCARGO_TARGET_DIR,--jobstuning.ubrn_fixture_testing/src/typescript.rs— Esbuild replaces Metro for bundling.ubrn_fixture_testing/src/metadata.rs— Path derivation fromCARGO_TARGET_TMPDIRreplacescargo metadata.cpp/hermes-rust-extension/CMakeLists.txt— Precompiled headers for JSI,-O0for test glue.cpp/test-harness/timers.js.inc— Steady clock for timer deadlines.Verification
Every optimization was checked three ways: a warm run (caches hold), a cache-bust (modifying a generated file triggers the right rebuild), and a full dev cycle (edit a template, rebuild the codegen binary, re-run tests — the whole chain invalidates and rebuilds correctly).