You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As a team member evaluating RFC 0001 (tracked in #93 and docs/rfcs/0001-native-desktop-rewrite.md), I want a benchmark comparison of the Rust audio-sync binary against the original AudioSyncer.js implementation, so that the spike's go/no-go recommendation is backed by real speed data, not just a correctness/ergonomics verdict.
Background
RFC 0001 proposes rewriting the editing pipeline in Rust, with rendering throughput as the primary motivating bottleneck (RFC §Context #1). The AudioSyncer spike (#93) was scoped to validate the agent-driven Rust dev loop and cross-hardware output parity — #100 (cross-machine verification and retro) explicitly deferred performance benchmarking out of its scope:
Performance benchmarking — this spike is about correctness and developer ergonomics only
That leaves a gap: RFC 0001's core argument is that a compiled-language core is faster, but the spike as scoped never measures whether the ported algorithm is actually faster in practice, or by how much. This issue fills that gap. It reuses the already-ported, already-verified algorithm (#96, #97, #98) and CLI (#99) plus the committed fixtures and JS baseline (#94) — no new algorithm code, just timing harnesses on both sides.
RFC 0001's whole premise is that hardware consistency across the team's actual machines is a first-class concern (§Context #2, §Verification approach: Mac M2, Mac M3, Windows/NVIDIA) — a single-machine speed number would leave open the question of whether the speedup holds, shrinks, or reverses on a different contributor's hardware. So this benchmark, like #100's correctness pass, runs on every contributor's device, not just one, and reports per-device numbers rather than a single global figure.
The result feeds directly into spike/RETRO.md (#100): a concrete, per-device speed delta is evidence for or against the RFC's "materially faster" goal, independent of the correctness/ergonomics verdict #100 already covers.
Acceptance criteria
Happy path
Given the committed fixture WAVs (spike/audio-sync/fixtures/video-audio.wav, audio-track.wav) When the Rust binary's core computation (compute_cross_correlation → find_best_lag → validate_peak, excluding process startup and WAV I/O) is timed over N=100 runs Thenspike/audio-sync/BENCHMARK.md records mean/median/p95 wall-clock time in milliseconds
Given the same fixture pair, decoded to the same sample data When the equivalent JS computation path in scripts/sync/AudioSyncer.js (the in-process correlation + lag + SNR logic, not the ffmpeg/ffprobe subprocess extraction steps) is timed over N=100 runs using Node's perf_hooks Thenspike/audio-sync/BENCHMARK.md records the same mean/median/p95 statistics for JS, using the same fixture data and iteration count as the Rust run
Given both harnesses exist and run cleanly When each contributor with access to a target machine (Mac M2, Mac M3, Windows/NVIDIA — same three machines as RFC 0001 §Verification approach and #100) runs both harnesses on their own device Then they post a comment on this issue with their raw mean/median/p95 numbers for both implementations, their machine spec (CPU/GPU, OS, RAM), and the resulting speedup ratio for that device
Given at least one result has been posted for each of the three target machines Whenspike/audio-sync/BENCHMARK.md is written Then it contains one row per device in a results table (device spec → Rust mean/median/p95 → JS mean/median/p95 → speedup ratio) plus a one-line summary noting whether the speedup is consistent across devices or varies meaningfully, and explicitly scopes every claim to this one algorithm — not the full pipeline or full RFC
Error path / edge case
Given a benchmark is run on a machine under variable load (other processes competing for CPU) When results are recorded Then the contributor's comment and the corresponding BENCHMARK.md row note the machine spec and state results are indicative, not a controlled-lab measurement — no false precision
Given cold-start/process-spawn overhead differs structurally between a compiled binary and a Node process When the benchmark is designed Then it separates "pure algorithm time" (the comparison that matters for the RFC's rendering-bottleneck argument) from "process startup time" (report both, but do not conflate them into a single misleading number) — on every device, not just one
Given one contributor cannot access one of the three target machines (e.g. no Windows/NVIDIA hardware available) When results are collected ThenBENCHMARK.md records which device(s) are missing data rather than silently omitting the row or fabricating a number — an explicit gap, not a silent one
Out of scope
Benchmarking any other pipeline stage (rendering, transcription, diarization) — this issue is scoped to the one already-ported algorithm
Statistical rigor beyond mean/median/p95 over repeated runs per device (no formal statistical significance testing across devices) — this is a spike, not a performance research paper
Fixing or optimizing either implementation based on results — this issue only measures and reports
Benchmarking on any machine beyond the three RFC 0001 target machines (Mac M2, Mac M3, Windows/NVIDIA) — a broader device matrix is a follow-up if the go/no-go decision needs it
Technical context
Rust side:spike/audio-sync/src/lib.rs already exposes compute_cross_correlation, find_best_lag, validate_peak (ported in #96, #97, #98). Add a benches/ entry (or a --bench flag on the existing CLI, whichever is less code) that loads the fixture WAVs once via load_wav_samples and then times only the three algorithm calls in a loop, excluding file I/O from the timed region.
JS side:scripts/sync/AudioSyncer.js (711 lines) contains computeCrossCorrelation (L145–186), findBestLag (L188–216), validatePeak (L218–235) — these are the exact functions #96/#97/#98 ported. Load the same fixture WAVs (wavefile package, already a repo dependency) once, then time only these three in-process functions in a loop — do not include the ffmpeg/ffprobe subprocess spawns (spawnProcess, getMediaDuration, getAudioChannels) in the timed region, since the Rust side has no equivalent subprocess step for this benchmark.
Fixture reuse:spike/audio-sync/fixtures/video-audio.wav, audio-track.wav, baseline.json (committed in #94) — same inputs on every device for a fair comparison.
Target machines (same as RFC 0001 §Verification approach and #100): Mac M2, Mac M3, Windows with NVIDIA GPU — whichever contributors already have access to these for #100's cross-machine verification pass are the natural people to also run this benchmark, since the machines are already set up with the toolchain.
Output:spike/audio-sync/BENCHMARK.md — new file, throwaway like the rest of spike/.
Implementation details
Add a timing harness to the Rust crate (either a benches/cross_correlation.rs using std::time::Instant in a simple loop, or a --bench CLI flag on the existing binary — pick whichever avoids adding a new dependency like criterion, since this is throwaway spike code).
Add a standalone Node script (e.g. spike/audio-sync/bench.js, not wired into npm run scripts) that imports the three functions from AudioSyncer.js and times them the same way, over the same fixture data and iteration count.
Post the harnesses in this issue (or as a small PR) so each contributor with access to a target machine can pull the branch and run both locally — no need to physically hand off hardware.
Each contributor runs both harnesses on their own device and comments on this issue with: device spec (CPU/GPU, OS, RAM), Rust mean/median/p95, JS mean/median/p95, and the device's speedup ratio.
Once all three target machines have a result comment, write spike/audio-sync/BENCHMARK.md: methodology, one row per device with its numbers, and a summary noting whether the speedup is consistent or diverges across hardware — with an explicit scope caveat.
N/A — this issue produces a measurement artifact, not new production logic; no unit tests beyond what already exists for the ported functions
Hard constraints
No changes to the already-ported algorithm logic (compute_cross_correlation, find_best_lag, validate_peak) — timing must wrap existing, already-verified code, not alter it
Results must be collected from all three target machines (Mac M2, Mac M3, Windows/NVIDIA) before BENCHMARK.md is finalized — a single-device number is not sufficient given RFC 0001's cross-hardware-consistency premise; a genuinely inaccessible device is recorded as a documented gap, not silently skipped
BENCHMARK.md must state each device's spec and iteration count — a bare ratio with no methodology is not acceptable
Depends on #99 (CLI binary + fixture round-trip test must be working before it can be wrapped in a timing loop) and #94 (fixture WAVs + JS baseline).
Naturally pairs with #100's cross-machine verification pass — the same contributors accessing the same three machines for correctness verification can run this benchmark in the same session, though the two issues don't block each other.
Feeds into #100's spike/RETRO.md go/no-go recommendation, but does not block or get blocked by #100 — either can land first; whichever lands second adds the cross-reference.
Independent of all existing refactor phase branches (refactor/p0-*, refactor/p1-*, etc.).
User story
As a team member evaluating RFC 0001 (tracked in #93 and docs/rfcs/0001-native-desktop-rewrite.md), I want a benchmark comparison of the Rust
audio-syncbinary against the originalAudioSyncer.jsimplementation, so that the spike's go/no-go recommendation is backed by real speed data, not just a correctness/ergonomics verdict.Background
RFC 0001 proposes rewriting the editing pipeline in Rust, with rendering throughput as the primary motivating bottleneck (RFC §Context #1). The AudioSyncer spike (#93) was scoped to validate the agent-driven Rust dev loop and cross-hardware output parity — #100 (cross-machine verification and retro) explicitly deferred performance benchmarking out of its scope:
That leaves a gap: RFC 0001's core argument is that a compiled-language core is faster, but the spike as scoped never measures whether the ported algorithm is actually faster in practice, or by how much. This issue fills that gap. It reuses the already-ported, already-verified algorithm (#96, #97, #98) and CLI (#99) plus the committed fixtures and JS baseline (#94) — no new algorithm code, just timing harnesses on both sides.
RFC 0001's whole premise is that hardware consistency across the team's actual machines is a first-class concern (§Context #2, §Verification approach: Mac M2, Mac M3, Windows/NVIDIA) — a single-machine speed number would leave open the question of whether the speedup holds, shrinks, or reverses on a different contributor's hardware. So this benchmark, like #100's correctness pass, runs on every contributor's device, not just one, and reports per-device numbers rather than a single global figure.
The result feeds directly into
spike/RETRO.md(#100): a concrete, per-device speed delta is evidence for or against the RFC's "materially faster" goal, independent of the correctness/ergonomics verdict #100 already covers.Acceptance criteria
Happy path
Given the committed fixture WAVs (
spike/audio-sync/fixtures/video-audio.wav,audio-track.wav)When the Rust binary's core computation (
compute_cross_correlation→find_best_lag→validate_peak, excluding process startup and WAV I/O) is timed over N=100 runsThen
spike/audio-sync/BENCHMARK.mdrecords mean/median/p95 wall-clock time in millisecondsGiven the same fixture pair, decoded to the same sample data
When the equivalent JS computation path in
scripts/sync/AudioSyncer.js(the in-process correlation + lag + SNR logic, not theffmpeg/ffprobesubprocess extraction steps) is timed over N=100 runs using Node'sperf_hooksThen
spike/audio-sync/BENCHMARK.mdrecords the same mean/median/p95 statistics for JS, using the same fixture data and iteration count as the Rust runGiven both harnesses exist and run cleanly
When each contributor with access to a target machine (Mac M2, Mac M3, Windows/NVIDIA — same three machines as RFC 0001 §Verification approach and #100) runs both harnesses on their own device
Then they post a comment on this issue with their raw mean/median/p95 numbers for both implementations, their machine spec (CPU/GPU, OS, RAM), and the resulting speedup ratio for that device
Given at least one result has been posted for each of the three target machines
When
spike/audio-sync/BENCHMARK.mdis writtenThen it contains one row per device in a results table (device spec → Rust mean/median/p95 → JS mean/median/p95 → speedup ratio) plus a one-line summary noting whether the speedup is consistent across devices or varies meaningfully, and explicitly scopes every claim to this one algorithm — not the full pipeline or full RFC
Error path / edge case
Given a benchmark is run on a machine under variable load (other processes competing for CPU)
When results are recorded
Then the contributor's comment and the corresponding
BENCHMARK.mdrow note the machine spec and state results are indicative, not a controlled-lab measurement — no false precisionGiven cold-start/process-spawn overhead differs structurally between a compiled binary and a Node process
When the benchmark is designed
Then it separates "pure algorithm time" (the comparison that matters for the RFC's rendering-bottleneck argument) from "process startup time" (report both, but do not conflate them into a single misleading number) — on every device, not just one
Given one contributor cannot access one of the three target machines (e.g. no Windows/NVIDIA hardware available)
When results are collected
Then
BENCHMARK.mdrecords which device(s) are missing data rather than silently omitting the row or fabricating a number — an explicit gap, not a silent oneOut of scope
Technical context
Rust side:
spike/audio-sync/src/lib.rsalready exposescompute_cross_correlation,find_best_lag,validate_peak(ported in #96, #97, #98). Add abenches/entry (or a--benchflag on the existing CLI, whichever is less code) that loads the fixture WAVs once viaload_wav_samplesand then times only the three algorithm calls in a loop, excluding file I/O from the timed region.JS side:
scripts/sync/AudioSyncer.js(711 lines) containscomputeCrossCorrelation(L145–186),findBestLag(L188–216),validatePeak(L218–235) — these are the exact functions #96/#97/#98 ported. Load the same fixture WAVs (wavefilepackage, already a repo dependency) once, then time only these three in-process functions in a loop — do not include theffmpeg/ffprobesubprocess spawns (spawnProcess,getMediaDuration,getAudioChannels) in the timed region, since the Rust side has no equivalent subprocess step for this benchmark.Fixture reuse:
spike/audio-sync/fixtures/video-audio.wav,audio-track.wav,baseline.json(committed in #94) — same inputs on every device for a fair comparison.Target machines (same as RFC 0001 §Verification approach and #100): Mac M2, Mac M3, Windows with NVIDIA GPU — whichever contributors already have access to these for #100's cross-machine verification pass are the natural people to also run this benchmark, since the machines are already set up with the toolchain.
Output:
spike/audio-sync/BENCHMARK.md— new file, throwaway like the rest ofspike/.Implementation details
benches/cross_correlation.rsusingstd::time::Instantin a simple loop, or a--benchCLI flag on the existing binary — pick whichever avoids adding a new dependency likecriterion, since this is throwaway spike code).spike/audio-sync/bench.js, not wired intonpm runscripts) that imports the three functions fromAudioSyncer.jsand times them the same way, over the same fixture data and iteration count.spike/audio-sync/BENCHMARK.md: methodology, one row per device with its numbers, and a summary noting whether the speedup is consistent or diverges across hardware — with an explicit scope caveat.spike/RETRO.mdonce spike(rust/verify): cross-machine verification and retro note #100 writes it (or note in this issue's closing comment that spike(rust/verify): cross-machine verification and retro note #100 should reference it) — this issue does not modifyspike/RETRO.mddirectly if spike(rust/verify): cross-machine verification and retro note #100 is done first; whichever issue lands second adds the cross-reference.Additional test scenarios
Hard constraints
compute_cross_correlation,find_best_lag,validate_peak) — timing must wrap existing, already-verified code, not alter itspike/audio-sync/Cargo.tomlunless required purely for benchmarking (preferstd::time::Instantover pulling incriterion, to keep the crate minimal per spike(rust/scaffold): create spike/audio-sync Cargo binary crate #95's constraints)BENCHMARK.mdis finalized — a single-device number is not sufficient given RFC 0001's cross-hardware-consistency premise; a genuinely inaccessible device is recorded as a documented gap, not silently skippedBENCHMARK.mdmust state each device's spec and iteration count — a bare ratio with no methodology is not acceptablespike/(same constraint as spike(rust): port AudioSyncer FFT correlation to Rust to validate agent-driven dev loop #93 and spike(rust/verify): cross-machine verification and retro note #100)Dependency issues
Depends on #99 (CLI binary + fixture round-trip test must be working before it can be wrapped in a timing loop) and #94 (fixture WAVs + JS baseline).
Naturally pairs with #100's cross-machine verification pass — the same contributors accessing the same three machines for correctness verification can run this benchmark in the same session, though the two issues don't block each other.
Feeds into #100's
spike/RETRO.mdgo/no-go recommendation, but does not block or get blocked by #100 — either can land first; whichever lands second adds the cross-reference.Independent of all existing refactor phase branches (
refactor/p0-*,refactor/p1-*, etc.).