spike(rust/decode): FFmpeg vs gstreamer-rs binding decision (#108)

[natashaannn]

Summary

Implements the issue #108 spike: a minimal Rust harness at spike/rust-decode-spike/ comparing ffmpeg-the-third vs gstreamer-rs as the decode/encode binding for the RFC 0001 native render engine.

Mac M3 results are real and verified in this branch (not just scaffolded) — see FINDINGS.md for the full evidence per criterion. Headlines:

• ffmpeg-the-third: pixel-exact frame-accurate seek+decode on both software and VideoToolbox hardware paths. No high-level hwaccel API exists in the crate — required ~40 lines of hand-written unsafe FFI (documented as a working reference for Epic 1).
• gstreamer-rs: hardware selection is just naming vtdec_hw instead of avdec_h264 (zero unsafe code), and it does negotiate/preroll successfully — but its pixel output is outside tolerance (confirmed not an off-by-one-frame bug by diffing against neighboring frames), likely a GL-readback colorimetry issue, plus a GStreamer-GL-WARNING about needing an NSApplication run loop that a render-engine binary won't have.
• Criterion Feat/video editor #5 (licensing) turned out to apply to both candidates, not just gstreamer-rs as originally scoped: this machine's Homebrew FFmpeg is --enable-gpl, so gstreamer-rs's software fallback (avdec_h264, via gst-libav) is GPL-contaminated, while vtdec_hw (Apple system frameworks only) is clean. Flagged as its own prerequisite issue for Epic 1 regardless of binding choice.

Recommendation: ffmpeg-the-third, contingent on the Windows/NVIDIA row below. Full rationale in FINDINGS.md.

Also corrected issue #108's spikes/ (plural) path to spike/ (singular) to match the established convention from the audio-sync spike (#93–#99) — standalone Cargo.toml per spike, no root-level workspace.

What's left for reviewers

Mac M2 and Windows+NVIDIA rows in FINDINGS.md are open — @Saloni @victoria, whichever of you has that hardware, please run the commands in spike/rust-decode-spike/README.md and paste your output into FINDINGS.md (or a comment here). The Windows candidates also need a nvh264dec/NVDEC code path added — TODOs are left in both main.rs files marking exactly where.

Test plan

• npm test — 20/20 suites pass (this change doesn't touch JS/TS)
• All 5 implementation-guide status checks pass (docs/implementation-guides/issue-108-rust-decode-spike.md)
• cargo test passes for both candidate crates (PPM/pixel-compare unit tests)
• Both candidates verified end-to-end against committed fixtures on Apple M3 (software + hardware paths)
• Mac M2 verification (open — needs contributor with that hardware)
• Windows + NVIDIA verification (open — needs contributor with that hardware, plus the NVDEC code path)

🤖 Generated with Claude Code

[natashaannn]

Ready-to-paste Claude Code prompts for the remaining hardware rows

Copy the block matching your machine into a Claude Code session in this repo. Each one is
self-contained — it explains what's already done and what's expected back.

Mac M2

This repo has a spike branch `spike/issue-108-decode-binding` (PR #109, issue #108) comparing
`ffmpeg-the-third` vs `gstreamer-rs` for hardware-accelerated video decode. Real Mac M3 results
are already recorded in `spike/rust-decode-spike/FINDINGS.md`. I'm on a Mac M2 — fill in the Mac
M2 rows in that file with real results from this machine, not scaffolding.

1. Checkout `spike/issue-108-decode-binding` and pull latest.
2. Read `spike/rust-decode-spike/README.md` for prerequisites (likely
   `brew install pkgconf ffmpeg gstreamer`) and install anything missing.
3. Read `spike/rust-decode-spike/FINDINGS.md` in full first so you match its existing level of
   detail and format — don't invent a new one.
4. Run both candidates against both fixtures, software and `--hw`, exactly as documented in the
   README (`ffmpeg-candidate` and `gstreamer-candidate`, timestamps 0.5 and 1.5, both reference
   PPMs), plus `cargo test` for both crates.
5. If anything differs from the M3 results already recorded — VideoToolbox doesn't engage,
   different pixel diffs, different build friction — investigate briefly like the M3 findings did.
   Don't just report pass/fail; explain what you saw. Also re-check the criterion #5 licensing
   table (`gst-inspect-1.0 <plugin> | grep License`, `otool -L` on the linked dylibs) and confirm
   it matches the M3 findings on this machine.
6. Update the Mac M2 cells in `spike/rust-decode-spike/FINDINGS.md` with your actual results, and
   update "Recommendation" only if your results actually change the conclusion.
7. Commit with a `docs(spike):` prefix and push to the existing branch
   `spike/issue-108-decode-binding` — don't open a new branch or PR, #109 already exists.
8. Comment on PR #109 summarizing what you found.

Don't modify the candidate source code unless something doesn't build on your machine — this is a
verification pass, not a rewrite.

Windows + NVIDIA

This repo has a spike branch `spike/issue-108-decode-binding` (PR #109, issue #108) comparing
`ffmpeg-the-third` vs `gstreamer-rs` for hardware-accelerated video decode. Real Mac M3 results
are already recorded in `spike/rust-decode-spike/FINDINGS.md`. I'm on Windows with an NVIDIA GPU —
the NVENC/NVDEC hardware path isn't implemented yet for either candidate (left as a TODO), so you
need to implement it, then fill in the Windows+NVIDIA rows with real results, not scaffolding.

1. Checkout `spike/issue-108-decode-binding` and pull latest.
2. Read `spike/rust-decode-spike/README.md` and `FINDINGS.md` in full, and read both
   `spike/rust-decode-spike/ffmpeg-candidate/src/main.rs` and
   `spike/rust-decode-spike/gstreamer-candidate/src/main.rs` — you're extending the existing
   pattern, not starting over.
3. Install prerequisites: FFmpeg dev libs + pkg-config (e.g. via vcpkg), and GStreamer's official
   Windows installer (development package) including the `nvcodec` plugin.
4. `ffmpeg-candidate`: the `hw` module currently only has a `#[cfg(target_os = "macos")]`
   VideoToolbox implementation. Add a `#[cfg(target_os = "windows")]` variant that uses
   `AVHWDeviceType::CUDA` instead of `AVHWDeviceType::VIDEOTOOLBOX` in `av_hwdevice_ctx_create`,
   with `get_format` looking for `AVPixelFormat::CUDA`. Wire it into the existing `--hw` flag the
   same way the macOS path is wired in. Leave the seek/decode/compare flow unchanged.
5. `gstreamer-candidate`: currently hardcodes `vtdec_hw` for `--hw`. Make the decoder-element
   choice platform-conditional: `nvh264dec` (from the `nvcodec` plugin) on Windows, `vtdec_hw` on
   macOS.
6. Build and run both candidates against both fixtures, software and `--hw`, plus `cargo test` for
   both crates.
7. Investigate honestly like the M3 findings did — if hardware decode fails to engage, produces
   wrong pixels, or the build has friction, document exactly what happened, not just pass/fail.
   Check plugin licensing for criterion #5 too (`gst-inspect-1.0 nvcodec | grep License`, and what
   the nvcodec DLL links against) and add a Windows/NVIDIA row to that table.
8. Update the Windows+NVIDIA rows in `spike/rust-decode-spike/FINDINGS.md`, and update
   "Recommendation" if your results change the conclusion — the current recommendation is
   explicitly contingent on this row, so it's entirely possible it does.
9. Commit code changes separately from findings (code as `spike(rust/ffmpeg):` /
   `spike(rust/gstreamer):`, findings as `docs(spike):`), and push to the existing branch
   `spike/issue-108-decode-binding`.
10. Comment on PR #109 summarizing what you found and whether the recommendation should change.

[missabawse]

Mac M2 verification complete

Hardware: Apple M2 (MacBook Air, Mac14,2), macOS 14.5, Rust 1.97.1

All results match M3 exactly across all 8 test cases:

Candidate / path	t=0.5s	t=1.5s
ffmpeg-the-third, software	mean=0.000, max=0 — PASS (pixel-exact)	mean=0.000, max=0 — PASS (pixel-exact)
ffmpeg-the-third, --hw (VideoToolbox)	HARDWARE PATH: active; mean=0.000, max=0 — PASS (pixel-exact)	HARDWARE PATH: active; mean=0.000, max=0 — PASS (pixel-exact)
gstreamer-rs, software	mean=1.391, max=6 — PASS	mean=1.377, max=6 — PASS
gstreamer-rs, --hw (vtdec_hw)	HARDWARE PATH: active; mean=3.855, max=168 — FAIL	HARDWARE PATH: active; mean=3.886, max=181 — FAIL

cargo test: 3/3 ffmpeg-candidate, 2/2 gstreamer-candidate — all pass.

One notable difference from M3: The GStreamer-GL-WARNING: An NSApplication needs to be running on the main thread did not appear on macOS 14.5 (Darwin 23.5.0). The vtdec_hw pixel failure is the same magnitude regardless — the colorimetry issue is not GL-warning-correlated but a consistent vtdec_hw characteristic across both M-series chips.

Criterion #5 licensing (M2): matches M3 findings exactly — applemedia/vtdec_hw is LGPL + Apple system frameworks only (clean); libgstlibav links this machine's Homebrew ffmpeg 8.1.2 (--enable-gpl), same risk profile as M3.

Build notes: Rust not preinstalled (rustup install required). pkgconf and gstreamer installed via Homebrew before the first build. No build errors or patching required on either candidate. First-build times ~17s each from cold.

Recommendation unchanged: ffmpeg-the-third. The M2 results add confidence — VideoToolbox pixel-exact behavior is consistent across both M-series targets, not an M3-specific result.

FINDINGS.md updated and pushed to this branch.