Flaky: run_tests.py intermittently fails host-package drift guard on hosts/claude/.claude-plugin/plugin.json

[Kyarha]

Symptom

python3 scripts/run_tests.py intermittently fails its final host-package drift guard:

ERROR: committed host packages are stale relative to source.
  1 path(s) differ ... hosts/claude/.claude-plugin/plugin.json

But standalone python3 scripts/build_host_packages.py --check always reports OK: committed host packages are in sync with source, git diff is clean, and build_all is byte-for-byte deterministic across repeated builds. So a test earlier in the suite contaminates process state so that check_drift's in-process scratch rebuild diverges from the committed tree — only when run inside the full suite, not standalone.

Failing test

scripts/test_build_host_packages.py::DriftCheckTests::test_check_passes_when_in_sync
(its assertEqual(code, 0, out.getvalue()) prints the drift text on failure; the
jig-drift-… tempdir prefix in the error confirms it's a DriftCheckTests method).
Inside that test the seed build and the check rebuild both build from REPO_ROOT,
yet _diff_packages reports claude/.claude-plugin/plugin.json differing between them.

Nature of the flakiness

• Per-process random, not test-order (discovery order is sorted/deterministic).
• Separate suite processes give different outcomes (repro'd once, passed several times), consistent with PYTHONHASHSEED / cold-cache sensitivity.
• Fixed-PYTHONHASHSEED sweeps (0–7, 12345, 99991) all passed; warm random runs mostly passed. The first run on a clean checkout (cold __pycache__) failed — a leading, still-untested hypothesis is that cold bytecode caches are the trigger (matches "reproduces on a clean checkout").

Ruled out

• build_claude_plugin.build copies source .claude-plugin/plugin.json byte-for-byte; output is deterministic given disk state.
• No test rebinds the build / install_contract / scaffold module attributes (only test_scaffold_mode.py patches scaffold_mod.atomic_write_text, restored in finally).
• No test writes or deletes the real source .claude-plugin/plugin.json (the .claude-plugin unlink/rmtree calls in tests operate on tempdirs / _seed_valid_repo).
• Passing runs show 0 write/delete/rename ops to source .claude-plugin (verified with an fs-tracer).
• CLAUDE_PLUGIN_ROOT is leaked (set to REPO_ROOT, never restored, by skills/memory-sync/test_memory.py:946 and :1035) — but that's deterministic, so it can't by itself explain a flaky failure. Worth ruling in/out as a contributor and fixing anyway (missing tearDown).

The paradox to crack

A pure byte-copy of an unchanged file should make the seed and check builds identical. So the divergence is most likely one of:

1. a presence difference (one tree missing plugin.json), or
2. a source mutation performed by a child process (an fs-tracer scoped to the parent process wouldn't see it), or
3. module/global state that changes the build's file enumeration between the two calls.

Next diagnostic step: capture a failing run's actual scratch-vs-committed bytes — e.g. a temporary dump inside check_drift under if drifted: guarded by an env var — to distinguish content-vs-<absent>, plus a child-process-aware fs-trace.

Repro harness gotcha

Running multiple run_tests.py concurrently deadlocks on git worktree/reservation locks and hangs the suite on a review.py record-review subprocess. Sweep for the flake strictly sequentially. (macOS has no timeout; use a Python Popen + watchdog.)

---

Not yet root-caused. Filed to keep a trace; help/second opinion welcome.

[ramboz]

Second opinion after tracing the build path: the strongest fault line is file enumeration, not CLAUDE_PLUGIN_ROOT, PYTHONHASHSEED, or bytecode itself.

build_claude_plugin.build() verifies that .claude-plugin/plugin.json exists, but then independently discovers package files through Path.rglob(). Python documents that Path.rglob() silently suppresses filesystem-scanning OSErrors. A transient scan failure could therefore let the build return success while omitting the manifest—the exact one-path presence difference we’re seeing.

I’d make the next failing run capture:

• source, seed, and scratch presence/size/SHA-256;
• whether .claude-plugin/plugin.json appeared in each build’s enumerated entries;
• source stat/hash before, between, and after the two builds;
• the preserved scratch directory behind an env flag.

If the scratch file is absent and enumeration omitted it, we have the root-cause class. If both files exist with different hashes, focus on source mutation instead.

One additional partition: the current environment is Python 3.14.0. I’d repeat the cold sequential harness under a current 3.14 patch release and 3.13. CPython fixed a 3.14 pathlib race affecting GIL builds less frequently, so runtime version is worth separating from repository behavior: python/cpython#139001

Separately, the leaked CLAUDE_PLUGIN_ROOT should be cleaned up as test-isolation debt, but I would not claim it fixes this flake.

[ramboz]

Diagnostic instrumentation is now available in draft PR #99, with the repo-native lifecycle record at docs/bugs/008-flaky-host-package-drift-guard.md (DIAGNOSING, gnarly/medium).

Run the sequential repro as:

JIG_HOST_DRIFT_DIAGNOSTICS=1 python3 scripts/run_tests.py

If the unexpected in-sync drift test fails, it reports source/seed/scratch presence, size, SHA-256 and mtime; records whether the Claude manifest was enumerated plus the active builder modules/include roots; and preserves that failing scratch tree. Intentional negative drift tests still clean their scratch trees.

Clean baseline runs completed on CPython 3.14.0 and 3.13.13. This does not close or root-cause the issue; the next real failure should distinguish presence omission from byte mutation.