flaky test: external_command_tests::non_interactive_stdin_is_dev_null fails under full-suite load

[tobert]

Pre-existing flake, first noted in the #48 session and hit again 2026-07-06 during an unrelated PR's cargo test --all run: non_interactive_stdin_is_dev_null failed with stdin should be /dev/null: socket:[...] under peak parallel load, then passed in isolation (and historically passes full-binary and 4× reruns). The #[ignore]'d sibling test notes the same stdin-harness sensitivity.

Worth a flake-hardening pass: either serialize the stdin-sensitive externals behind a mutex, or make the assertion tolerant of the harness's socket-stdin when the test runner itself owns fd 0.

[tobert]

Spent a genuine effort trying to reproduce this under load; could not, despite pushing well past the conditions under which it was originally seen. Leaving this open with the evidence rather than shipping a speculative fix (mutex-serialize vs. loosened assertion) with nothing to verify it against.

What was tried (branch fix/flaky-stdin-test-135, no code changes made):

1. Baseline: cargo test -p kaish-kernel --test external_command_tests --features subprocess -- --test-threads=64 — 1 run, clean (target test ok).
2. hammer.sh: 15x repeats of the same isolated-binary invocation via cargo test — 15/15 clean.
3. Direct cross-process hammering (bypassing cargo, invoking the compiled external_command_tests binary directly, many concurrent OS processes at once, repeated in rounds):
   • 8 rounds x 8 concurrent copies x --test-threads=32 = 64 copy-runs
   • 40 rounds x 24 concurrent copies x --test-threads=96 = 960 copy-runs
   • 60 rounds x 32 concurrent copies x --test-threads=128 = 1920 copy-runs
   • 50 rounds x 48 concurrent copies x --test-threads=16 = 2400 copy-runs
   • 30 rounds x 64 concurrent copies x --test-threads=32 = 1920 copy-runs
   • 7264/7264 copy-runs clean — zero failures of the target test at any concurrency/thread-count combination tried.
4. Full-suite cargo test --all --features subprocess -- --test-threads=<N> (closest match to how the flake was originally observed):
   • One run at --test-threads=128: fully clean (grepped the whole log for FAILED|panicked|should be /dev/null — no hits).
   • A batch at --test-threads=64: target test passed in every run where it got to execute; the batch did hit other, unrelated pre-existing flakes (see below) that made cargo test's fail-fast stop before reaching external_command_tests in one run.
   • One attempt at --test-threads=256 was abandoned after ~30 min — that thread count caused severe scheduling thrash (the process sat futex-blocked on 2 threads instead of exercising real parallelism) rather than useful additional contention.

Grand total: ~7,283 executions of non_interactive_stdin_is_dev_null, 100% pass rate.

This was run against a genuinely, heavily loaded host — not synthetic contention. Host load average fluctuated between ~22 and ~129 on a 32-core machine for the full duration, from many concurrent unrelated cargo build/cargo test processes (other agents working in sibling worktrees at the same time), confirmed via ps//proc/loadavg throughout.

Collateral finding: this hammering did surface two different, unrelated pre-existing timing-sensitive flakes that fire reliably under this kind of load — filed separately as #148 and #149 — which is good evidence the load itself was real and biting genuine timing assumptions elsewhere in the suite. non_interactive_stdin_is_dev_null simply never joined them across ~7,300 tries.

Conclusion: without a reproduction, I'm not shipping a speculative mutex/serialization or a loosened assertion — there's nothing to verify either fix against, and per project convention a silent-fallback-style loosening without cause is exactly the kind of change that shouldn't go in un-evidenced. Leaving this open. If it recurs, capturing the exact failing environment (concurrency level, ulimit -n, whether the harness's own stdin really is a socket at that moment, cgroup/container details if any) at the moment of failure would help a lot more than further blind hammering.

[tobert]

Root cause found and fixed while bringing up CI (PR #169, commit 76b567f).

This was never a load/timing flake. The test's comment said "Use /bin/readlink to bypass the builtin," but the code ran bare readlink — which resolves to the kaish builtin, and the builtin resolves /proc/self/fd/0 of the test process itself, not of any spawned child. So the assertion was measuring whatever stdin the test runner happened to inherit from its environment: /dev/null under most local invocations, socket:[...] under the harness conditions this issue recorded, pipe:[...] on the GitHub Actions runner (where it failed deterministically on CI's first-ever run — 0% flaky there, 100% wrong).

The fix makes the code match its comment: /bin/readlink, so a real child is spawned and Stdio::null() is actually observed. Reproduced red locally with echo poke | cargo test -p kaish-kernel --features subprocess --test external_command_tests non_interactive_stdin_is_dev_null, green after the fix under the same piped stdin, and green across three full CI runs since. The #[ignore]'d interactive sibling got the same treatment for intent-consistency.

🤖 Generated with Claude Code