Skip to content

fix(test-runtime): isolate run-graph Codex logs per run to stop flaky live-run detection#274

Merged
ElonSG merged 1 commit into
devfrom
fix/run-graph-codex-log-hermeticity
Jul 14, 2026
Merged

fix(test-runtime): isolate run-graph Codex logs per run to stop flaky live-run detection#274
ElonSG merged 1 commit into
devfrom
fix/run-graph-codex-log-hermeticity

Conversation

@ElonSG

@ElonSG ElonSG commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Root cause

fkst.test.run_graph invoked every department with DeptRunOptions::default_env(), supplying no per-run FKST_RUNTIME_LOG_DIR. fkst.codex_runs (live-run detection) therefore read the ambient user-global Codex log stream (~/Library/Logs/fkst) via sdk_codex's fallback. A matching running record — or a recent completed record from a preceding test invocation — makes a department (e.g. github-devloop-pr.fix) take skip-idempotent(live-exec-ref) after its driving delivery is accepted, so the graph quiesces without spawning the mocked Codex.

Effect: any run-graph test that asserts a Codex spawn is flaky — it fails whenever any ambient codex process runs on the host (e.g. concurrent operator workers) or a prior test seeded the global log dir, and passes in clean CI. This was diagnosed from a github-devloop-pr composed run-graph test (run_graph_polluted_merge_gate_stream_reaches_fix) that failed locally (ambient codex running) but was green in CI; controlled reproduction (seeding the global log dir) flips PASS→FAIL, and max_steps is irrelevant (a passing graph spawns Codex at step 5 of 11; cap 20 still fails with a seeded dir).

Fix (test-runtime only; production untouched)

run_graph now creates a per-invocation runtime_log_dir under the temp dir it already allocates and threads it through run_record to every department execution via FKST_RUNTIME_LOG_DIR (RUNTIME_LOG_DIR_ENV). One shared dir per run_graph call ⇒ codex-run statuses created within a graph stay visible to later steps of that same graph; a fresh dir per call ⇒ graphs are isolated from each other and from the ambient/global stream.

  • Production sdk_codex (read_codex_status_records, runtime_log_dir/default_runtime_log_dir) has zero diff — real runs are unaffected.
  • max_steps unchanged; no production live-run dedup change.

Test evidence (operator-verified outside sandbox)

  • New regression run_graph_isolates_codex_logs_per_run_and_preserves_intra_graph_visibility: seeds a matching ambient codex-run record, asserts the in-graph spawn is NOT suppressed, and asserts intra-graph status visibility + inter-run isolation. RED without the fix (exited 101, "ambient codex status suppressed in-graph spawn", run_graph exceeded max_steps with pending=1); GREEN with it.
  • Full run_graph_cli suite: 6 passed / 0 failed.
  • cargo build -p fkst-framework: clean.

Review record

Operator diff-review: production isolation confirmed (no sdk_codex.rs diff), with_env_var helper is pub(crate) test-infra, per-graph tempdir scoping guarantees both isolation and intra-graph visibility, no existing test weakened. Diagnosed via an independent codex investigation (controlled flaky reproduction) as the sole root cause of the local-red/CI-green divergence.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C8iuJ16FXCA7ejVruPEoZG

⟦AI:FKST⟧

Run-graph Department executions inherited the ambient FKST_RUNTIME_LOG_DIR, so fkst.codex_runs could see user-global live or recent records and suppress mocked Codex spawns.

Give each run_graph invocation a tempdir-scoped Codex log directory shared by all of its Department steps. Cover ambient isolation, same-graph status visibility, and repeated-run isolation.

Tests:
- cargo build -p fkst-framework
- cargo test -p fkst-framework --test run_graph_cli
- cargo test -p fkst-framework --test sdk_codex codex_runs

The full crate test was attempted but the managed sandbox denies the observe integration test Unix socket bind with EPERM.

⟦AI:FKST⟧
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 9 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/fkst-framework/src/test_runtime.rs 0.00% 5 Missing ⚠️
crates/fkst-framework/src/test_department_env.rs 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@ElonSG
ElonSG merged commit 5ec925b into dev Jul 14, 2026
6 checks passed
@ElonSG
ElonSG deleted the fix/run-graph-codex-log-hermeticity branch July 14, 2026 10:24
ElonSG added a commit that referenced this pull request Jul 14, 2026
Generalize #274's per-run_graph Codex log isolation to each top-level Lua test. This closes the producer-side ambient FKST_RUNTIME_LOG_DIR leak that let observation tests pollute later run_graph consumers in the same harness process.

Reuse the existing Department env guard for error-safe restoration, and cover cross-test status isolation plus error and panic cleanup.

⟦AI:FKST⟧

Co-authored-by: Loning <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants