Skip to content

fix: codex status lane heals itself — Rescan forwarded, busy-deadman force-reads (kata namg) - #574

Merged
danshapiro merged 7 commits into
mainfrom
fix/codex-lane-self-healing
Jul 29, 2026
Merged

fix: codex status lane heals itself — Rescan forwarded, busy-deadman force-reads (kata namg)#574
danshapiro merged 7 commits into
mainfrom
fix/codex-lane-self-healing

Conversation

@danshapiro

Copy link
Copy Markdown
Owner

Fixes kata namg — the codex rollout status lane could fail to deliver terminal.turn.complete FOREVER if a single inotify event was missed (a codex done chime that never fires; observed as a load-only test red that burned its full budget).

Root cause (systematic-debugging code audit, verified against the notify crate's vendored source): the lane was a single unbroken chain with zero redundant delivery — both file watchers discarded notify's IN_Q_OVERFLOW Rescan signal (EventKind::Other not in the relevant-events filter), and codex lacked the self-healing floor amplifier already had (its busy-deadman only flipped phase and never emitted ForceRead; the hub's ForceRead arm for codex was a dead no-op).

Fix:

  1. Both lanes now forward the watcher's Rescan signal as an immediate catch-up read
  2. The codex busy-deadman now re-reads the rollout file and stays busy, re-arming per window, with a submit-time staleness escape (load-bearing check found removing the old give-up path could eat a chime when typing into a stale pane — pinned) and a disarm-on-resumed-output fix (plan review caught a hot-loop)
  3. The hub drains codex force-reads, pinned by a deterministic watcher-dropped-system-heals test

Deferred (per spec authorization): RolloutTailer IO-error degrade parity with amplifier — named residual risks recorded.

Cleanup: Orphaned e2e test of an already-retired protocol channel deleted (verified pre-existing failure).

Gates: cargo fmt/clippy/workspace green, npm suites green, test:port green, release build + 4 e2e suites green.

…idation (kata namg)

Validated the plan's load-bearing assumptions (ledger in
.worktrees/.the-usual-logs/codex-lane-self-healing/load-bearing-ledger.md):

- D1 clause (c) FALSIFIED by state-machine trace: removing the
  Busy->Unknown demotion made a submit into a wedged pane lose its chime
  entirely (queued branch spends the real BEL on the phantom). D1 revised:
  keep ForceRead/stay-Busy/re-arm, add a submit-time staleness escape in
  note_input's Busy branch (old demotion's submit semantics, same
  threshold); Task 3 gains the red-first pin + re-derived CE2 test.
- Flake-attribution narrative FALSIFIED against the record (f2c505e +
  deflake report): the 30s widening absorbed the DELAYED-under-load class;
  the LOST class (this fix) blew even 30s. Constraint retained, narrative
  corrected, Task 5 Step 2 gains an honest scope note + contingency.
- D2 deferral verified against the recovered spec verbatim; named residual
  risks added (silent attach failure, busy-edge loss, codex-version pin)
  plus a landing-time kata-comment note.
- Task 1 gains a line-anchor drift guard over the four touched files.
- Plan bug fixes: Task 2 Step 6 grep expectation (two matches after the new
  test), Task 3 multi-filter cargo test commands (filters after --), stale
  'no local commits yet' wording.
…eyes iter 1, kata namg)

Independent cross-model review found a blocking defect in the plan's
verbatim tracker code: the amplifier idiom never clears a fired
next_force_read_at when Busy-phase output resumes, pinning next_deadline()
at a past instant while the fire guard stays false -- the hub loop spins
expire_due at wait=0 until the turn ends. Plan now adds Task 3 Step 3g
(note_output liveness branches reset the anchor + warn latch), a red-first
pin (resumed_output_disarms_the_fired_deadman_anchor), a module-doc
divergence note, and a named D2 residual for the amplifier template's
latent copy (out of scope; follow-up kata). Sub-steps 3g/3h renumbered to
3h/3i with cross-references updated.
…etired in 4767b7e (Task 5 e2e gate, kata namg)

The 'fresh codex terminal: candidate adoption stamps sessionId on
turn-complete' test injected terminal.codex.candidate.persisted and waited
for an identity-bearing upsert -- but 4767b7e (feat(ws)!: retire
terminal.codex.candidate.persisted writer) made the server's only handler
arm accept-and-ignore by design, so the test could never pass. It was added
one day before the retirement and missed by that campaign (which touched no
e2e specs); deterministically red on main since 2026-07-26, surfaced by
Task 5's e2e gate.

No live-behavior coverage lost: fresh-pane locator identity is pinned by
crates/freshell-ws/tests/codex_locator_activity.rs, the accept-and-ignore
contract by codex_candidate_inert.rs, and wire-level sessionId-on-
turn-complete by this spec's passing restartAbrupt (G9) test. Header
comment updated to stop citing the deleted codex_candidate.rs adopt path.
@danshapiro
danshapiro merged commit c1c6746 into main Jul 29, 2026
3 checks passed
pull Bot pushed a commit to HinchK/freshell that referenced this pull request Jul 29, 2026
…danshapiro#574 codex-lane-self-healing) into rebind-review-polish

Semantic merge of the opencode signal lane, preserving both sides:

- main's OpencodeDrainOutcome split (rebinds + hellos), plugin-alive hello
  delete-on-read, HelloTracker with once-per-pane grace-window warning
- branch's SignalDisposition ladder (Acted/Retain/Discard) with explicit
  foreign-provider signal discard, stale .tmp staging reap, and all other
  polish items (deterministic claude drain ordering, portable
  timestamp-first nonce, sidecar guard parity, warn-once fork-ambiguity,
  percent-encoded plugin URL, codex_locator test split)

Conflicts (crates/freshell-ws/src/opencode_signal.rs):
- drain() doc + signature: kept OpencodeDrainOutcome return with combined
  doc covering hellos AND stale-.tmp reaping
- drain_and_rebind_opencode(): kept main's hello-tracker pass, folded the
  rebind loop onto the branch's SignalDisposition match
- adapted branch test drain_reaps_stale_tmp_staging_files_but_keeps_fresh_ones
  to the OpencodeDrainOutcome shape

Hello files are discriminated at parse time (before the rebind ladder), so
the foreign-signal Discard path can never consume a hello as "foreign".

Verified: cargo fmt --check, clippy -D warnings, cargo test --workspace
(2198 passed / 0 failed), contract freeze (test:port + regen idempotency +
freshell-protocol --locked), coordinated npm run check.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant