CodexTmuxSession watchdog force_restart loop: inflight-head false-wedge + replay amplification (murzik, 2026-07-02)

[olegbrok]

Tonight (2026-07-02 ~03:50-04:25 UTC, post-26.07.001) murzik-tmux entered a self-amplifying force_restart loop. Mitigated by disabling the agent + killing the session (~04:35 UTC); re-enable scheduled for morning.

Signature (logs/api.log)

Every ~610s:

tmux[murzik]: inflight head aged 611.0s > 600.0s, transcript quiet + REPL not idle — REPL stuck; scheduling force_restart (deque depth=3 → 6 → 7 → 7)
tmux[murzik]: requeued 2 → 5 → 6 → 6 turn(s) for replay after force_restart
tmux[murzik]: codex_cmd_built mode=resume force_fresh=False prior_rollout=True
tmux[murzik]: wake_prompt_sent reason=wake_resume prompt_chars=1945

Anatomy

1. Likely trigger: murzik's legitimate review turns run >600s (his Investigate Anthropic Beta Advisor Tool for sub-model delegation #212/Agent safety profiles using CC sandbox + budget controls #213/Stream Codex CLI incremental events through SSE to Chat UI #214 reviews took ~10 min each) — the inflight-head ceiling killed a live session mid/post-turn.
2. False-wedge persistence: after restart, "transcript quiet + REPL not idle" keeps evaluating true on the Codex composer — the inflight head never clears, so the watchdog re-fires every 600s. Smells like the embed() failures not validated — empty embeddings stored silently #291 false-wedge class surviving on the CodexTmuxSession path (fix(tmux): stop self-heal clobbering a fresh transcript bind to a stale file (#291) #838 fixed the Claude tailer mtime-floor; codex REPL-idle detection is a different predicate).
3. Amplification: each force_restart requeues turns for replay; codex resume replays rollout history; murzik re-processes OLD inter-agent messages and re-acks them (I received ~10 duplicate acks) → deque GROWS each cycle (3→6→7). Relates to feat(presence): server-side last_seen stamping for non-CC agents #280 dedup (PR fix(broker): agent-reply auto-route as true fallback with dedup (#280) #831, still OPEN/HELD — tonight is fresh evidence).
4. Config not honored (separate finding): murzik's agent config has watchdog_config: {"enabled": false}, yet the inflight watchdog ran and force_restarted. Either the inflight-head watchdog ignores the agent-level toggle or the toggle gates only a different watchdog — decide + document which; relates to the Add PreCompact hook to save agent context before compaction #230 watchdog-scope question.

Mitigation applied (reversible)

• PUT /agents/murzik {"enabled": false} (signed internal API, same mechanism as the feat(api): make heartbeat_interval editable via PUT /agents/{name} #826 incident) + tmux kill-session -t pinky-codex-murzik. Verified no respawn.
• Morning re-enable scheduled; loop-resumption watch included.

Fix directions to evaluate

• Codex-aware REPL-idle predicate (composer state) for the inflight-head check, or exclude/raise the ceiling for CodexTmuxSession where long turns are normal (review work routinely >600s; also see pr-ops watchdog lessons fix(sweep): tmux - delivery retries, REPL death detection, transcript swap race #713/inflight watchdog false-positive: long silent foreground tool call + queued turns → force_restart kills the tool child #731).
• Cap/expire the replay deque: never requeue turns already answered (needs fix(broker): agent-reply auto-route as true fallback with dedup (#280) #831 dedup or a delivered-marker).
• Honor watchdog_config.enabled=false in the inflight watchdog, or rename/split the config so its scope is explicit.

🤖 Opened by Barsik

[olegbrok]

Mitigation correction — the first containment attempt was insufficient and the loop resumed once:

• PUT {"enabled": false} + tmux kill-session did NOT stick: the transport supervisor treated the killed pane as a wedge and healed it (transport_state[murzik-tmux]: reconnecting → connected via internal — 'in-flight connected → reconnecting resolved'), respawning at 21:44:12 PDT and running one more force_restart/replay cycle. enabled=0 gates spawn/routing, not a live session object in the manager.
• Correct mitigation: POST /agents/murzik/stop (manager-level emergency stop) — closed the session cleanly (sessions_closed: 1); verified no respawn after 90s with the tmux session gone.

Two more findings for the fix work:

1. PUT /agents/{name} does not persist auto_restart (response and DB kept auto_restart=1 after PUT with false) — either add it to the updatable field set or document it as immutable.
2. Ops doc-worthy: to stop a looping tmux agent, use the /stop endpoint — killing the tmux session out from under the daemon actively triggers the self-heal you're trying to escape.

🤖 Opened by Barsik