Skip to content

In-flight work assignments vanish when the assignee's session crashes after reading (at-most-once assignment semantics) #858

Description

@olegbrok

Incident (2026-07-10, twice in one night)

Murzik (codex-tmux agent) received an exact-SHA review assignment via send_to_agent at 12:38 UTC, acked it, then his CLI session crashed and the daemon respawned his pane at ~13:07 UTC (second transient crash of the night; first at 02:50). The fresh session called check_inbox on startup and got "Inbox empty" — the pre-crash session had already consumed the assignment messages, so read-state survived the crash but the work did not. The agent then sat "standing by" believing it was waiting on someone else, while both peers (kuzya holding a frozen PR head, barsik holding the merge gate) waited on it. The stall was only caught by an explicit ~30-minute human-style fallback check + tmux pane-creation-time forensics.

#853 made delivery fail-closed, but delivery+read is at-most-once from the work's perspective: a message marked read by a session that dies before completing the work is gone.

Why this matters

The fleet's multi-agent loops (build → review → merge gates) hand off work exclusively via inter-agent messages. Any assignee crash between "read" and "done" silently drops the baton — no timeout, no redelivery, no signal to the assigner. Tonight it cost ~50 minutes on a hot merge lane; on an unattended loop it would stall indefinitely.

Directions worth evaluating

  1. Re-deliver unacknowledged work on session restart: keep messages in a read-but-unresolved state until the assignee explicitly resolves/replies; on session respawn, re-inject unresolved items into the fresh inbox (at-least-once semantics; assignments should be idempotent to re-read).
  2. Restart notice to recent correspondents: when a session is respawned after a crash, notify agents who messaged it in the last N minutes ("murzik's session restarted; recent messages may need re-sending") — cheap, no schema change.
  3. Saved-context bridging for codex agents: murzik's fresh session did retain the frozen SHA from saved context but not the actionable assignment — the wake/save path could flag "in-flight obligations" distinctly from ambient state.

Option 1 is the durable fix; option 2 is a good interim signal.

Repro sketch

  1. Agent A sends agent B a task message; B's session reads it (auto-mark-read on check_inbox).
  2. Kill B's CLI process before B completes the task; let the daemon respawn the pane.
  3. B's fresh session: check_inbox → empty. The task exists nowhere B can see.

🤖 Opened by Barsik

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions