fix(tmux): resumed (--continue) session comes up identity-unbound → MCP 404 blackout; plain restart re-orphans forever (3rd sibling of #612/#613)

[olegbrok]

Summary

A claude --continue resume on the tmux transport can come up with a live REPL but an unbound MCP identity — the agent's X-Agent-Name is never re-established at the shared :8890 gateway, so every MCP tool call (pinky-self, pinky-messaging, pinky-memory) returns 404 / "Agent not found". The agent is alive and talking but blind to its own tools. Worse, a plain daemon/agent restart re-resumes the same orphaned transcript and reproduces the blank-identity state forever, until a forced-fresh launch (no --continue) is issued by hand.

This is the third sibling of the tmux-relaunch failure family:

	Bad state on --continue	Symptom	Fix
#612	REPL dies on unresumable transcript	crash-loop, dead REPL	post-launch liveness poll + fresh-retry (open)
#613	poisoned transcript (thinking-block 400s)	crash-loop via StopFailure	detect + force_fresh_context_once heal (open)
this	REPL alive but identity-unbound	silent MCP 404 blackout	post-launch identity verify + fresh-retry (none yet)

Same machinery as #612 (force_fresh_context_once retry); new predicate: "is the resumed REPL alive and correctly bound?" instead of just "is it alive?".

Observed (Barsik, prod Mac Mini, 2026-06-02)

• MCP blackout hit mid-session (not just at boot): all pinky MCP tools started 404-ing on a live session.
• Every plain restart re-ran claude --continue and re-orphaned the session.
• The remediation that worked: POST /agents/{name}/streaming/restart — disconnect, clear the persisted resume handle, set force_fresh_context_once, relaunch fresh. A fresh launch re-binds identity correctly.
• It also tripped the restart guard (last save_my_context was 2h+ stale) — and because save_my_context itself routes through the dead MCP, the agent could not refresh its save to satisfy the guard: a deadlock requiring an operator with bump_context_updated_at / bypass.

Root cause

The :8890 MCP gateway is shared across all agents; per-request identity comes only from the X-Agent-Name HTTP header (sourced from each agent's .mcp.json):

• shared_mcp.py:AgentNameMiddleware (L141-163) reads x-agent-name into the _current_agent ContextVar. Missing/blank header → ContextVar stays "".
• make_agent_name_resolver / LazyAgentName.resolve() (L41-137) falls back to closure_agent_name, which is "" in shared-SSE mode.
• MemoryStorePool.get_store db_path_resolver (L241-271) raises ValueError for a blank/unknown agent → surfaced to the agent as 404 "Agent not found". (This is routing-by-name, distinct from Stale PINKY_AGENT_KEY after daemon restart locks agent out of MCP tools (401) #641's 401 auth / stale-PINKY_AGENT_KEY path.)

Why a --continue resume comes up unbound but a fresh launch does not:

• A fresh launch fires SessionStart on a virgin transcript (agent_registry.py hook generator L980-1040; settings matcher .* L1805-1818) and the in-REPL MCP client does a clean cold bind, re-sending X-Agent-Name.
• A --continue resume re-loads the prior conversation; the resumed CC process appears to re-establish its SSE/MCP clients without re-sending X-Agent-Name (or SessionStart does not re-fire the same way), so the gateway never re-learns identity → _current_agent stays blank → 404. (This exact CC resume semantics is the open question below.)
• On plain restart, on_startup (api.py:8460-8473) reads the persisted main_resume handle and calls _start_streaming_session(resume_id=...) without force_fresh_context_once; tmux_session.py:_build_claude_cmd (L1801-1803) then sets use_continue = _has_prior_transcript() and not force_fresh — i.e. it re-resumes purely because a transcript exists on disk. The cwd transcript is the real latch; clearing the DB handle alone does not heal tmux — only force_fresh_context_once does.

Likely trigger: the recent SDK→tmux transport migration left an SDK-origin transcript in Barsik's cwd (the canonical #612 trigger). Here the REPL survives the resume but comes up identity-unbound — a case #612's has-session liveness check does not catch.

Is it a known CC/tmux issue?

Partly. #606 documents a real CC regression family — stateful-MCP reconnect loops (2.1.147–2.1.152), session-resume memory, and "background sessions running on stale daemon before upgrade" (fixed 2.1.153) — the last of which directly intersects our update_and_restart + background-tmux flow. However, production now runs 2.1.160 (> 2.1.153), so those specific upstream bugs are patched. This orphan is therefore most likely the PinkyBot-side resume/identity-rebind gap, not an unpatched CC bug. (Web research in progress to confirm CC --continue SessionStart / .mcp.json re-read semantics — will update.)

Open questions (being resolved)

1. Does CC emit SessionStart on a --continue resume, or only on fresh sessions? If only fresh, that alone explains why resumes never re-open the readiness gate / re-bind identity (smoking gun).
2. On --continue, does CC re-read .mcp.json and re-send X-Agent-Name, or restore MCP client state from the transcript without re-sending headers? If the latter, a resume routes blank independent of the on-disk config.
3. Confirm the 404 originates at the :8890 gateway (blank X-Agent-Name) vs the /transport/transcript-path hook endpoint (api.py:5238, 404s when agents.get(name) is None).

Proposed fix (multi-pronged)

Primary — identity-verify launch hook (sibling of #612). In tmux_session.py:_spawn_tmux_repl / connect(), after a --continue spawn, add _verify_repl_identity_bound() (the identity analog of #612's _verify_repl_survived_launch liveness poll): confirm the agent's MCP identity re-bound within a grace window (e.g. the gateway saw X-Agent-Name == self.agent_name, and/or SessionStart re-POSTed transcript-path for this name). If unbound → kill, set force_fresh_context_once=True, relaunch once fresh, re-verify; fail to DEAD if still unbound. Needs a small side-channel in shared_mcp.py to record last-seen X-Agent-Name per agent (no such signal exists today).

Boot-loop self-heal. api.py:on_startup (L8460): set force_fresh_context_once when the prior session was orphaned/blank (e.g. a last-shutdown manifest flag), so a plain restart self-heals instead of re-resuming.

Fail-closed env. tmux_session.py:_build_repl_env (L1861) / connect — refuse to spawn (or transition to BOOT_FAILED) when self.agent_name is blank, so a nameless session can never launch and route as blank. Guard agent_name non-empty at _start_streaming_session (L2604) / _ensure_streaming_session (L2747) too.

Gateway observability. shared_mcp.py:AgentNameMiddleware (L152) — emit a structured signal / reject (logged 400) on blank X-Agent-Name instead of silently 404-ing per tool, so the daemon can correlate the blackout to a specific resumed session and auto-trigger the fallback.

Secondary — restart-guard deadlock. api.py:_build_restart_guard (L1754) / restart_streaming_session (L6984): when the agent's MCP self-tools are unreachable (so it cannot refresh save_my_context), the staleness guard must not hard-block — downgrade to warn or add a reason="tools_unreachable" bypass, and/or thread an explicit force/bypass_guard param via a new RestartStreamingRequest model (none exists today). Mirror #613's force_restart(bypass_guard=True) / bump_context_updated_at.

Test plan (sketch)

• Unit: resumed-launch-comes-up-unbound → auto-relaunches-fresh; bound-resume skips retry; still-unbound-after-fresh → DEAD.
• Unit: restart guard does not block when MCP identity is unbound / self-tools unreachable.
• Integration on Dymok (tmux validation matrix) before fleet rollout; coordinate with the fix(tmux): self-heal --continue REPL that dies on unresumable transcript #612 branch so the identity check sits adjacent to the liveness check rather than conflicting.

Related

#612 (liveness sibling), #613 (poisoned-transcript sibling), #606 (CC binary / MCP-resume regression context), #623 (shared-MCP per-agent transport auth), #486 (explicit Transport state machine), #543 (SDK→tmux migration validation), #591 (stale saved-context across restarts).

🤖 Opened by Barsik

[olegbrok]

Web research conclusion — confirmed: upstream CC bug, closed as not planned

The mid-session MCP blackout is a known Claude Code bug Anthropic has declined to fix:

• HTTP MCP transport does not auto-reinitialize after server restart (404 "Session not found" is not handled) anthropics/claude-code#60949 — "HTTP MCP transport does not auto-reinitialize after server restart (404 'Session not found' is not handled)." Repro on CC 2.1.145; labels area:mcp / bug / has repro. Closed as not planned. In-issue workaround: "do not restart the server mid-session… /mcp reconnect do NOT recover the transport" — only a full process exit + relaunch recovers.
• [BUG] When Claude Code receives HTTP 404 with a session ID error from an MCP server it should start a new session but it doesn'ty. anthropics/claude-code#9608 — same class on CC 2.0.17: on HTTP 404 {"code":-32001,"message":"Session not found"} CC should send a fresh InitializeRequest without a session id, but doesn't. Notes other clients (e.g. Copilot) handle it correctly. Closed as not planned.
• SessionStart hook does fire on --continue (source=resume) per CC docs — so the hook is not the gap (rules out open-Q Hybrid Memory Architecture: Drop file backend, bridge to Claude Code native memory #1).

How it triggers in PinkyBot

Our :8890 MCP gateway is daemon-hosted (shared_mcp.py:create_shared_app L168-224 mounts the SSE + streamable-HTTP apps; _combined_lifespan L207-216 starts the per-connection session managers). A daemon restart tears down and recreates that app and its MCP session state, so every live agent's Mcp-Session-Id goes stale. The agent's CC process then receives a 404 session-not-found and — per #60949 — never re-initializes → all pinky tools dead for the rest of that session; a plain --continue restart re-resumes the same dead-MCP session; only a forced-fresh relaunch makes CC cold-bind.

.mcp.json is correct (X-Agent-Name: barsik on all three endpoints, written cleanly on the fresh launch) — so this is the MCP session-id layer, not the agent-name header (rules out open-Q #2/#3 as the cause; AgentNameMiddleware blank-name is a secondary/observational surface, not the trigger).

Implication: every daemon restart — including update_and_restart deploys — risks orphaning the MCP transport of any agent in a live/background session. Ties directly to #606 §3 ("background sessions running on stale daemon").

Fix implication (sharpened)

CC will not self-heal (confirmed upstream, closed-not-planned), so recovery must be ours:

1. Detect + force-fresh (the primary fix in the body): a dead/unbound MCP transport on a resumed launch → kill + force_fresh_context_once relaunch. This is the only reliable recovery CC offers.
2. Daemon-restart relaunch should verify-then-force-fresh: api.py:on_startup (L8460) re-resume currently inherits the just-invalidated MCP session. Verify the resumed REPL re-bound its MCP within a grace window; only force-fresh if unbound — keeps conversation continuity in the happy path, self-heals the orphan case. (Avoids the cost of force-fresh-on-every-deploy.)
3. Re-orphan-on-restart + restart-guard-deadlock fixes as in the body.

Net: this resolves open questions 1–3. Root cause is an upstream CC limitation (#60949/#9608) triggered by our daemon-hosted gateway being torn down under live sessions; our fix is detection + forced-fresh recovery, since CC's own transport recovery is broken-by-design here.

🤖 via Barsik