Session status sticks on wrong value ("Action required" while working) — durability asymmetry between turn-status and agent-state writes

[danljungstrom]

Summary

Session status frequently goes stale and "sticks" on the wrong value — most often "Action required" shown while the session is actually working, and less often "Working…" shown when the session is idle / awaiting review.

Root cause is a durability asymmetry between two independent daemon→server write channels:

• Channel A — turn status (latestTurnStatus): a durable, disk-backed outbox, retried with backoff, replayed on reconnect and on daemon restart.
• Channel B — agent-state pending counts (pendingPermissionRequestCount / pendingUserActionRequestCount): best-effort. Bounded in-memory retry, then the failure is swallowed. No disk persistence, no restart replay, and no re-push on reconnect.

When a Channel-B "clear pending counts" write is lost, the server keeps pending…Count > 0 indefinitely. The status presentation layer then revives that stale flag as soon as the next (durable) turn starts and sets working = true, and the status cascade ranks action_required/permission_required above thinking. Result: a working session displays "Action required".

This is provider-agnostic (Claude / Codex / OpenCode all converge on the same shared code) and is aggravated on self-hosted servers (SQLite/pglite write contention + flakier networking raise the drop rate).

All file references are against origin/dev HEAD.

Evidence

1. Status cascade ranks action/permission above working

apps/ui/sources/utils/sessions/sessionUtils.ts — getSessionStatus returns action_required and permission_required before the isThinking (working) branch. So whenever both are "true", the action/permission label wins.

2. Presentation revives a stale pending flag using working

apps/ui/sources/sync/domains/session/attention/deriveSessionRuntimePresentationState.ts:88-91:

freshActionRequired:
    input.hasPendingUserActionRequests === true
    && isLiveRuntime
    && (working || hasFreshPendingRequest),

The || working clause means a stale, never-aged pending flag is treated as fresh whenever the session is working — instead of being suppressed by it. Same pattern for freshPermissionRequired.

(Related smell: readFreshInProgressRuntimeSignalTimestamps at :107-116 uses generic activeAt as in-progress work-freshness, contradicting the module's own doc comment "Generic activity, presence, and active process state do not imply active work." This is the secondary "stuck Working" path for sessions with latestTurnStatus === null, where the legacy thinking flag is re-written true by presence keepalives — apps/server/sources/app/presence/sessionPresenceWritePlan.ts thinking-true branch.)

3. Pending counts clear only via a non-durable, swallowed path

• Sink: apps/cli/src/api/session/sessionWritesBestEffort.ts — updateAgentStateBestEffort calls session.updateAgentState(...) and .catch(...) logs the failure as "non-fatal". No re-throw.
• Underlying write: apps/cli/src/api/session/stateUpdates.ts:144 updateSessionAgentStateWithAck, wrapped in backoff (apps/cli/src/utils/time.ts:95, default maxFailureCount: 8, maxDelay: 1000). After ≤8 attempts (~5–8s) it throws → swallowed. No outbox, no persistence.
• All providers route through it: each backend's permission/question event → shared apps/cli/src/agent/permissions/agentStateRequestStore.ts completeRequest() / publishRequest() → updateAgentStateBestEffort. permission vs user_action is a kind field (apps/cli/src/agent/permissions/requestKind.ts: AskUserQuestion/ExitPlanMode/AcpHistoryImport → user_action), not a provider split.

4. Turn status IS durable (the asymmetry)

apps/cli/src/api/session/mutations/createSessionMutationOutbox.ts: disk-backed outbox, exponential backoff, replayed on startup (flush('startup')) and on reconnect, socket→HTTP fallback, waits for server ACK.

5. No self-heal of pending counts on reconnect

apps/cli/src/api/session/sessionClient.ts:689-722 (onConnected): on reconnect it flushes the durable turn outbox (sessionMutationOutbox.flush('connect')) but does not re-push agent-state. syncSessionSnapshotFromServer only runs when agentStateVersion < 0 (uninitialized) and is a server→client pull (apps/cli/src/api/session/snapshotSync.ts — applies server state only when serverVersion >= localVersion), so it cannot correct a stale-but-versioned server count; it would propagate it. Net: a dropped clear is corrected only by the next successful agent-state write — i.e., the next permission/user-action publish or completion with a healthy socket. If the session then runs a long stretch of pre-approved work (no new prompts), the stale "Action required" persists for that whole stretch.

6. No server-side coupling

apps/server/sources/app/session/sessionWriteService.ts — applySessionTurnMutationInTx writes latestTurnStatus / legacy thinking but never touches pendingPermissionRequestCount / pendingUserActionRequestCount. Turn begin does not zero pending counts; turn complete does not zero them. updateSessionAgentState (the only pending-count writer) and the turn mutation are independent, provider-agnostic functions with no per-provider branching. The only robust clear-on-done (pendingRequestsCount: 0 on the ready event in apps/cli/src/session/transport/socket/sessionSocketAgentState.ts) is client-side only and never persisted to the server.

Reproduction (hypothesized — code-traced, not yet live-reproduced)

1. Agent raises a permission / user-action request → server pending…Count = 1.
2. User answers; daemon attempts the clear, but the socket is down for > ~8 attempts (or the daemon restarts mid-flight) → the clear is dropped and swallowed.
3. Agent starts the next turn → durable outbox sets latestTurnStatus = in_progress → working = true.
4. UI: freshActionRequired true via || working; cascade ranks it above working → "Action required" while working, surviving reloads and reconnects until the next successful agent-state write.

Provider scope

Provider-agnostic. Claude / Codex / OpenCode all feed the shared agentStateRequestStore → shared best-effort sink; single server function; no per-provider branch; no durable escape hatch in any backend. Frequency skews toward Claude only because Claude prompts most.

Self-hosted exposure

Same code on all hosts, but self-hosting raises the trigger rate:

• DB engine. Hosted = PostgreSQL. Self-hosted "light" single-binary = SQLite (or pglite) — apps/server/sources/storage/prisma.ts (initDbSqlite, @/flavors/light/sqliteConnectionConfig). The pending-count clear lands via updateSessionAgentState → inTx (SQLite retry: maxRetries 8, totalRetryBudgetMs 5_000). Under concurrent presence flushes + turn writes + agent-state writes, SQLITE_BUSY contention can exhaust the budget → ack error → swallowed → dropped clear. The presence cache also self-throttles on SQLITE_BUSY (apps/server/sources/app/presence/sessionCache.ts, dbFlushBackoffUntil = 30s). Postgres rarely fails this write.
• Network. Self-hosted is typically reached over a home / VPN link with more daemon↔server disconnects → more Channel-B losses (which, per §5, do not self-heal on reconnect). Channel A survives; Channel B does not.

Proposed fixes (pick 1 primary + presentation hardening)

• Server coupling (recommended, smallest robust fix): in applySessionTurnMutationInTx, zero pendingPermissionRequestCount / pendingUserActionRequestCount when latestTurnStatus transitions to in_progress (and on terminal). Closes the gap regardless of Channel-B delivery. As a same-transaction write it adds no extra SQLite contention and inherits Channel-A durability — good for self-host.
• Or make Channel B durable: route pending-count clears through an outbox like turn mutations.
• Presentation hardening (do regardless): drop || working from freshActionRequired / freshPermissionRequired so a stale pending flag must age out on its own pendingRequestObservedAt; stop using generic activeAt as in-progress work-freshness.
• Optional server reaper: TTL that zeroes pending counts / demotes in_progress when no genuine turn/agent-state write (not presence) has landed in N seconds.

Caveats

• All findings are code-traced against origin/dev, not yet live-reproduced. The real-world frequency depends on socket-outage / restart / SQLite-contention timing, which has not been instrumented.
• Any behavior-changing fix should land test-first (TDD): a failing test reproducing steps 1→4 should precede the fix.