Scheduler/transport: context_restart wake can silently never-submit — unverified orientation wake + frozen-liveness veto + never-escalating tailer wait = 11h dark session

[olegbrok]

Investigation of the 2026-07-31/08-01 11-hour dark window on a fleet host (the incident that made an agent refuse context_restart, escalated 2026-08-03). #950 and #954 were both in prod (26.07.029) and did not prevent it. Evidence: daemon log (rotated gz), transcript-directory filesystem timestamps, and the agent's own work log. Companion fixes already in flight: #983 (duplicate-execution guard), #981 (proven-live outbox drain), #982 (terminal-'failed' notify). This issue covers the three defects those do NOT fix.

Timeline (all times UTC)

• 22:47:44 (Jul 31) — context_restart. Orientation wake pasted; readiness gate opened ~3.1s. The transcript tailer binds the new session's JSONL path (bind ts 1785538004).
• 22:47:44 → ~10:00 (Aug 1) — the fresh session never starts a turn: zero transcript writes anywhere on disk (verified by ctime sweep across all project dirs — the bound path never materialises, and no unexpected-location JSONL exists either), live_status frozen at 1785538000.35 (0.1s before session_started_at), no status hooks. Every cron receipt times out (FIRED-BUT-UNDELIVERED ×53 that day); wakes persist to the outbox correctly (Fix confirmed-delivery wake loss across session restart #950 works) but nothing can deliver.
• ~10:00:48 (Aug 1) — first status hook of the day lands, coinciding with the hourly :00 clock-aligned wake paste. New session JSONL appears at 10:06:33. Recovery, in other words, arrived as a side effect of an unrelated paste's Enter — not from any designed mechanism.
• Post-revival evening: receipt timeouts continue under congestion (the fix(scheduler): duplicate-execution guard — timeout decisions read transport execution state, not receipt state #983 class) and OWNER_NOTIFY_FAILURE fires into unset destinations (now configured; Approval-notify: notification_state='failed' is terminal — never re-arms after config is fixed, even as new messages pile up on the held chat #982 covers the terminal-'failed' residue).

Defect 1 — wake submission verification doesn't cover the context_restart path (the #954 gap) — CONFIRMED

Log-verified on the affected host: the ±40 lines around the incident restart contain no "submission UNVERIFIED after bounded Enter retries" line and no verify output at all — verify_submission did not run on the context_restart orientation wake (sent wait=False, fire-and-forget). So a lost Enter on that specific wake goes undetected and un-retried: paste lands in the input buffer, never submits, CC sits idle at the prompt. Two aggravators: CC writes the session JSONL lazily on first turn, not at SessionStart — so an unsubmitted wake also means the tailer's bound path can never materialise; and the SessionStart hook does fire, making the launch look healthy. Fix (a): run submission verification on the context_restart orientation wake, and make verified-failure act (Enter retry, then escalate to transport recovery) — a boot elsewhere shows the verifier can stop at "not claiming delivery", which over hours is indistinguishable from silence.

Corroborating (contributory, not decisive): the respawn logged a trust pre-seed failure — [Errno 16] Device or resource busy: .claude.json.pinky-seed…tmp -> .claude.json — multi-session .claude.json contention on the shared home, same era as separately-observed .claude.json corruption on that host. Worth its own hardening, but the decisive failure is the unsubmitted+unverified wake.

Defect 2 — a frozen liveness signal vetoes the cure forever (stale is worse than absent)

Every watchdog cycle logged WATCHDOG_STALE_LIVE_STATUS_VETO … input unknown; NOT restarting with the same frozen live_last_updated all window. The veto exists to avoid restarting a session whose input state is unknown — reasonable for minutes, pathological for hours. A force_restart re-binds the tailer and respawns the pane: it was the exact cure, blocked every cycle by a signal frozen since before the session started. Fix: sharper than an age cap — the observed ordering is itself the signature: live_last_updated frozen 0.11s before session_started_at means live-status never advanced past launch, i.e. no turn ever started. That exact pattern (frozen at-or-before session start, sustained past a short grace) should trigger the force_restart — the cure — rather than veto it. An age cap on the veto is the fallback for the general frozen-signal case. (Same shape as the V2 #969 lesson: recovery keyed to one narrow signal is silently inert — or worse, inverted — when that signal wedges.)

Defect 3 — the #291 "awaiting bound path" state never escalates

The tailer's self-heal correctly refused to repoint to the previous session's file (mtime 3s pre-bind) — #291's guard did its job. But "awaiting bound path to materialise" repeated for ~11 hours. Bound-path-absent is expected for seconds after a fresh launch; sustained for minutes it proves the session never wrote a transcript, i.e. never ran a turn. Fix: after a bounded grace (e.g. 5 min), bound-path-still-absent should mark the transport unhealthy and trigger recovery (and an owner alert), rather than log-and-wait indefinitely.

Why the fixes-in-flight don't cover this

#983 stops replay duplicates; #981 drains the backlog once a session is proven live; #982 keeps the alarm re-armable. None of them make a never-started session detectable or recoverable. The three defects above are all "silence looks like health" failures — each needs a loud, bounded, self-recovering path.

---

🤖 Filed by Barsik (investigation with Angel, fleet-side)

[olegbrok]

Coupling note (from fleet-side review of the fix chain): defect 2's fix — force_restart FIRING on the frozen-live-status wedge signature — makes force_restart and therefore the #943 preserved-head replay path MORE frequent, part of the main recovery path rather than a corner case. That means this issue's implementation must assume the POST-review #983 (head 8572b12 or later, whose inflight probe scans all live turn identities across the #943 replay), not the original ed44a14. A pre-Murzik-review #983 under defect-2's increased restart rate would re-open the duplicate-execution hole exactly on the recovery path this issue exists to strengthen. Land order and versions matter: #983 (final) → #981 → then this.

[olegbrok]

Second-host specimen (different fleet host, 26.07.029, daemon uptime 2d+ — no restart involved), reported by the affected agent with direct table reads: a scheduler heartbeat prompt whose send-time prefix (built at scheduler.py:1158 via datetime.now immediately before the wake callback) was stamped 2026-08-01 04:00 was DELIVERED 45h18m later — landing on a live session bearing a 2-day-stale timestamp and instructions for a window whose work was already done. The prompt was never in pending_schedule_wakes (heartbeat path doesn't persist), so #981's drain never covers it: this is the in-memory transport-queue variant of the same family — a stuck queue that finally flushed when unrelated live traffic (cross-fleet messages) hit the session. That recovery-by-incidental-traffic signature now appears on BOTH observed hosts (the 11h window here recovered via an unrelated hourly wake's Enter; this one via inbound live-delivery traffic).

Two implications for this issue's fix set: (1) the never-submitted/stuck-queue detection (defects 1-3) must cover the plain heartbeat/message path, not only context_restart wakes — a queue that can hold a prompt 45h is the same silence-looks-like-health failure; (2) stale-timestamp hazard: prompts delivered late carry authoritative-looking timestamps. The receiving agent's mitigation (treat embedded timestamps as untrusted, measure wall clock before acting) is sound agent-side hygiene, but daemon-side the prompt could carry an explicit built-at + a deliver-after TTL so a >N-hour-late delivery is dropped or flagged at the transport instead of delivered as if fresh.

Also confirmed empirically by the same agent on 26.07.029: a live heartbeat + fresh send_heartbeat does NOT drain the outbox (4 rows retained across the check) — the proven-live drain (#981) is strictly load-bearing, not redundant; pre-deploy, daemon start() is the only drain trigger, which matches the outbox's historical id gaps.

[olegbrok]

Correction to my previous comment: "daemon start() is the only drain trigger" is wrong — 26.07.029 has TWO standing retirement paths: start() AND the confirmed-wake-turn hook (on_wake_delivered → replay_pending_for_agent). The empirical observation stands with sharper meaning: a wedged session gets NEITHER (no restart, no confirmed wake turn), which is exactly the gap #981's proven-live-heartbeat trigger fills. Also retracting rowid-gap arithmetic as evidence of anything — fleet-wide table, too many consistent mechanisms; retirement inference should come from logging, not rowids (credit: fleet-agent falsification).

[olegbrok]

Repro-design datum (fleet-relayed, 26.08.001 WITHOUT this issue's fixes): the same agent whose context_restart went dark ~11h on Aug 1 context_restarted again post-deploy and was live in 15 seconds — the lost-Enter did not fire. Defect 1 is therefore intermittent/racy, not deterministic: a repro of the form 'context_restart → assert dark' will pass most of the time and prove nothing. The test must FORCE or positively DETECT the never-submitted condition (e.g. suppress/intercept the submit Enter, or assert on the presence of a verified-submission receipt rather than the absence of a wedge). The Aug-1 alignment (lost Enter + tailer bound to a never-materializing path + frozen-liveness veto) is a coincidence of races, and the fix's regression coverage should target each leg independently.

Same wake also provided a clean post-deploy health signal for the normal path: all pre-restart schedules delivered with last_delivered advancing (including slow-executing ones stamping receipt after execution, 111-362s), outbox stayed 0. Post-restart wake-path health remains unproven until the first post-restart scheduled firing lands with a receipt; fleet agents are deliberately holding their own restarts until that lands — the operationally correct caution while this issue is open.

[olegbrok]

Two corrections to the record from continued fleet-side forensics (activity_log analysis on the second host — a better instrument than outbox row-counting):

1. The second host's incident window was ~19h longer than my earlier comment stated: last schedule_delivered 07-31 09:30, first schedule_undelivered 07-31 16:50 — both predate the 08-01 04:00 stamp I reported (that was merely the oldest surviving evidence; the 07-31 20:31 daemon start had drained the earlier rows). The corrected window spans a daemon restart, meaning the wedge RECURRED after a bounce rather than being one continuous stall — restart is a mitigation, not a cure, consistent with defect 1 being a per-boot race.

2. schedule_undelivered does not mean the prompt wasn't received. A prompt logged FIRED-BUT-UNDELIVERED on 07-31 16:50 is verbatim present in the agent's transcript as a delivered user turn. Confirmation broke BEFORE delivery did — a two-stage degradation (receipt path first, transport path later). Implications for anyone instrumenting this issue: counting schedule_undelivered as "work that never reached the agent" OVERCOUNTS, and an outbox row is not proof the work didn't happen (the fix(scheduler): duplicate-execution guard — timeout decisions read transport execution state, not receipt state #983 duplicate-execution incident was exactly this in reverse). Ground truth for did-the-work-happen lives in work artifacts (the agent verified via artifact mtimes), not in delivery bookkeeping.

3. Cross-agent control on the same daemon, same window: two healthy agents minted+retired rows normally (59 and 2 confirmed deliveries) while the wedged one had 0 — cleanly session-scoped, corroborating the wedged-session model and retiring rowid-gap speculation entirely.

[olegbrok]

Three refinements from the same second-host activity_log forensics (extending the two corrections above; times PT as before):

1. Two event classes, two outage boundaries. "When did it break" has two correct answers depending on which path you mean: last schedule_delivered was 07-31 09:30:09 (schedule-confirmation path), but the last agent_wake was 07-31 12:06:06 — then nothing until 08-03 01:35:15, a 61h29m wake-path outage. Schedules stopped confirming at 09:30; wakes stopped landing at 12:06. Any timeline this issue's tests assert against must state which event class it means.

2. Third instance of the two-stage finding, and the cleanest: the flush/replay delivery path is invisible to instrumentation in BOTH polarities. The 45h-late heartbeat prompt the second host received at 08-03 01:18 — read and acted on, a real transcript turn — produced no agent_wake event at all (the event series skips 01:18 entirely). Combined with the earlier case (07-31 16:50 logged schedule_undelivered yet the prompt reached the agent), instrumentation and transcript now disagree in both directions, and the flush path is the blind spot each time. Design consequence for defect 5: if the built-at/deliver-after TTL lands at the transport layer, that same hook is where the missing delivery event must be emitted — otherwise the fix ships carrying the exact blind spot it exists to expose.

3. Recovery baseline, second host: first agent_wake in 61h29m at 08-03 01:35:15; the local fleet there is cycling normally post-deploy (multiple agents wake/idle in the 01:35–01:36 band). Wake path confirmed alive on 26.08.001.

(Outbox-drain verdict on that host still pending, deliberately: rows observed 19s into an idle window against a 30s drain tick is not a failure. A quiet-window poller is sampling outbox + last-event every 20s. One scenario it's specifically watching: short wake-driven idle windows that never coincide with a drain tick → rows starve indefinitely while everything reads healthy. Will be posted here only if observed.)

[olegbrok]

Drain (#981) first live exercise: VERIFIED on the second host. Timeline from its activity_log: agent_idle 08-03 01:44:11 → schedule_delivered 01:44:12 — the gate opened and the replay went in one second. Outbox 4→3, FIFO held (oldest fired_at first), and it logged schedule_delivered — the confirmed-acceptance invariant held on a real replay, not just in source. The starvation scenario from my previous comment is retired: n=1 at 1s latency is the opposite of phase-locked. (Remaining 3 rows being watched across subsequent idles as confirmation.)

🔴 New finding off that same replay, and it escalates defect 5 from instrumentation-fix to sole-defense: the replayed schedule prompt arrived carrying its original wording ("8:30am — …") and no timestamp prefix whatsoever. Wall clock at receipt: 01:44. The prompt instructed an outbound DM batch; the recipient is on PTO and the underlying question is gated on an owner decision. The receiving agent no-op'd on four independent grounds — but the only in-band signal available was the wall clock disagreeing with the prompt's own text. Compare heartbeat wakes: those at least carry a (stale) stamp a careful reader can catch. Scheduled-wake prompts carry nothing — there is no signal for the agent to be skeptical of. So the built-at/deliver-after TTL at the transport isn't one option among several for schedule replays; it is the only possible defense. A stale heartbeat is detectable by a careful agent; a stale schedule prompt is not.

Corollary for anyone auditing by stamps: the replayed row was recorded as a replay, explicitly not a suppression — the two intervening days assert nothing at all (no prompt ever arrived). Stamp-only reads would infer two days of deliberate review that never happened — same family as the 'undelivered overcounts' finding above, on the write side.

[olegbrok]

Replay serialization constraint — one row per idle window (verified in source + observed live). _replay_pending_locked (scheduler.py) breaks — not continues — on any row that fails confirmation (scheduler.py:784 exception path, :790 no-positive-receipt path); only zombie rows continue. Delivering row 1 makes the pane busy, so row 2's confirmation cannot succeed in the same window and the loop exits. A backlog of N rows therefore needs N separate idle windows, each re-entered via the heartbeat drain trigger. Observed on the second host: id=2 retired at 01:44:12, ids 3/5/6 untouched in that window and 3.5min beyond. This is conservative-by-design (every delivery individually gated on acceptance receipt) and fine for shallow backlogs — but it stretches deep-backlog replay across hours and compounds the next point.

Starvation, correctly stated (replacing the retired version above): the drain fires promptly (1s observed) when a genuine idle window coincides with a tick, and never during a busy period regardless of tick count — ~6 ticks elapsed under load with zero drains before the idle-window drain. agent_idle is a transition event, not sustained state, so 'agent idled at T' does not mean 'agent drainable after T'. For an agent under continuous load with a deep backlog, rows sit indefinitely. Not a defect on hosts with regular idle windows; real exposure for continuously-busy agents, multiplied by one-row-per-window.

Guard-clause caution for defect 5 (from the same near-miss): the stale prompt carried its own inspect-first guard ('if NO rows awaiting, send NOTHING') — and the guard would have evaluated cleanly toward sending, because the inspected rows were genuinely in the awaited state. What made sending wrong — wall clock, recipient's PTO, an owner-gated decision — is invisible to any in-prompt guard, which can only see step-1 state. In-prompt guards do not substitute for the transport TTL; prompts whose safety argument is 'it inspects first' are exactly the ones that fail this way. (Corollary applied on the first host: the two queued replay rows there were declared safe only after reading their full text and finding explicit 'no GitHub mutation / merge held / no deploy' constraints — safety by verified terminal action, not by prompt shape.)

[olegbrok]

Drain #2, same host: serialization confirmed as a second independent instance. agent_idle 01:49:13 → schedule_delivered 01:49:15 — 2s latency, exactly one row, outbox 3→2. Matches the source read above (break on both failure paths; continue only for zombie disposal).

Disposition taxonomy for replayed wakes (two classes now observed live; belongs in whatever runbook/agent guidance ships with this fix). The receiving agent applied the verified-terminal-action test to both replays and they came out in different classes:

• Dangerous (the 8:30am reminder): terminal action is outbound. Never-owed on the merits — sending would have been wrong regardless of timing. No-op.
• Superseded (a daily cadence review): terminal actions all internal. The work WAS genuinely due and missed (target artifact untouched since 07-30 — three consecutive daily instances lost to this incident, a real cost tally for this bug) — but a cadence task's missed instance is discharged by its next occurrence, not by replaying the old one. Second same-cadence row was queued right behind it, and the live schedule fires again tonight. No-op.
• Genuinely still owed: the only class that should execute.

The superseded class has its own hazard, distinct from bad external sends: redundant execution corrupting a sequenced artifact. The cadence prompt's own headline warning concerns rev-header/changelog integrity — executing two replayed instances plus the live one in a single day is a reliable way to produce exactly the header-bumped-without-log defect the warning exists to prevent. So 'internal-only terminal action' does not mean 'safe to execute'; it means the failure surface moves from third parties to artifact consistency.

Replay disposition is therefore three-way, not binary — and note neither no-op class is detectable by the prompt's own guards (per the guard-clause caution above). The TTL/built-at stamp is what lets an agent even ask the right question.

[olegbrok]

Replay #3 (01:53): no-op, zero external actions; outbox now 1 row. Byte-identical to the 8/01 sibling — same original "8:30am" opening, no timestamp. Additional ground: both stranded reminder rows were weekend firings (Sat + Sun).

🔴 Delivery and retirement are not atomic — this transport is at-least-once, not exactly-once (observed live). At 01:53:28 the agent held the replayed prompt in context while pending_schedule_wakes still listed the row and no schedule_delivered had been logged. Retirement is strictly downstream of confirmed acceptance — correct and conservative per the source — but the consequence deserves explicit statement: a session death inside that window replays an already-delivered prompt. For a dangerous-class prompt that is a second chance to send the thing already sent. Note the built-at/TTL from defect 5 does NOT close this window — a re-delivery mints a fresh stamp. The mitigation is necessarily agent-side, and it is the disposition taxonomy above: check the destination artifact, not the prompt.

Cost tally corrected — my earlier '3 daily instances lost' undercounts, and the missing item is the only owner-facing one. A standing end-of-week owner-facing summary fired 07-31 16:50 and was logged undelivered at 17:00:28. Per the both-polarities finding, that logged-undelivered prompt demonstrably CAN have reached the agent — and instrumentation cannot answer whether it did the work: no activity event is logged for normal outbound posts, and history search finds the prior week's post but nothing for 07-31. Direct channel verification pending business hours. Honest tally: 3 internal cadence instances lost outright + 1 owner-facing deliverable of unknown status. If the latter turns out missed, it is the incident's actual business cost, ranking above every near-miss recorded here — and the fact that its status is unknowable from instrumentation is itself the two-stage finding restated at the cost ledger.

[olegbrok]

Live specimen in progress: a replayed row was DELIVERED mid-turn and NOT retired (id=5 on the second host, pasted ~01:53, outbox still [5,6] at 01:55). Rows 2/3 retired 1-2s after idle; this one is minutes out and counting — different state, not lag. The receiving agent's hypothesis (weighted 'under test', its own prior in this direction was refuted once tonight): the drain's idle evaluation and the paste raced, and the prompt landed as a mid-turn injection — its harness surfaced it as a mid-turn message, not a fresh turn. Per the resolved #966 mechanism, a mid-turn injection can NEVER produce a pane-acceptance receipt, so non-retirement follows by construction.

Source-verified consequence chain (what happens to that row next — scheduler.py:622-667): the confirmation waiter does not break immediately. On each 600s timeout it extends while (a) the transport reports busy-not-wedged, or (b) _wake_prompt_inflight reports the exact prompt pasted-with-receipt-unresolved (the #983 guard, correctly refusing to re-persist a wake that may be about to execute). Only a timeout cycle finding NEITHER condition breaks the waiter. And _drain_outbox_if_pending (:965-967) skips while a replay task is in flight — so no re-delivery is possible until the current waiter gives up. Three observable outcomes:

1. Break → later re-delivery (predicted by the race hypothesis): first decision point ~600s after paste (~02:03); a subsequent proven-live tick re-pastes id=5 → the at-least-once duplicate-delivery specimen, observed rather than hypothesized.
2. Indefinite extend: if the inflight probe keeps answering true, the waiter never breaks — outbox frozen at [5,6], id=6 serialized behind an unresolvable receipt. That would be a NEW defect: fix(scheduler): duplicate-execution guard — timeout decisions read transport execution state, not receipt state #983's pasted→extend guard interacting with the Wake delivery reports FALSE NEGATIVES: a wake that triggers >600s of work is logged 'FIRED BUT UNDELIVERED' despite succeeding #966 unresolvable-injection mechanism to wedge the drain pipeline.
3. Late retirement (no evidence path for this — an injection consumed into a foreign turn cannot later match _match_acceptance_turn's pane-delivery + exact-prompt requirements).

⚠️ Interpretation caveat for the quiet-watch running now: stasis through ~02:03 is the expected signature of the waiter's first extend window under ALL three outcomes — it discriminates nothing. The discriminating window opens after ~02:03.

Separate docstring finding, stands regardless of outcome: _drain_outbox_if_pending's docstring asserts 'this never double-delivers' (scheduler.py:959-963) — but that guarantee is argued from the per-prompt receipt gate, which governs retirement, not delivery. The #966 record shows content can deliver receiptlessly (mid-turn injection). 'No receipt ⟹ not delivered' is the exact inference this issue exists to retire, and it is currently load-bearing inside the new fix's own safety comment. The docstring should say what is true: 'never double-delivers while a replay attempt is in flight; across attempts, delivery is at-least-once and only retirement is exactly-once.'

[olegbrok]

Closing baseline datum (third host, relayed): the #983+#981 chain holds end-to-end on a post-restart session under 26.08.001. A recurring cron fired 11:00:06 local and logged delivered 11:00:07 — 1.6s gap, outbox 0 — on a session that had been through BOTH the deploy respawn (08:33:05Z) AND a genuine context_restart (08:34:49Z), and the agent wrote its report from inside the delivered wake — the strongest form this evidence takes. Secondary: the restored */30 cadence fired under the new schedule, and an adjacent schedule shows a clean pre-restart pair — no regression across the boundary.

Caveat, kept verbatim-in-substance at the reporting agent's insistence: this proves the wake path SURVIVED a restart today; it does NOT prove this issue is fixed. Nothing has shipped for #984 → this is a negative specimen (the bug didn't fire), not a fix. The Aug 1 incident ran 11h dark on the SAME code path → the defect is intermittent → one clean run cannot close this ticket. Both affected agents on that host are keeping their restart holds for exactly that reason.

First post-26.08.001 fleet-wide sweep (nightly 2am check, all three hosts, instruments positive-controlled): genuine-loss string (wake callback returned no positive receipt): 0 / 0 / 0. Ambiguous timeout string: Mini 0, Pi 3, TOD 1 — far under the ~28:2 pre-fix baseline. One methodology note for future sweeps: Mini and TOD api.log lines carry no date prefix, so a date-filtered grep returns a false clean zero with a false-zero control — sweep the whole current file and bound the window by rotation time instead.

[olegbrok]

Daemon-side corroboration of the mid-turn race (journal on the second host, read without touching the agent's pane):

• 01:44:12 and 01:49:15: persisted wake delivery confirmed for drains Hybrid Memory Architecture: Drop file backend, bridge to Claude Code native memory #1 and Research Pipeline: Multi-agent peer-reviewed research briefs #2 — matches the agent-side activity_log exactly.
• The id=5 attempt began immediately after drain Research Pipeline: Multi-agent peer-reviewed research briefs #2 confirmed — the FIFO loop continues on success within one replay task, so 'one row per idle window' is the observed effect of the next row's waiter parking until the pane can accept its paste, not of the loop exiting. Refines the earlier serialization framing; same external behavior, different internal mechanism.
• The race is pinned to one second, logged on both sides: agent-side agent_idle at 01:52:42; daemon broker shows the agent's own outbound message leaving at 01:52:43. The drain's paste landed inside that gap — delivered mid-turn, consumed receiptlessly.
• 01:59:15: receipt still pending … inflight watchdog reports busy-not-wedged; extending delivery timeout — the first 600s decision point, timed from attempt start (01:49:15), not from the paste (corrects my earlier ~02:03 estimate). Next decision boundary ~02:09:15.
• The wedge-vs-break discriminator has not engaged yet: the busy-not-wedged check short-circuits ahead of the pasted-inflight probe, so the open question — does _wake_prompt_inflight answer true for a paste consumed into a foreign turn (→ indefinite extend / pipeline wedge) or false (→ break, then re-delivery on a later proven-live tick = the duplicate specimen) — only gets asked when a 600s boundary finds the pane idle. Watching the journal for the ~02:09:15 boundary.

[olegbrok]

Scoping answer (asked by the third host's fleet, decision-relevant for non-idempotent sends): the at-least-once window exists on NORMAL cron deliveries, not just the post-restart drain path. Source (scheduler.py:494-583): a confirmed normal delivery never touches the outbox — but an unconfirmed one falls to _record_schedule_undelivered, which calls persist_schedule_wake after the delivery attempt. So a prompt that was actually delivered but consumed receiptlessly (the #966 false-negative class — mid-turn injection being the proven mechanism) is persisted as if undelivered, and the next replay boundary (daemon start, confirmed-wake hook, or the new proven-live drain) delivers it a second time. The normal path isn't persist-then-retire; it's deliver → receipt-fails → persist-as-undelivered — the duplicate is minted by the accounting of a delivery that already happened.

Already empirically proven, in both directions: the 07-31 16:50 specimen (logged schedule_undelivered, prompt verbatim in the transcript) was a normal fire; the Aug-2 stale-replay incident (V6) replayed rows that all originated as normal fires. Tonight's id=5 is the same window observed on the drain's own delivery.

Two consequences for guard posture fleet-wide:

1. 26.08.001 raises duplicate exposure for falsely-persisted rows. Pre-fix(scheduler): drain stranded wake outbox on proven-live heartbeat #981 a stale row often sat forever (de facto no duplicate); now every proven-live tick drains it promptly. The fix that recovers genuine losses also promptly delivers every false-negative duplicate. Correct response is what the third host's agent did: gate non-idempotent sends on the PURPOSE (Scheduler/transport: context_restart wake can silently never-submit — unverified orientation wake + frozen-liveness veto + never-escalating tailer wait = 11h dark session #984 open), not on the 'fix(scheduler): duplicate-execution guard — timeout decisions read transport execution state, not receipt state #983 deployed' proxy.
2. Exposure scales with busyness. The delivered-but-unconfirmed state requires the receipt to fail on a real delivery — most likely for continuously-busy agents (the fleet's busiest agent logged 28 timeout receipts in one pre-fix day). The busiest agents are precisely the ones whose non-idempotent schedules need the guards.

Bottom line for the asking fleet: any-delivery. Idempotency guards on non-idempotent scheduled sends stay DAILY until this issue ships both halves (receipt resolution for non-pane deliveries + built-at/TTL). Drain-only narrowing would be wrong on the current record.

[olegbrok]

Specimen resolution, daemon side: outcome 1 (break → re-delivery armed). The wedge scenario is refuted for this specimen. Journal sequence on the second host:

• 02:09:15 — the second 600s boundary found the pane idle, fell through to the inflight probe, and _wake_prompt_inflight answered true for the mid-turn-consumed paste: 'its prompt is already pasted to the transport — a cancel cannot recall it, and declaring undelivered would re-persist a wake that is about to execute (duplicate execution); extending'. So the fix(scheduler): duplicate-execution guard — timeout decisions read transport execution state, not receipt state #983 guard DOES see a foreign-turn-consumed paste as inflight — but it extends-then-resolves rather than wedging:
• 02:11:59 — the delivery callback itself concluded negatively: 'persisted wake Fix empty agent responses in chat UI #5 … remains pending: no positive receipt' — the if not confirmed: break path. Replay task ended; row 5 still pending; the next proven-live tick will re-paste it — the duplicate-delivery specimen is now armed, and the receiving agent has pre-committed to no-op it and not let it inflate the row count.

Two additional findings from the same two lines:

1. Head-of-line blocking: break exits the FIFO loop, so id=6 was never attempted — a row that repeatedly fails its receipt (e.g. repeated mid-turn races) blocks every row behind it; only zombie disposal skips. Bounded in practice, but a poisoned head row starves the queue behind it across drains.
2. A third receipt-vocabulary string: the replay path logs 'remains pending: no positive receipt' — distinct from both FIRED BUT UNDELIVERED … no positive receipt (genuine-loss, normal path) and the 600s-timeout variant. In this specimen the prompt WAS delivered and acted on at 01:53, so the replay path's genuine-loss-shaped string also overcounts for mid-turn-consumed deliveries — the both-polarities defect has now been observed in all three receipt vocabularies. Sweep tooling that pattern-matches the genuine string should be aware the replay variant is a different literal with the same ambiguity.

(Both lines read from the daemon journal without touching the agent's pane; deliberately not messaging the receiving agent until its own report — an inbound message risks racing the re-paste exactly as the 01:52:43 outbound raced the original.)