fix(scheduler): retire late-confirmed persisted wakes - #993
Merged
Conversation
olegbrok
marked this pull request as ready for review
August 3, 2026 22:53
olegbrok
approved these changes
Aug 3, 2026
olegbrok
left a comment
Collaborator
There was a problem hiding this comment.
Verified independently at a4e94dd in a clean worktree: 241 passed (scheduler 124 + registry 117), ruff clean. Registry method is atomic under _rmw_lock, keyed on UNIQUE(schedule_id, fired_at), includes parked rows, logs only on an actual retire with row id; scheduler confirmed branch retires-or-falls-back with exactly one last_delivered stamp either way; fired_at key = schedule.last_run, same immutable value _record_schedule_undelivered persists. Duplicate-cohort second confirm degrades to the fallback stamp harmlessly. Receipt semantics untouched. End-to-end regression proves no replay duplicate after a late primary confirm.
🤖 Reviewed by Barsik
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
confirm_pending_schedule_wake_by_fire(schedule_id, fired_at, *, delivered_at)to atomically stamplast_deliveredand delete the exact persisted fire under the registry RMW lock_deliver_scheduleconfirmation branch use that keyed retirement, falling back to the existing delivery stamp when no row was persistedPERSISTED_WAKE_RETIRED_ON_LATE_CONFIRMwith the durable row and schedule IDs only when a row matchedWhy / root cause
The primary delivery path could persist an outbox row after an unconfirmed receipt, then later confirm that same schedule fire without retiring the row. Only replay called the ID-keyed confirmation method, so every later replay boundary could deliver an already-executed wake again.
Live evidence on the Pi box showed one fire persisted at 15:41:16, a primary-path confirmation at 15:43:07, and the still-live row replaying and retiring only at 15:46:06. This change keys the late primary confirmation to the same immutable
(schedule_id, schedule.last_run)pair used when the fire was persisted.Impact
A late positive receipt on the primary path now completes the durable outbox lifecycle immediately. Ordinary confirmations with no persisted row behave as before, and parked dead letters are removed when the exact fire is conclusively confirmed.
Receipt waiting, matching, and acceptance semantics are unchanged.
Out of scope follow-up
The same live trace also contains a second primary-path confirmation for the same firing 28 seconds after the first. A read-only caller scan found one production chain only:
_check_schedules→_deliver_schedule_group→_deliver_schedule. That narrows the separate defect to duplicate cohort dispatch or scheduler-instance/read-visibility behavior; this PR intentionally does not change that path.Checks
uv run pytest -q tests/test_scheduler.py: 124 passeduv run pytest -q tests/test_agent_registry.py: 117 passeduv run ruff check .: passedgit diff --check: passedRefs #991
🤖 Opened by Murzik