Skip to content

fix(scheduler): retire late-confirmed persisted wakes - #993

Merged
olegbrok merged 1 commit into
mainfrom
agent/scheduler-late-confirm-retire
Aug 3, 2026
Merged

fix(scheduler): retire late-confirmed persisted wakes#993
olegbrok merged 1 commit into
mainfrom
agent/scheduler-late-confirm-retire

Conversation

@bradbrok

@bradbrok bradbrok commented Aug 3, 2026

Copy link
Copy Markdown
Owner

What changed

  • add confirm_pending_schedule_wake_by_fire(schedule_id, fired_at, *, delivered_at) to atomically stamp last_delivered and delete the exact persisted fire under the registry RMW lock
  • make the primary _deliver_schedule confirmation branch use that keyed retirement, falling back to the existing delivery stamp when no row was persisted
  • retire active or parked rows alike and log PERSISTED_WAKE_RETIRED_ON_LATE_CONFIRM with the durable row and schedule IDs only when a row matched

Why / root cause

The primary delivery path could persist an outbox row after an unconfirmed receipt, then later confirm that same schedule fire without retiring the row. Only replay called the ID-keyed confirmation method, so every later replay boundary could deliver an already-executed wake again.

Live evidence on the Pi box showed one fire persisted at 15:41:16, a primary-path confirmation at 15:43:07, and the still-live row replaying and retiring only at 15:46:06. This change keys the late primary confirmation to the same immutable (schedule_id, schedule.last_run) pair used when the fire was persisted.

Impact

A late positive receipt on the primary path now completes the durable outbox lifecycle immediately. Ordinary confirmations with no persisted row behave as before, and parked dead letters are removed when the exact fire is conclusively confirmed.

Receipt waiting, matching, and acceptance semantics are unchanged.

Out of scope follow-up

The same live trace also contains a second primary-path confirmation for the same firing 28 seconds after the first. A read-only caller scan found one production chain only: _check_schedules_deliver_schedule_group_deliver_schedule. That narrows the separate defect to duplicate cohort dispatch or scheduler-instance/read-visibility behavior; this PR intentionally does not change that path.

Checks

  • focused late-confirm regressions: 3 passed
  • uv run pytest -q tests/test_scheduler.py: 124 passed
  • uv run pytest -q tests/test_agent_registry.py: 117 passed
  • uv run ruff check .: passed
  • git diff --check: passed

Refs #991

🤖 Opened by Murzik

@olegbrok
olegbrok marked this pull request as ready for review August 3, 2026 22:53

@olegbrok olegbrok left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified independently at a4e94dd in a clean worktree: 241 passed (scheduler 124 + registry 117), ruff clean. Registry method is atomic under _rmw_lock, keyed on UNIQUE(schedule_id, fired_at), includes parked rows, logs only on an actual retire with row id; scheduler confirmed branch retires-or-falls-back with exactly one last_delivered stamp either way; fired_at key = schedule.last_run, same immutable value _record_schedule_undelivered persists. Duplicate-cohort second confirm degrades to the fallback stamp harmlessly. Receipt semantics untouched. End-to-end regression proves no replay duplicate after a late primary confirm.

🤖 Reviewed by Barsik

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants