Skip to content

Recover approval-gate backlogs - #999

Merged
olegbrok merged 1 commit into
mainfrom
agent/approval-gate-recovery
Aug 4, 2026
Merged

Recover approval-gate backlogs#999
olegbrok merged 1 commit into
mainfrom
agent/approval-gate-recovery

Conversation

@bradbrok

@bradbrok bradbrok commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes the systemic approval-gate black-hole described in #998 across the daemon registry, API, and broker.

  • re-prompts aged pending approvals at 24h and 72h, with durable bounded counters and no new inbound message required
  • exposes fleet-wide and per-agent pending/undelivered backlog diagnostics
  • settles approval-request state on every registry approval transition and continuously flushes approved-but-stranded rows, including startup/migration leftovers
  • sends an immediate loud alert when a pending chat holds a message from an already-approved principal of the same agent

Implementation

  • adds durable aging_reprompt_count and high_signal_alerted_at migration columns
  • mirrors the incident SQL oracle by grouping undelivered pending_messages on (agent_name, chat_id) and joining both gate status and sender approval status
  • adds GET /system/approval-backlog, agent health checks.approval_backlog, and backlog detail on the pending-messages response
  • runs approved-backlog reconciliation at startup before pollers and continuously in the approval maintenance loop
  • serializes each approval flush and checkpoints a row only after the streaming handoff succeeds, so offline startup reconciliation cannot silently mark a dropped message delivered
  • preserves the existing transport retry and new-message re-notification behavior

Validation

  • focused approval/registry/broker/grandfather suite: 207 passed
  • repo-wide Ruff: passed
  • frontend i18n extraction, tests, and production build: passed
  • full Python suite: 4,607 passed / 3 skipped; two unrelated failures inherited live-host PINKY_DREAM_TRANSPORT, PINKY_AUTH_DENY_DEFAULT, and PINKY_SHARED_MCP settings
  • both environment-contaminated failures pass when rerun with those host overrides removed (2 passed)

No release or deploy is included; this rides the next natural cut.

Closes #998

@olegbrok
olegbrok marked this pull request as ready for review August 4, 2026 22:25

@olegbrok olegbrok left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed + verified: all four #998 fixes implemented with the right discipline — aging re-prompts bounded (24h/72h) and counted separately from transport retries; owner-visible backlog surface (/system/approval-backlog + per-agent health with needs_attention escalation for stranded/high-signal states); gate-state transitions synced at the registry layer for ANY caller plus reconcile_approved_pending_messages at startup and continuously (this morning's Dmitri/Olga black-holes would have self-healed); high-signal alert pages once durably with principal ids. 49 focused tests pass in my worktree at final SHA 94c34f3; tests map 1:1 to fixes plus the failure edges (failed handoff keeps rows undelivered). CI green on the final SHA across all matrices. NOT released today — tonight's bounce validates 26.08.006 unchanged; this rides the next natural cut with #997.

🤖 Reviewed by Barsik (murzik authored — codex lane)

@olegbrok
olegbrok merged commit 8d0149b into main Aug 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Approval gate silently black-holes group chats for weeks — needs re-prompt sweep + undelivered-backlog visibility

2 participants