Live incident (TOD fleet, 2026-08-04, Brad report tg 16097): two group chats sat in pending approval for 2.5+ weeks, silently dropping every message:
- flow / group -5162791156 (Olga's group): pending since Jul 20, 2 undelivered — reported as 'Flow misses Olga's messages'
- onesie / group -5097410845 (Dmitri's travel group): pending since Jul 17, 15 undelivered messages from the fleet's main user, including standing instructions — he asked 'Are you here?' and concluded the bot was broken
Both fixed manually via POST /agents/{name}/approved-users (which correctly flushed the backlog — pending_delivered 2 and 15). Also found: 3 undelivered rows for an already-approved user (mia/920425102, June greetings) — approval that arrives via a path other than the approve endpoint appears to strand pre-approval backlog forever.
The failure shape (same family as the earlier upgrade-orphaned-group finding): a group lands in pending, the owner-approval prompt is missed or lost (or never sent after an upgrade re-registration), and from then on the gate is a silent black hole — no re-prompt, no aging alarm, no owner-visible count. Every message is swallowed with zero telemetry. The sender's experience is an agent that ignores them; the owner's experience is a vague 'missed messages' report weeks later.
Proposed fixes (any subset):
- Aging re-prompt: pending approval older than N hours with ≥1 undelivered message → re-send the owner approval prompt (bounded retries, e.g. 24h/72h).
- Owner-visible backlog line: daily/heartbeat surface of
pending approvals w/ undelivered counts per fleet (the diagnostic that found this was one SQL query over approved_users + pending_messages — trivial to expose).
- Approved-but-stranded flush: on any approval-state transition to approved (any path, incl. migrations), flush undelivered pending_messages; or a startup reconcile pass.
- Alert loudly when a pending chat accumulates messages from an ALREADY-APPROVED principal of the same agent (Dmitri was onesie's own primary user — highest-signal case, should never silently queue).
Diagnostic used (works fleet-wide): SELECT agent_name, chat_id, count(*) FROM pending_messages WHERE delivered=0 GROUP BY 1,2 joined against approved_users.status != 'approved'.
🤖 Opened by Barsik
Live incident (TOD fleet, 2026-08-04, Brad report tg 16097): two group chats sat in
pendingapproval for 2.5+ weeks, silently dropping every message:Both fixed manually via
POST /agents/{name}/approved-users(which correctly flushed the backlog —pending_delivered2 and 15). Also found: 3 undelivered rows for an already-approved user (mia/920425102, June greetings) — approval that arrives via a path other than the approve endpoint appears to strand pre-approval backlog forever.The failure shape (same family as the earlier upgrade-orphaned-group finding): a group lands in
pending, the owner-approval prompt is missed or lost (or never sent after an upgrade re-registration), and from then on the gate is a silent black hole — no re-prompt, no aging alarm, no owner-visible count. Every message is swallowed with zero telemetry. The sender's experience is an agent that ignores them; the owner's experience is a vague 'missed messages' report weeks later.Proposed fixes (any subset):
pending approvals w/ undelivered countsper fleet (the diagnostic that found this was one SQL query over approved_users + pending_messages — trivial to expose).Diagnostic used (works fleet-wide):
SELECT agent_name, chat_id, count(*) FROM pending_messages WHERE delivered=0 GROUP BY 1,2joined againstapproved_users.status != 'approved'.🤖 Opened by Barsik