fix(pipeline,usenet): open fresh DB connections per transition/claim#190
Merged
Conversation
Same bug class as transcribe-item (PR #189): rt911-db sets idle_session_timeout=10min on the video_grabber database, but these flows held one connection across stages that routinely outlive it — a multi-GB IA download, an encode, a threading build, or a dispatcher's blocking run_deployment. The next DB touch then hit a dead socket, and the except-path reused the same poisoned connection, so jobs stuck in-flight for the orphan supervisor to reclaim in a loop. - transition_job / transition_usenet_job now open their own short-lived connection (signature drops the held-db param) - process-item does resolve_job and the wasabi_key UPDATE on fresh connections after the long download/encode stages - both dispatchers claim each job on a fresh connection instead of holding one across the blocking run_deployment - get_job / get_usenet_job now close their connection (they leaked one per call — the likely source of the 2026-07-08 connection exhaustion) - scan/build-channel/requeue wrap their connection in a context manager Regression tests simulate idle_session_timeout killing every open connection during each long stage. Co-Authored-By: Claude Fable 5 <[email protected]>
|
✅ Playwright E2E — 1 passed · 0 failed · 0 flaky · 0 skipped |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Follow-up to #189, which fixed
transcribe-item. The video pipeline and usenet flows have the same latent bug class:rt911-dbsetsidle_session_timeout=10minon thevideo_grabberdatabase, but these flows hold a single DB connection across stages that routinely outlive it:process-item: a multi-GB IA download or an encode between transitionsprocess-usenet-item: mbox download / threading build between transitionsdispatch-discovered/dispatch-usenet: a blockingrun_deploymentfor the whole job between claim queries — guaranteed to exceed 10 minutes for any real job, killing the dispatcher loop after its first dispatchget_job/get_usenet_job: opened a connection and never closed it — one leaked connection per job processed, the likely source of the 2026-07-08 connection-pool exhaustion (99/100) that motivated the timeout in the first placeFix
Every DB touch opens its own fresh, short-lived connection:
transition_job(job_id, ...)/transition_usenet_job(job_id, ...)open their own connection (held-db param dropped; live-Postgres integration tests updated to exercise the real path via the test engine)process-itemrunsresolve_joband thewasabi_keyUPDATE on fresh connections after the long stagesget_job/get_usenet_jobclose their connection via context managerscan-collections/scan-usenet/build-channel/requeue-pending-reviewwrap connections in context managers (no more leaks on flow exit)Tests
New regression tests simulate the production failure: every connection opened before a long stage (download, encode, threading,
run_deployment) is killed mid-stage, and the flow must still record its terminal stage —complete/processedon success,failedon error — on fresh connections. Dispatcher tests verify each claim survives the previous iteration's blocking run.pytest tests/— 309 passed, 8 skipped (up from 303)ruff check video_grabber/ tests/— clean🤖 Generated with Claude Code