perf(runtime): keep cattle conversations open across terminal runs#400
Open
Yevanchen wants to merge 5 commits into
Open
perf(runtime): keep cattle conversations open across terminal runs#400Yevanchen wants to merge 5 commits into
Yevanchen wants to merge 5 commits into
Conversation
Production data (14d): 29/32 sessions get their first message within 2s of creation, so prewarm-on-create cannot beat the first-turn cold boot; the lever left is the follow-up turn, where the warm prepare path still paid ~4s because closeOnRunTerminal tore down the conversation session (and the driver process in it) after every run. Stop closing cattle conversations on run terminal and let a maintenance sweep close the ones idle past the cattle subject grace instead. The close path already arms the subject inactive deadline, so container reclamation rides the existing chain unchanged; conversations with an active run lease are never treated as idle, and session delete/cleanup paths still close conversations explicitly before the row cascades. Keeping the driver resident also pins the codex app-server thread for the session, which stabilizes the provider-side prompt_cache_key routing that per-run thread churn was scattering (langgenius/mosoo-agent-driver#82).
90s covered only a third of observed follow-up gaps; five minutes matches the read-think-type rhythm and the pet grace. With conversations now held open through the grace, this is the window in which a follow-up turn hits the resident driver instead of a container boot.
Queue names are account-global, so isolated environments deploy prefixed queues (e.g. mosoo-perf-stage-b-api-command). The consumer matched exact bare names only, so every batch on a prefixed queue was implicitly acked without processing — scheduled maintenance (and with it the runtime subject sweep) has never run on such stacks. Match by suffix; bare production names behave exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the V-C2 (warm-floor) direction of #387.
Problem
Production data (14d): 29/32 sessions receive their first message within 2s of creation — prewarm-on-create cannot beat the first-turn cold boot. The lever left is the follow-up turn, where the warm prepare path still paid ~4-5s because
closeOnRunTerminaltore down the conversation session — and the driver process in it — after every cattle run. Per-run driver respawn also minted a fresh codex thread every turn, which scattered OpenAI'sprompt_cache_keyrouting (codex keys it on its thread id; langgenius/mosoo-agent-driver#82).Change
closeOnRunTerminal: false— the conversation session and its resident driver survive between turns inside the idle grace.mosoo-perf-stage-b-api-command); the consumer matched bare names only and silently acked every batch unprocessed. Scheduled maintenance had therefore never run on prefixed stacks (it stopped on the perf stacks the moment they switched to prefixed queues on 07-20). Suffix match; bare production names behave exactly as before.Evidence (paired staging gate, openai instrument, stage-a=main
7aa09730vs stage-b=this branch, same container image)4 batches × 8 pairs — 32/32 pairs faster:
prepare_runwarm median: 5.0–8.6s → ~1.0s (990–1,053ms across batches); branch-side batch medians vary ±50ms while main wobbles ±2s (driver-respawn lottery removed).cache_read= [10112, 12160, 12160, 12160] (4/4 near-full) vs [0, 0, 12160, 0] on main under the identical experiment — resident driver ⇒ stable codex thread ⇒ stableprompt_cache_key.Cost ceiling: one extra ≤5min of container residency per session after its last run (the container already lives through the old 90s grace; the delta is grace length only).
Verification: api tc green, 1,053 api tests pass (incl. new sweep filter test), fmt/lint green.