refactor(runtime): collapse the sandbox error status into cold#404
Merged
Conversation
The runtime subject lifecycle had six states; `error` was a contradiction —
"activation failed but still reclaimable". A failed prepareFilesystem marked
the subject `error` without destroying the container, and `error` was both
kept out of the maintenance sweep and left claimable, so the next run
reclaimed the same dead container DO and failed again — a permanent death
loop (observed in production on a DeepSeek acp-fallback agent: sandbox
01KYC1ZB…, run after run `runtime.provision_failed: filesystem prepare timed
out after 15000ms`).
Seven patches existed only to prop up this contradiction: `error` in
CLAIMABLE_STATUSES, in isClaimableRuntimeSubjectStatus, the
canRecoverFromMountError gate, the RECOVERABLE_RUNTIME_SUBJECT_ERROR_CODES
whitelist, the error→{active,backing_up,destroying} machine edges, the
error-writing failure paths, and the isCold special case.
Delete the state instead of adding an eighth patch. A failed lifecycle step
is physically unambiguous — the container DO is untrustworthy — so:
- activation failure now destroys the container and returns the subject to
`cold` (diagnostic retained in last_error); the next run cold-starts a
fresh container and self-heals. No manual recreate, no retry patch, no
sweep patch.
- the lifecycle machine drops the `error` state and the `.fail` event;
failure is a `runtime_subject.cold` transition from any in-flight state.
Five states remain, each physically meaningful: cold=no container,
restoring=starting (claim lock), active=live, backing_up/destroying=locks.
- the SandboxStatus contract type drops `error`.
- migration 0003 converges any existing `error` rows to `cold`.
The DB CHECK keeps `error` as a dead-but-allowed value (commented) to avoid a
full SQLite rebuild of the core sandbox table; the app can never produce it.
Tests reproduce the production death loop and prove convergence (container
destroyed once, subject returns to cold, second activation succeeds), plus
machine coverage for the new failure→cold transitions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem — a production death loop, from a contradiction in the state machine
A DeepSeek acp-fallback agent on try.mosoo.ai (sandbox
01KYC1ZB0J6C00G5WVQ6E0CZWK, pet) got stuck: run after run failed withruntime.provision_failed"filesystem prepare timed out after 15000ms", the sandbox row pinned atstatus=error, andrestartDriverdid nothing.Root cause is the
errorstatus itself — a physical contradiction, "activation failed but still reclaimable":markRuntimeSubjectActivationFailedsetstatus=errorwithout destroying the container DO (keep-alive left on).errorwas kept out of the maintenance sweep (onlyactiveis swept) yet left claimable (isClaimableRuntimeSubjectStatus+RUNTIME_SUBJECT_CLAIMABLE_STATUSESincluded it), gated bycanRecoverFromMountError+ aRECOVERABLE_RUNTIME_SUBJECT_ERROR_CODESwhitelist that containedruntime.subject_activation_failed.isCold = status==='cold'was false forerror, skipping restore/rebuild),prepareFilesystemtimed out again →erroragain. Permanent loop.Seven patches existed only to prop up this contradiction (CLAIMABLE membership, isClaimable, the canRecover gate, the whitelist, the
error→{active,backing_up,destroying}machine edges, the error-writing failure paths, the isCold special-case).Fix — delete the state, don't add an eighth patch
A failed lifecycle step is physically unambiguous: the container DO is untrustworthy. So failure now destroys the container and returns the subject to
cold(diagnostic kept inlast_error); the next run cold-starts a fresh container and self-heals — no manual recreate, no retry/sweep/whitelist patch.errorstate and the.failevent; failure is aruntime_subject.coldtransition from any in-flight state. 6 states → 5, each physically meaningful:cold=no container,restoring=starting (claim lock),active=live,backing_up/destroying=maintenance locks.SandboxStatuscontract type dropserror.0003converges any existingerrorrows tocold.erroras a dead-but-allowed value (commented) to avoid a full SQLite rebuild of the coresandboxtable; the app can never produce it.Production-log walk-through (why the real loop converges under the new machine)
Real events from sandbox
01KYC1ZB:completedadditional_directories_dropped→ session →end_turn(DeepSeek replied)provision_failed"filesystem prepare timed out 15000ms"provision_failed; sandbox pinnedstatus=errorforevererror, container NOT destroyed → 07:24 reclaims the same wedged DO (whitelist allows it, isCold=false skips rebuild) → same timeout →error→ ∞.cold→ 07:24 is a normal cold start that gets a fresh container DO (Cloudflare reschedules, typically onto a healthy node) →active. The input that produced the loop no longer exists; the worst terminal state iscold, which is where normal runs begin. Convergent, not divergent.Verification
runtime-subject-lifecycle.test.tsreproduces the exact loop — inject aprepareFilesystemtimeout, assert the container is destroyed once and the subject returns tocold, then a second activation reachesactive. Plus machine coverage for the new failure→cold edges. api tc + 1054 tests green.Migration safety
0003is a data-only, idempotentUPDATE sandbox SET status='cold' WHERE status='error'. No schema/table rebuild. Blast radius is tiny (only rows stuck in the retired state — a handful in production). Rollback is a no-op: reverting the code leaves those rowscold, from which they cold-start normally.Driver submodule unchanged (pinned at the production
020cfae7). API-only refactor.