Skip to content

refactor(runtime): collapse the sandbox error status into cold#404

Merged
Yevanchen merged 1 commit into
mainfrom
refactor/collapse-runtime-subject-error-state
Jul 25, 2026
Merged

refactor(runtime): collapse the sandbox error status into cold#404
Yevanchen merged 1 commit into
mainfrom
refactor/collapse-runtime-subject-error-state

Conversation

@Yevanchen

Copy link
Copy Markdown
Collaborator

Problem — a production death loop, from a contradiction in the state machine

A DeepSeek acp-fallback agent on try.mosoo.ai (sandbox 01KYC1ZB0J6C00G5WVQ6E0CZWK, pet) got stuck: run after run failed with runtime.provision_failed "filesystem prepare timed out after 15000ms", the sandbox row pinned at status=error, and restartDriver did nothing.

Root cause is the error status itself — a physical contradiction, "activation failed but still reclaimable":

  • markRuntimeSubjectActivationFailed set status=error without destroying the container DO (keep-alive left on).
  • error was kept out of the maintenance sweep (only active is swept) yet left claimable (isClaimableRuntimeSubjectStatus + RUNTIME_SUBJECT_CLAIMABLE_STATUSES included it), gated by canRecoverFromMountError + a RECOVERABLE_RUNTIME_SUBJECT_ERROR_CODES whitelist that contained runtime.subject_activation_failed.
  • so the next run reclaimed the same dead container DO (isCold = status==='cold' was false for error, skipping restore/rebuild), prepareFilesystem timed out again → error again. Permanent loop.

Seven patches existed only to prop up this contradiction (CLAIMABLE membership, isClaimable, the canRecover gate, the whitelist, the error→{active,backing_up,destroying} machine edges, the error-writing failure paths, the isCold special-case).

Fix — delete the state, don't add an eighth patch

A failed lifecycle step is physically unambiguous: the container DO is untrustworthy. So failure now destroys the container and returns the subject to cold (diagnostic kept in last_error); the next run cold-starts a fresh container and self-heals — no manual recreate, no retry/sweep/whitelist patch.

  • lifecycle machine drops the error state and the .fail event; failure is a runtime_subject.cold transition from any in-flight state. 6 states → 5, each physically meaningful: cold=no container, restoring=starting (claim lock), active=live, backing_up/destroying=maintenance locks.
  • SandboxStatus contract type drops error.
  • all seven patches deleted.
  • migration 0003 converges any existing error rows to cold.
  • DB CHECK keeps error as a dead-but-allowed value (commented) to avoid a full SQLite rebuild of the core sandbox table; the app can never produce it.

Production-log walk-through (why the real loop converges under the new machine)

Real events from sandbox 01KYC1ZB:

time UTC outcome
07:14:00 first cold start completed
07:17:42 additional_directories_dropped → session → end_turn (DeepSeek replied)
07:18:14 provision_failed "filesystem prepare timed out 15000ms"
07:24:09 same provision_failed; sandbox pinned status=error forever
  • OLD: 07:18 fail → error, container NOT destroyed → 07:24 reclaims the same wedged DO (whitelist allows it, isCold=false skips rebuild) → same timeout → error → ∞.
  • NEW: 07:18 fail → destroy the DO → cold → 07:24 is a normal cold start that gets a fresh container DO (Cloudflare reschedules, typically onto a healthy node) → active. The input that produced the loop no longer exists; the worst terminal state is cold, which is where normal runs begin. Convergent, not divergent.

Verification

  • Unit: runtime-subject-lifecycle.test.ts reproduces the exact loop — inject a prepareFilesystem timeout, assert the container is destroyed once and the subject returns to cold, then a second activation reaches active. Plus machine coverage for the new failure→cold edges. api tc + 1054 tests green.
  • Migration: applied to stage-b — sandbox rows went to 343 cold, 0 error (existing error rows converged); idempotent, data-only.
  • No-regression gate (stage-b = this branch, openai instrument, 8 pairs): 8/8 succeeded, prepare_run/ttft in normal range — the normal activate/run path is unaffected.

Migration safety

0003 is a data-only, idempotent UPDATE sandbox SET status='cold' WHERE status='error'. No schema/table rebuild. Blast radius is tiny (only rows stuck in the retired state — a handful in production). Rollback is a no-op: reverting the code leaves those rows cold, from which they cold-start normally.

Driver submodule unchanged (pinned at the production 020cfae7). API-only refactor.

The runtime subject lifecycle had six states; `error` was a contradiction —
"activation failed but still reclaimable". A failed prepareFilesystem marked
the subject `error` without destroying the container, and `error` was both
kept out of the maintenance sweep and left claimable, so the next run
reclaimed the same dead container DO and failed again — a permanent death
loop (observed in production on a DeepSeek acp-fallback agent: sandbox
01KYC1ZB…, run after run `runtime.provision_failed: filesystem prepare timed
out after 15000ms`).

Seven patches existed only to prop up this contradiction: `error` in
CLAIMABLE_STATUSES, in isClaimableRuntimeSubjectStatus, the
canRecoverFromMountError gate, the RECOVERABLE_RUNTIME_SUBJECT_ERROR_CODES
whitelist, the error→{active,backing_up,destroying} machine edges, the
error-writing failure paths, and the isCold special case.

Delete the state instead of adding an eighth patch. A failed lifecycle step
is physically unambiguous — the container DO is untrustworthy — so:
- activation failure now destroys the container and returns the subject to
  `cold` (diagnostic retained in last_error); the next run cold-starts a
  fresh container and self-heals. No manual recreate, no retry patch, no
  sweep patch.
- the lifecycle machine drops the `error` state and the `.fail` event;
  failure is a `runtime_subject.cold` transition from any in-flight state.
  Five states remain, each physically meaningful: cold=no container,
  restoring=starting (claim lock), active=live, backing_up/destroying=locks.
- the SandboxStatus contract type drops `error`.
- migration 0003 converges any existing `error` rows to `cold`.

The DB CHECK keeps `error` as a dead-but-allowed value (commented) to avoid a
full SQLite rebuild of the core sandbox table; the app can never produce it.

Tests reproduce the production death loop and prove convergence (container
destroyed once, subject returns to cold, second activation succeeds), plus
machine coverage for the new failure→cold transitions.
@Yevanchen
Yevanchen merged commit 7b8d13f into main Jul 25, 2026
10 checks passed
@Yevanchen
Yevanchen deleted the refactor/collapse-runtime-subject-error-state branch July 25, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant