fix: contexts read binary-detection + transient-disconnect worker health (6003e3bb, ab8a68f0)

[federicodeponte]

Summary

Fixes two bugs surfaced by live testing of an agent-mode worker: feedback tickets 6003e3bb (CLI contexts read) and ab8a68f0 (transient-disconnect worker health + agent-mode latency). Two commits, one per ticket. Backend changes are in apps/api (in-boundary — same floom monorepo); rollout of the API + any env flags is a separate ops step via the workeros-cloud pinned engine submodule, out of this repo.

Ticket 6003e3bb — floom contexts read printed "Binary file." for UTF-8 JSON/text

Root cause: contextsReadCommand (apps/mcp/src/commands/contexts.ts) called client.requestJson() expecting a JSON wrapper {content, is_binary, ...}, but the API endpoint GET /contexts/{name}/files/{path} (apps/api/routers/contexts.py::get_context_file) returns the raw file bytes (served inline with the mime-type for text, Content-Disposition: attachment for binary/active-markup). parseResponse then JSON-parses the body — a .json file becomes a parsed object with no top-level content string, other text becomes {text} — so typeof file.content === "string" was always false and every text file fell through to the "Binary file." branch. This was broken for all text files (json, md, yaml, txt), not just JSON.

Fix: contextsReadCommand now mirrors the MCP server's already-correct readContextFile (apps/mcp/src/server.ts): fetch the context detail (GET /contexts/{name}) for the server-computed is_binary metadata (the single source of truth — is_binary_file returns False for all text including active-markup); if truly binary, show the notice + download URL; otherwise fetch the raw bytes via a new requestText() client helper (apps/mcp/src/lib/api.ts) and print the UTF-8 text. A missing file now returns a clear non-zero error.

Files: apps/mcp/src/commands/contexts.ts, apps/mcp/src/lib/api.ts, apps/mcp/test/contexts-read.test.js (new regression tests: text JSON prints content, markdown prints content, genuinely-binary shows the notice and does not dump bytes, missing file errors).

Tests: npm run build clean; npm test 174/174 pass (independently re-run, includes the 3 new tests).

Ticket ab8a68f0 — transient "Server disconnected" left the worker in needs_attention; agent-mode latency

(a) Transient disconnect must not flip worker health (fixed)

Root cause: two issues. (1) Transport "Server disconnected" errors were bucketed into the broad catch-all agent_runtime_error in agent_driver.py, conflating pure infra blips with genuine agent/worker bugs. (2) _resolve_worker_status (apps/api/services/worker_serialize.py) downgraded HEALTHY -> NEEDS_ATTENTION whenever the last run FAILED, with no exemption for transient/retryable infra failures that are being retried or later succeed — so a single disconnect flipped the worker to needs_attention even though the agent had already reached its correct outcome.

Fix:

• New dedicated retryable-infra classification for transport disconnects (agent_runtime_disconnected): agent_driver.py now walks the exception cause chain and recognizes httpx/httpcore/openai/litellm transport drops (RemoteProtocolError, ConnectError, ReadError, "server disconnected"/"connection reset"/…) before the generic catch-all. Added to the transient retry set and to run_metrics as category network.
• Extracted the retry-classification sets into a shared apps/api/services/retry_classification.py (single source of truth; run_service.py re-imports under its existing private names — no behavior change there). DRY.
• _resolve_worker_status now takes last_run_error_code and does not downgrade to NEEDS_ATTENTION when the last failure is a transient-infra code. Threaded through both call sites (list path via worker_listing.py, detail path via _build_worker_detail). Genuine failures (permanent codes, schema/validation, missing secret/connection, real agent bugs) still downgrade exactly as before.
• Auto-clear: the detail path resolves from the newest run (recent_runs[0]), so a successful retry clears health naturally; while the retry is queued, the transient parent is exempt.
• overview.py failure-cluster derivation now skips a lone (failure_count == 1) transient-infra failure so a single blip is not surfaced as a worker-health problem; genuine clusters still surface.

Files: apps/api/runner_sandbox/agent_driver.py, apps/api/run_service.py, apps/api/services/retry_classification.py (new), apps/api/services/run_metrics.py, apps/api/services/worker_serialize.py, apps/api/routers/worker_listing.py, apps/api/routers/overview.py, apps/api/tests/test_worker_health_transient_disconnect.py (new, 8 tests).

Tests: py_compile clean on all 8 changed Python files; the new test_worker_health_transient_disconnect.py 8/8 pass (independently re-run). test_monitoring_apis.py 53 pass / 2 fail and test_1080_...overview pass — the 2 failures are ModuleNotFoundError: croniter/agents, a pre-existing environment dependency gap that fails identically on clean main (verified in a scratch worktree), not caused by this change. This dev host has no apps/api/venv, so full-suite runs are gated by those missing optional deps.

(b) Agent-mode latency (~3-5 min for trivial decisions) — investigation + low-risk fix + proposal

Investigation (grounded in code): agent-mode runs the LLM/tool loop in AgentDriver._run_agent_inner and do not enter the script driver's sandbox acquisition (E2BSandboxDriver._run_in_sandbox). An E2B sandbox is created only when the agent invokes run_command, via AgentDriver._run_command_e2b. So a true no-op run ("found 0 inbound, decided not to send") already avoids sandbox cold-start entirely — the dominant latency for trivial decisions is the model loop, MCP connection, and remote tool calls, not sandbox spin-up. (Conclusion from code-path analysis; production timings were not available on this host.)

Implemented (low-risk, gated off by default): wired the agent-mode run_command E2B path through the existing warm sandbox pool (_warm_pool_lease/_warm_pool_return), which previously only benefited the script driver. It is gated behind WORKEROS_E2B_WARM_POOL_ENABLED (default off, so production default behavior is unchanged), uses a distinct agent-command: pool key namespace (agent command sandboxes have a different directory lifecycle than script sandboxes and must not share entries), and adds _reset_agent_command_sandbox to clear per-call files before a sandbox is returned to the pool. Verified all imported warm-pool helpers exist with matching signatures.

Proposed (not implemented — needs ops + measurement):

• Enable and tune the warm pool at the deploy layer: WORKEROS_E2B_WARM_POOL_ENABLED=1, start WORKEROS_E2B_WARM_POOL_SIZE_PER_KEY=1, keep WORKEROS_E2B_WARM_POOL_MAX_AGE_SECONDS=900 (consider 300-900 after measuring utilization/cost); raise size to 2 only for concurrent same-key command workloads. Rollout is via the workeros-cloud engine submodule + deployed API config, outside this repo.
• Dry / read-only decision fast path: add an opt-in decision phase near _run_agent_inner before the full agent loop that exposes only tools with verified read-only metadata (withholding run_command and all side-effecting MCP/connection tools) and returns immediately when the structured decision is "no action required"; if action is required, pass the decision transcript into the existing full loop. Launch in shadow mode first and measure context-staging, MCP-connect, model-call, tool-call, and E2B timings separately. Primary risks: incorrect side-effect classification, false no-op decisions, missing read-only metadata, and adding a second model call to actionable runs.

🤖 Generated with Claude Code