VFS: cap concurrent scripts and enforce execution deadline

[henry-dowling]

Why

On 2026-07-09, one customer's agent fired about 20 concurrent VFS scripts. Each script retained a full filesystem model and the documents it read, pushing the 512 MB API instance into an OOM restart. The restart caused product-wide 502s and dropped every in-flight request.

The server still had no concurrency or elapsed-time guard. PR #779 proposed one, but its VFS route, service, and tests all conflicted with current main.

What

• Allow at most four VFS scripts per API process.
• Reject excess work immediately with 429 and Retry-After: 2; do not queue memory-heavy requests.
• Give nested ASGI reads only the time remaining in the 60-second script budget, and cancel a request that exceeds it.
• Keep the worker attached until it actually exits, then release capacity in finally on success or failure.
• Preserve current partial-grep scan budgets, search auditing, downloads, and path resolution.

This supersedes #779.

Testing

• Added a real overlapping-request test for the concurrency cap and 429 contract.
• Added failure-path coverage proving the slot is released after a crashed command.
• Added deadline coverage for both an expired command and a stalled nested request.
• pytest -q backend/tests/test_vfs.py --no-cov -k 'not grep_searches_connected_source_documents' — 17 passed, 1 deselected. The deselected existing test aborts this local Python process inside sentence-transformers/Torch during connected-source embedding setup; the existing partial-grep budget test passes.
• ruff check backend/ cli/
• ruff format --check backend/ cli/
• git diff --check

---

Note

Medium Risk
Changes resource limits on a hot agent-facing path; mis-tuned caps or timeouts could cause false 429/413s, but the design preserves existing auth, scan budgets, and audit behavior.

Overview
Adds process-level guards on the HTTP VFS endpoint so agent bursts cannot OOM the API: at most four concurrent scripts per process, with excess requests rejected immediately as 429 and Retry-After: 2 (no queuing).

Each script now has a 60-second wall-clock budget. Nested in-process ASGI reads wait only for the remaining time; overdue work is cancelled and surfaced as VfsBudgetExceeded (HTTP 413), same as the existing document-read cap. The concurrency slot is always released in finally, including after crashes.

Tests cover overlapping 429 behavior, slot release on failure, command-level timeout, and cancellation of a stalled nested request.

^{Reviewed by Cursor Bugbot for commit cc4e68c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[assert-app]

Review on Assert →

Merge candidate is ready!

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
stash-web	Ready	Preview	Aug 9, 2026 10:06pm