Skip to content

perf: only scan the run store when pruning dev-runtime snapshots#735

Open
ryanda9910 wants to merge 1 commit into
vercel:mainfrom
ryanda9910:perf/prune-scan-run-store-only
Open

perf: only scan the run store when pruning dev-runtime snapshots#735
ryanda9910 wants to merge 1 commit into
vercel:mainfrom
ryanda9910:perf/prune-scan-run-store-only

Conversation

@ryanda9910

Copy link
Copy Markdown

Addresses the boot-time half of #474.

The problem

pruneDevelopmentRuntimeArtifactsSnapshots protects snapshots that a still-running turn depends on. To find them it walks all of .workflow-data and readFiles every file under 1MB:

const source = await readFile(path, 'utf8');
if (!shouldScanWorkflowDataSource(source)) return;   // ← the filter runs *after* the read

So boot cost scales with everything the local workflow world has ever written, not with live work. The store in #474 had reached ~10,900 files, and a full read of it ran on every eve dev boot (and on every authored-source change, via dev-authored-source-watcher).

The fix

A snapshot reference only reaches disk through a runinput.serializedContext["eve.bundle"]. The other stores of @workflow/world-local (events, steps, hooks, streams, waits, .locks) never carry one, and they are almost the entire store: the reported breakdown was 8,466 events / 1,460 steps / 459 hooks against 500 runs — runs are ~5% of the files.

Those directories are now skipped by name, before any read happens. Runs are still scanned exactly as before, so nothing a live run depends on loses its protection.

Measured

Rebuilt a store at the reported proportions (10,885 files) and ran the collector both ways:

files read time
before 10,885 ~253 ms
after 500 ~12 ms

500 reads = exactly the run count, i.e. every run is still visited.

Test

Added an integration test that pins both halves:

  • a snapshot referenced by a non-terminal run survives the prune (no regression in protection)
  • a snapshot path planted in the event store does not survive — which is what proves the directory is actually skipped

Verified by mutation: removing the skip makes exactly that new test fail, and the 13 pre-existing tests pass either way.

pnpm vitest run --config vitest.integration.config.ts src/internal/nitro/dev-runtime-artifacts.integration.test.ts14 passed. oxlint and tsc clean. Patch changeset included.

Not covered here

#474 also reports per-operation directory scans inside @workflow/core's local queue, and runs stranded in running forever by killed dev servers. Both live outside this file — happy to take either as a follow-up if you'd like them split that way.

The snapshot pruner walked all of `.workflow-data` and read every file under 1MB
looking for snapshot references, so `eve dev` boot cost scaled with everything the
local workflow world had ever written instead of with live work. A store reported in
vercel#474 had grown to ~10,900 files, and a full read of it ran on every boot.

Only runs can reference a snapshot — the path reaches disk through a run's
`input.serializedContext["eve.bundle"]`. The event, step, hook, stream, wait and lock
stores of `@workflow/world-local` never carry one, and they are the bulk of the files
(the reported store was ~78% events, ~13% steps, ~4% hooks against ~5% runs). They are
now skipped by name before any read happens.

Measured on a store rebuilt to those proportions (10,885 files): 10,885 file reads and
~253ms before, 500 reads and ~12ms after — every run is still scanned, so nothing a
live run depends on loses its protection.

Adds a test that pins both halves of that: a snapshot referenced by a non-terminal run
survives the prune, while a snapshot path planted in the event store does not — which
fails if the skip is removed.
@vercel

vercel Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

@ryanda9910 is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant