perf: only scan the run store when pruning dev-runtime snapshots#735
Open
ryanda9910 wants to merge 1 commit into
Open
perf: only scan the run store when pruning dev-runtime snapshots#735ryanda9910 wants to merge 1 commit into
ryanda9910 wants to merge 1 commit into
Conversation
The snapshot pruner walked all of `.workflow-data` and read every file under 1MB looking for snapshot references, so `eve dev` boot cost scaled with everything the local workflow world had ever written instead of with live work. A store reported in vercel#474 had grown to ~10,900 files, and a full read of it ran on every boot. Only runs can reference a snapshot — the path reaches disk through a run's `input.serializedContext["eve.bundle"]`. The event, step, hook, stream, wait and lock stores of `@workflow/world-local` never carry one, and they are the bulk of the files (the reported store was ~78% events, ~13% steps, ~4% hooks against ~5% runs). They are now skipped by name before any read happens. Measured on a store rebuilt to those proportions (10,885 files): 10,885 file reads and ~253ms before, 500 reads and ~12ms after — every run is still scanned, so nothing a live run depends on loses its protection. Adds a test that pins both halves of that: a snapshot referenced by a non-terminal run survives the prune, while a snapshot path planted in the event store does not — which fails if the skip is removed.
Contributor
|
@ryanda9910 is attempting to deploy a commit to the Vercel Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the boot-time half of #474.
The problem
pruneDevelopmentRuntimeArtifactsSnapshotsprotects snapshots that a still-running turn depends on. To find them it walks all of.workflow-dataandreadFiles every file under 1MB:So boot cost scales with everything the local workflow world has ever written, not with live work. The store in #474 had reached ~10,900 files, and a full read of it ran on every
eve devboot (and on every authored-source change, viadev-authored-source-watcher).The fix
A snapshot reference only reaches disk through a run —
input.serializedContext["eve.bundle"]. The other stores of@workflow/world-local(events,steps,hooks,streams,waits,.locks) never carry one, and they are almost the entire store: the reported breakdown was 8,466 events / 1,460 steps / 459 hooks against 500 runs — runs are ~5% of the files.Those directories are now skipped by name, before any read happens. Runs are still scanned exactly as before, so nothing a live run depends on loses its protection.
Measured
Rebuilt a store at the reported proportions (10,885 files) and ran the collector both ways:
500 reads = exactly the run count, i.e. every run is still visited.
Test
Added an integration test that pins both halves:
Verified by mutation: removing the skip makes exactly that new test fail, and the 13 pre-existing tests pass either way.
pnpm vitest run --config vitest.integration.config.ts src/internal/nitro/dev-runtime-artifacts.integration.test.ts→ 14 passed. oxlint andtscclean. Patch changeset included.Not covered here
#474 also reports per-operation directory scans inside
@workflow/core's local queue, and runs stranded inrunningforever by killed dev servers. Both live outside this file — happy to take either as a follow-up if you'd like them split that way.