Skip to content

fix(unplugin): per-delivery watch-input derivation rescans the whole reference graph, so graph-envelope builds never complete #1007

Description

@samchon

Problem

A Vite production build that serves modules from @ttsc/unplugin's project transform cache re-derives every module's watch-input list from scratch on every cache hit. The derivation rebuilds the full reference-graph index — once in selectReachableEdges and again in selectReachableSources (via selectResolutionCandidateInputs) — and computes a filesystem identity key for every path it touches. With a graph-bearing transform envelope, the cost of one module delivery is O(edges); a build that delivers M modules pays O(M × edges) identity computations.

On macOS each pathIdentityKey call additionally performs real syscalls (fs.existsSync + fs.realpathSync.native) even when its case-sensitivity verdict is already cached (packages/unplugin/src/core/transform.ts:1604-1633); the win32/linux branches are pure JS. The per-delivery graph rescan therefore becomes a per-delivery syscall storm on macOS, and real application builds effectively never finish.

This is the residual of #970. PR #980 removed the per-module full-project revalidation, but per-delivery watch-input derivation keeps the same O(M × graph) scaling. The version boundary the reporter verified — typia 13.1.1 fast, typia 13.2.0 stalled, both on @ttsc/unplugin 0.21.0 — is the envelope shape: typia ≥ 13.1.19 stamps graph + per-file dependencies (samchon/typia#2100, samchon/typia#2140), and 13.1.1 stamps neither. With no graph, the derivation selectors return early and the per-delivery cost vanishes.

Expected: one transform generation pays graph-index construction once; each module delivery is bounded by that module's own reachability closure, not by the whole graph.

Evidence

Reproduction (this repository, experimental/unplugin-perf, extended with a graph-envelope scenario that drives the real Rollup adapter over a synthetic project; process.platform is flipped to darwin after the compile so the macOS probe volume is observable on any host):

modules graph edges whole-project compiles fs probes probes / module delivery wall time (Windows)
100 2,599 1 665,432 6,721 75.8 s
100 5,099 1 1,195,132 12,072 87.6 s
100 10,099 1 2,254,531 22,773 95.2 s

probes/module ≈ 2.2 × edges, on pure cache hits with exactly one compile. A real react-router-scale program (edges in the tens of thousands across node_modules .d.ts, ~2k delivered modules) extrapolates to hundreds of millions of syscalls — hours for a build the reporter kills after minutes, with the reporter's exact CTRL-T signature (system CPU ≈ 2× user CPU, i.e. syscall-bound). The same run with a graph-free envelope (the typia 13.1.1 shape) derives nothing and shows no probes.

Root causes, all in packages/unplugin/src/core/transform.ts:

  1. selectReachableEdges (transform.ts:540-578) and selectReachableSources (transform.ts:585-626) each rebuild the complete edge index — O(edges) pathIdentityKey calls — per module delivery; selectResolutionCandidateInputs (transform.ts:467-497) then rescans every candidate source per delivery.
  2. pathIdentityKey (transform.ts:1597-1633) costs existsSync + realpathSync.native per call on darwin; only the case-verdict toggle is memoized, not the probe.
  3. declaresFile (transform.ts:686-700) linearly scans volatile / dependenciesComplete with two identity computations per member per delivery (isVolatileFile runs at least twice per delivery).
  4. selectTransformedSource (transform.ts:1482-1491) and selectFileDependencies (transform.ts:722-732) fall back to O(map) identity scans per delivery whenever the fast project-relative key misses (out-of-root keys — monorepo layouts).

All hosts funnel through transformTtsc, so every adapter (Vite, Rollup, webpack, Rspack, esbuild, Farm, Turbopack, Bun, Metro) carries the defect; macOS amplifies it into the reported stall.

Consequence surface

  • Who: any bundler user whose plugin stamps a graph envelope — typia ≥ 13.1.19 (including 13.2.0) projects are the entire known population. Worst on macOS; large Windows/Linux builds still pay the O(M × edges) pure-JS derivation.
  • Where: watch-input derivation runs identically in one-shot production builds, build --watch, and dev, so the stall hits all three; dev/serve additionally keeps the deliberate per-delivery complete-input validation from fix: harden transform caches and playground package boundaries #980 (measured separately: linear, not quadratic — out of scope here).
  • Freshness invariant: the derived watch lists are per-build registrations bundlers consume for invalidation; the fix must preserve their exact content and order per module (pinned by existing tests such as test_transformttsc_notifies_watch_files_on_cached_transforms and the transform-graph suite).
  • Memoization boundary: a memo may live at most one generation. Every derivable path is a recorded external/project input, so persistent-mode validation already proves those paths unchanged on every non-build-scoped hit; a change invalidates the generation and its memo with it.

Approach

Owner: packages/unplugin/src/core/transform.ts. Introduce per-envelope derivation state (a WeakMap keyed by the compiler result object, dying with the generation):

  • build the direct-edge index, source spellings, candidate entries, and the volatile / dependenciesComplete identity sets once per envelope;
  • memoize pathIdentityKey results inside that state;
  • memoize the final derived watch-input list per file;
  • give selectTransformedSource and selectFileDependencies first-match identity indexes built once per envelope so their fallback scans disappear.

The invariant is: per delivery, work is O(reach of that module) with zero filesystem probes beyond the first derivation; per generation, O(edges) exactly once. Derived lists must remain byte-identical to today's derivation (same union, same order, same dedup, same exclusions of the module itself and the disposed temporary tsconfig).

Acceptance and verification

  • experimental/unplugin-perf Scenario C (already committed with this reproduction) gates probes/module ≤ 64 under the simulated-darwin probe counter at graph sizes 2.6k/5.1k/10.1k edges — it fails with the numbers above pre-fix and must pass post-fix, with pluginRuns == 1 preserved.
  • New unit coverage in tests/test-unplugin: cache-hit deliveries of sibling modules from one graph-bearing generation must not re-probe the filesystem per delivery (probe counter + simulated darwin), while the derived watch list per module stays exactly correct.
  • The existing transform-graph, transform-dependencies, transform-complete, and project-cache suites must pass unmodified (derivation parity).
  • Negative/boundary: envelopes without graph, malformed graph members, dependenciesComplete narrowing, volatile interplay, cycles pointing back at the module, missing candidate paths.
  • Broader: pnpm --filter @ttsc/unplugin build, the unplugin feature tests, and the PR's ordinary CI.

Coordination

  • Residual of upgrading from 0.19 to 0.20 build never completes #970 (closed; reporter confirmed 0.21.0 + typia 13.2.0 still stalls). This issue is the remaining root cause; a comment cross-links it there.
  • Unrelated to the v0.22.0 release run: that run completed successfully in its historical duration (evidence gathered during discovery; no release-pipeline defect).
  • The Vite serve-mode complete-validation cost with large external input sets is deliberate (fix: harden transform caches and playground package boundaries #980 freshness contract) and linear; quantified during discovery and deliberately not changed here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions