feat: add export origin analysis by BridgeAR · Pull Request #229 · guybedford/es-module-lexer

BridgeAR · 2026-07-23T13:41:52Z

Stacked on #204: the leading perf commits are that work rebased onto the current v3; the export-analysis change is the feat commit and its perf follow-ups.

Supersedes #215 and #222.

Summary

Detached exports can precede their imports, so source-order tracking cannot classify them or preserve their module request. This adds a full-build tagged union for direct exports, named and namespace reexports, and bare star reexports. Reexports link to the existing imports array through fi.

The minimal build keeps its v2 output and compiles the analysis path out.

Why

Nothing about a binding is stored while parsing. A module that ends the parse with an unresolved export re-reads its import clauses once in the finalize pass, resolving names against a hash table of the pending exports; a module without one never builds the table and never reads a clause twice. That is the shape suggested in review, and it matches how real code is written: across 878 ESM files in two node_modules trees, 84.6% have imports and no clause export and pay nothing for the feature, while the 5.7% that do run the finalize walk carry a median of 2 import statements.

Three properties keep the common paths off the analysis:

The main loop scans an import clause for its closing brace rather than tokenizing specifiers, because only the finalize pass needs binding structure. This is why a module with imports and no clause export parses no slower than main.
A module with no import clause skips the table outright, and pending names are hashed at table insert rather than at the clause.
Table buckets are chained rather than open addressed, so one local exported under many names stays linear.

Analysis records live in the existing arena. The reservation is unchanged from main, and the arena grows only if a parse actually needs more.

Parsing those 878 real ESM files through parse() measures −1.1%, +0.6% and −1.1% against main over three runs — no measurable cost — and the 93 files that run the finalize walk land in the same band.

Synthetic shapes, median of five ABBA rounds against main on Node 24 / V8 13.6:

input	vs main
real-world bundles (angular, d3, rollup, magic-string)	+0.1% geometric mean, nothing above the ±4 round spread
400 imports plus direct exports	+1.2%
barrel of 400 `export … from`	+4.2%
120 imports, 400 exports, trailing clause export	+8.2%
400 detached reexports	+20.5%

The last two rows are the trade this design takes: they carry far more imports per clause export than real modules do, so they pay the clause re-read at a rate real code does not reach. Small modules carry a fixed cost of roughly 20 ns per parse — 187 → 207 ns at the Wasm boundary for import { a } from 'dep'; export const b = a; export { a as c };.

Artifact sizes: lib/lexer.wasm 11,800 → 14,991 bytes and dist/lexer.asm.js 28,179 → 35,585. The minimal builds grow by 117 and 248 bytes.

Fixes: #76
Refs: nodejs/import-in-the-middle#259

* perf(lexer): skip identifier and number runs On the 3,057 KiB sample corpus, Wasm drops from 15,356.31 to 13,010.46 us per sweep on Node 18.20.8 (-15.28%) and from 11,724.34 to 9,856.69 us on Node 24.18.0 (-15.93%). The asm.js build drops from 15,934.34 to 12,945.57 us on Node 24.18.0 (-18.76%). The fast path adds 379 raw / 105 gzip bytes to the full Wasm binary. Measurements used 1,500 warmup sweeps followed by 9 interleaved trials of 50 sweeps, dropping the best and worst trials.

lexer.js reimplemented the WebAssembly lexer in hand-written JS, kept in sync by hand, but was unreachable through the package `exports` (only `.` and `./js` are exposed - the wasm and asm.js builds) and `chomp test` never ran it, so it could drift silently. The asm.js build already covers the no-WebAssembly case, and the removed `if (!js)` test guards now run unconditionally, matching their existing behavior under the wasm and asm suites.

guybedford · 2026-07-26T05:52:34Z

I've pushed two maintainer commits to this branch:

ci: CI was only wired to main, which is why no checks were running here — the workflow now also triggers for v3 pushes and PRs, and the build is green on this branch (the full emsdk/fastcomp build, test, footprint and bench steps all ran).
fix: the optional strip patterns in build/combine-asm.mjs now carry anchors — a pattern miss only passes silently when the function is genuinely absent from the emcc output after EMSCRIPTEN_END_FUNCS, while a still-present export with a stale regex throws. Also dropped ess/et/eii/eit from the wasm type in src/lexer.ts (they are only exported by the asm.js builds now) and noted es/ee/els/ele as minimal-build-only.

I also updated the PR description — the "binding capture is gated" paragraph described a previous revision, so it now reflects the unconditional inline collection.

guybedford · 2026-07-26T05:52:52Z

On the allocation design: I don't think the ImportBinding records need to exist at all. The Import record already carries the statement range, so binding capture can be deferred to the end of the parse and gated on pending_export_count != 0: walk the import list, re-scan each static import clause bounded by the stored statement ranges, and resolve each binding against the pending-export table as it is read — streamed rather than stored.

That removes the ImportBinding struct, its list and its arena traffic, and takes binding collection off the hot path entirely, which should recover the 5.08 -> 5.28us regression on plain imports — modules without detached exports would pay nothing. Modules with them pay one bounded second read of just the clause text. I realize an earlier revision had a prescan that was dropped for cost — this is a different shape: no speculative forward scanning, only a deferred re-read of import clauses, and only when a detached export actually exists.

Beyond that, I want to keep the v2 allocation model as a hard constraint: static allocation with known limits and a known memory layout — no dynamic growth, no bail-and-retry. So ensureAnalysisCapacity, the wasm memory_grow path, sal() and the asm.js -1 retry should all go. With ImportBinding gone the worst case becomes a knowable constant: a 32-byte Export plus an 8-byte bucket slot per 2-char clause name is ~20 bytes per source char, with imports well under that. The JS wrappers can reserve source.length * K for the analysis region up front (wasm grows once before the parse; asm.js folds it into memBound) and the whole growth machinery disappears. Note v2's implicit 2 bytes/char budget is already violable today by dense export clauses (~12 bytes/char), silently running into the asm.js stack region — sizing by the true constant fixes that too. K can be shaved if the constant feels heavy (u32 bucket indices instead of pointers, tighter Export packing).

The remaining per-record cost then looks right to me: +8 bytes on Export for classification, and the transient bucket table at 8 bytes per pending export (which could also fall back to a linear scan below a small threshold if we want to skip the table for typical modules).

It does require the clause reader to be callable outside the main loop (save/restore pos), but the collect functions are essentially already that.

A dynamic import whose entire argument is a template literal with substitutions returned n: undefined, so a consumer resolving the specifier (a bundler or glob importer) had nothing to work with. It now reports the static skeleton as a glob with each ${...} collapsed to a single "*": import(`./locales/${x}.js`) yields "./locales/*.js". Only a lone template literal qualifies. A template concatenated with anything else (import(`a` + b)) has no static skeleton and still returns undefined. Fixes: guybedford#137

The glob walker that builds a dynamic-import template skeleton skips over strings, nested templates, and comments inside each ${...} substitution, but it cannot tell a regex literal from division without the main parser's token context. A regex carrying a "}" closed the substitution early and emitted a wrong specifier instead of bailing: import(`a${ /x}y/g }b`) reported "a*y/g }b" rather than undefined. skipInterpolation now flags a bare "/" (one that does not open a // or /* comment) and the three decoders drop n to undefined rather than guess. This over-bails the rare division case (import(`a${ b/c }d`)), which is acceptable: a missing glob is recoverable, a wrong one is not. Fixes: guybedford#137

The interpolated-template glob walked each ${...} substitution with a hand-rolled scanner in all three decoders. None of them could tell a regex literal from division without token context, so a regex carrying a "}" closed the substitution early: import(`a${ /x}y/g }b`) reported "a*y/g }b". The prior fix bailed to undefined on any bare "/", which also dropped legitimate division (import(`a${ b/c }d`)). The parser already resolves regex vs division for the whole source and descends into ${ ... } for nested-import detection, so it now records each top-level substitution's end on the dynamic import (struct TemplateSpan). The decoders splice a "*" per span and jump the body, dropping their skipInterpolation / skipQuoted / skipComment scanners and the interpolationError bail. Both ambiguous cases now resolve correctly: "a*b" and "a*d". Fixes: guybedford#137

…ects ## Summary Interpolated template glob tracking leaked into minimal builds and diverged between decoders. Missing parser spans could evaluate a substitution, reload an exhausted span list into a loop, or lose the outer glob around a nested import. ## Why Keep span tracking behind LEXER_MIN, use parser-recorded spans as the substitution boundary, and copy static source without eval. The full Wasm and asm.js builds now agree while minimal output stays unchanged. Static parts remain raw source, so escapes are not cooked and a literal * stays literal. ## Test plan - chomp test Fixes: guybedford#137

## Summary - Add `StaticReexportStar` as import type 8 for `export * from`. - Report the matching `*` export with a span that correlates to the import. - Cover full and minimal builds without reading fields omitted by the minimal API. ## Why Star re-exports are dependency edges and exports, but the lexer only exposed their specifier as a plain static import. Consumers had to recover the missing distinction by parsing the source again. ## Test plan - `chomp test:wasm test:minimal:wasm` Refs: guybedford#76 Refs: nodejs/import-in-the-middle#259

… name A detached `export { x }` (no `from` clause) emits an identical export record whether `x` is a locally-declared binding or one introduced by an `import`. A consumer that needs to resolve re-exported imports from the leaf module's namespace (import-in-the-middle) cannot tell the two apart and is forced onto a slow fallback. Record the local binding names an import introduces - named specifiers (the `as` target, or the imported name when there is no `as`), the default binding, and a `* as ns` namespace - and, when a detached export resolves to one of them, report it with no local name (`ln === undefined`, `ls === le === -1`), exactly as `export { x } from` already does. A genuine local keeps its name. The match is by name only: a lexer cannot do scope analysis, so an import that follows the export (single pass) or a same-named local that shadows an import is not resolved - the same assumption the `export { x } from` case already makes. The named part of a combined `default, { a }` clause is not tracked; such a re-export keeps its local name.

This fixes the detached re-export classifier so import bindings with comments after `as` are still recorded by their target name. Without this, `as` itself could be mistaken for an imported binding and later local exports named `as` were reported as re-exports.

The minimal build must retain its existing export records, so it compiles the imported-binding tracking state and parser paths out. String-named imports such as `import {'a-b' as c}` were mistaken for module specifiers by the generic quote scan. Parse the named clause before reading the module string.

Detached exports can precede their imports, so source-order tracking cannot preserve their module request. Resolve complete module bindings and link reexports to the imports array. On Node 24 / V8 13.6, imports plus a direct export measured 5.08 -> 5.28 us, while 1,000 detached reexports measured 169.3 -> 145.2 us export-first and 166.5 -> 159.0 us import-first (seven trials, drop best and worst). The minimal build remained within noise.

Binding capture was gated on a JS-side prescan of the whole source, so a module's classification depended on its length and both wrappers paid the scan on every parse. On the sample corpus that scan cost more than the lexing it guarded: 722 us of the 878 us angular.js regressed, and effectively all of the asm.js build's. Capturing unconditionally is never slower, because the binding scanner also replaces a tolerant clause loop that called commentWhitespace per character: 1000 imports with direct exports measure 98.9 us captured against 141.7 us through that loop. The gate, eac(), the record re-parse and the loop are gone, and one scanner serves every source. The single scanner exposed four defects the size gate had hidden: 1. `export * from <non-string>` stored through a null import head; in the asm.js build that overwrote a keyword table, silently dropping every import from every later parse in the process. 2. A bare `export *` emitted a reexport record with no import, so both wrappers threw a TypeError instead of a ParseError. 3. An import-clause character that cannot start a binding was re-read forever. 4. `as` directly followed by a comment did not read as a rename. The asm.js analysis arena is now bounded in C and re-parsed into a larger buffer on overflow instead of truncating records silently, which retires the last two full-source scans. Against the previous commit the corpus runs 26% faster and the Wasm binary drops 3197 bytes. The feature now costs 2.3% and 2802 bytes over the unmodified lexer.

`br` is loop invariant, so testing it on every character of a comment body only costs. Comment-heavy sources gain the most: angular.js drops 7.8%, 8.6% and 8.9% across three runs on Node 24.18.0 (best of 21 interleaved trials), for 37 bytes of Wasm.

Export origin analysis needs to know which import a re-exported local name is bound by, and storing a record per binding during the parse bills every module that imports names. The answer is only read for detached exports — `export { a }` where `a` came from an import — which most modules do not have. The bindings are still in the source, so a module that ends the parse with an unresolved export re-reads its import clauses in the finalize pass and resolves the names against the export table as it goes; a module without one never reads a clause twice. `ImportBinding` and its list are gone, along with the per-binding allocation and its capacity check.

This speeds up every module with a named import clause. Only the finalize pass resolves bindings, and only when a detached export exists to consume them, so the main loop needs the position of the closing brace and nothing else. A scan stepping over the four constructs able to hide a brace — string specifiers, line comments, block comments and `\u{...}` escapes — replaces the specifier reader there, and one entry point picks between the two so they cannot disagree about where a clause ends. Wasm, min of 21 runs in one process: imports without detached exports -10.4%, detached export-first -6.3%, detached import-first -6.0%, a module ending in a clause export -4.0%, angular.js -4.2%.

Open addressed inserts probe for a free slot, so every export sharing a local name walks past all the earlier ones and a clause builds its table in quadratic time. Exporting one local under 200,000 names — `export { a as x0, a as x1 }`, valid and reachable from a generator — took 10.7 seconds where the same count of distinct names took 11 milliseconds. Bundlers and CDNs run this parser on untrusted input. Chaining the bucket makes an insert an unconditional prepend with no comparison at all, and one walk resolves every export a name covers. The same 200,000 aliases now take 17 milliseconds, and a corpus A/B is unchanged.

`ensureAnalysisCapacity()` inlines into every record allocation, and carrying the page-grow branch along costs the allocators more than the bounds test it guards. Record-dense sources measured 9.8% slower than an unchecked allocator; with the grow path out of line they are within noise of it at 0.5%, so the arena keeps its bound without the records paying for it.

A module with a clause export and no import clause — `const a = 1; export { a }` — allocates, zeroes, populates and sweeps a hash table that has nothing to match against, and hashes every pending export name to fill it. Only a binding introduced by an import clause ever resolves one of those exports. The clause readers now record that the module has such a binding, and the name hash moves to the table insert so it happens only where a table exists. That shape drops from 156.6 ns to 147.2 ns, against 136.2 ns for the same build with the analysis removed entirely, and the rest of the corpus is unchanged.

Re-reading import clauses in the finalize pass is the dominant cost of the analysis: 8 of the 12 microseconds a 120-import module spends over a build without it, and 13 of 15 on a clause-heavy one. Most of those clauses cannot contain the name being looked for. An import statement now records the 32-bit set of characters its binding names are drawn from, accumulated in the brace-skip on code units it already loads. A pending export name that occurs in a clause necessarily has its first character in that set, so an import missing every pending first character is skipped without being read. An escape sets the mask to all ones on either side, because `\u{61}` and the `a` it denotes share no code unit. That module goes from 63.4 us to 59.7 us, against 56.2 us for the same build with the walk removed outright. Sources where every import really does bind an exported name are unchanged.

…ding export" This reverts commit 96e7c1b. The filter was measured against fixtures carrying 120 and 400 import statements. Real modules that reach the finalize walk carry a median of 2: across 878 ESM files in two node_modules trees, 50 run the walk at all, between them re-reading 186 import clauses, of which the filter skips 34. Parsing that corpus measures 0.4% to 3.3% slower with the filter over four runs, and slower in every one of them on the 93 files that run the walk. The per-code-unit mask accumulation costs more than the clause reads it avoids.

## Summary Export metadata reads the same import records that carry interpolated-template spans. Keep the span fields, readers, and parser recording when export analysis is enabled so dynamic import globs remain available. ## Why The full build needs both metadata sets from one import record. Minimal builds compile the glob fields and readers out. ## Test plan - chomp test

BridgeAR force-pushed the BridgeAR/2026-07-22-export-analysis branch 2 times, most recently from 953f241 to 6811b35 Compare July 23, 2026 15:25

BridgeAR force-pushed the BridgeAR/2026-07-22-export-analysis branch from 6811b35 to 7982cd7 Compare July 25, 2026 12:23

guybedford marked this pull request as ready for review July 26, 2026 05:26

BridgeAR and others added 21 commits August 2, 2026 15:42

ci: run the build workflow for the v3 branch

3754702

fix: anchor optional asm strip patterns, drop stale wasm getter types

b505bc9

BridgeAR force-pushed the BridgeAR/2026-07-22-export-analysis branch from d755568 to 779e5bd Compare August 2, 2026 14:31

Navigation Menu

Uh oh!

feat: add export origin analysis - #229

feat: add export origin analysis#229
BridgeAR wants to merge 23 commits into
guybedford:v3from
BridgeAR:BridgeAR/2026-07-22-export-analysis

BridgeAR commented Jul 23, 2026 •

edited

Loading

Uh oh!

guybedford commented Jul 26, 2026

Uh oh!

guybedford commented Jul 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

BridgeAR commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Uh oh!

guybedford commented Jul 26, 2026

Uh oh!

guybedford commented Jul 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BridgeAR commented Jul 23, 2026 •

edited

Loading

guybedford commented Jul 26, 2026 •

edited

Loading