feat: lex type-only TypeScript imports and exports

[BridgeAR]

Summary

Teaches the full Wasm and asm.js builds to lex the type-only TypeScript subset that Node's stripTypeScriptTypes erases: type-only import/export clauses and type / interface declarations. Type-only module edges are reported with tp: true, and declaration bodies are skipped opaquely so nested import() types do not leak into the runtime graph.

Why

A single lexer avoids a separate TypeScript transform and gives module-graph consumers both runtime and type-only edges. Plain JavaScript remains metadata-compatible. TypeScript declaration lookahead only runs after shallow inline keyword gates, and tp reuses existing import/export reader values instead of adding a Wasm boundary call per specifier. Minimal builds remain JavaScript-only and compile out the type-only fields and paths.

The supported boundary follows Node's type stripping for these forms. Value-position annotations and generics, as / satisfies, non-null !, and non-erasable syntax such as enum and runtime namespace remain out of scope.

Fixes: #72

[guybedford]

Thanks Ruben — this is a strong first increment: the write-head sentinel guards are correct, the minimal build reader/typings are properly insulated, and the inline-modifier semantics get the subtle case right (import { type A } keeps a runtime edge in Node's stripping output, so statement-level tp: false there is exactly right). Requesting changes on the items below.

Bug: import type from 'x' must be tp: false

Verified against both Node stripTypeScriptTypes (v26) and tsc 5.9 verbatimModuleSyntax: this form is a value import of the default export bound to the local name type — both keep it verbatim. Only import type from from 'x' (default import named from) is type-only:

import type from 'm';       =>  kept verbatim (value import)
import type from from 'm';  =>  stripped (type-only)

The PR marks the first tp: true, and the test comment ("Node strips the whole import") is incorrect. Since import type from 'x' is also valid plain JavaScript, a consumer eliding tp: true imports would drop a real runtime import from a pure-JS file — this breaks the JS-superset guarantee the PR is built on. The type follower exclusions need a third case: from followed (after whitespace/comments) by a quote. Please add import type from from 'x' as the disambiguation test.

Gating must be complete — zero cost to the minimal build

src/lexer.c is fully behind #ifdef LEX_TS, but src/lexer.h is not:

• bool type_only; on struct Import / struct Export
• the type_only = false stores in addImport / addExport
• the itp() / etp() accessors

The minimal build currently pays two live stores per record and +4 bytes heap per export (the bool lands between pointer fields in the min Export layout), and only DCE keeps itp/etp out of the min wasm. All of these need #ifdef LEX_TS. This composes with the asm requirement below: the asm build defines LEX_TS and gets the field for real; the minimal build carries zero trace.

asm.js build must be included

The hand-maintained-dictionary blocker no longer exists: the build now auto-extracts the keyword dictionary from the fastcomp memory image (the lib/lexer.asm.in.js task scans lexer.layout.js.mem and substitutes {{WORDS}}/{{OFFSET}}), so adding YPE propagates automatically. Concretely:

• -D LEX_TS and _itp/_etp in EXPORTED_FUNCTIONS on both fastcomp tasks (lib/lexer.emcc.asm.js and the layout sidecar must keep identical layout-affecting flags)
• src/lexer.asm.js non-minimal branch: tp: !!asm.itp() instead of hardcoded false
• run the test/typescript/ suites under ASM=1 — _harness.cjs hard-imports dist/lexer.js; give it the same WASM/ASM env switch the legacy suites use
• fix the now-stale "MANUAL ASM DICTIONARY CONSTRUCTION" note at the top of lexer.c, and the README/lexer.ts JSDoc claims that the asm build is JS-only

Type declarations must be tracked

export type Foo = ... and export interface Foo {} need to record tp: true exports — top-level type / interface dispatch from the statement-start check exactly like import/export. Implementation notes:

• For type Foo, read the name, then stop at the = and skip the RHS. Note default type parameters put a = before the real one (type Foo<T = string> = ...), so after the name skip a balanced <...> region first (>> closes two, => in function types doesn't close, {}/[]/() nest inside). The same angle skipper serves interface Foo<T> extends Bar<T> {.
• Interface bodies should be skipped opaquely (brace-matching, not the main token loop): a member like import(): void otherwise matches the dynamic-import dispatch and records a bogus import edge — a hazard that exists latently today for bare interface blocks.
• Guards for the JS superset: require an identifier-start char after the keyword (type = 5, type(x) are plain JS) and no line break between type and the name (type\nX = 5 is two JS statements via ASI).
• Dotted names can't be declared (export type Foo.Bar is invalid TS); qualified names only appear as RHS references (export type Bar = Foo.Bar;), which the RHS skip covers for free. import Bar = Foo.Bar / export import are non-erasable and stay out of scope.
• Scope boundary: bare interface Foo {} + later export { Foo } should not be resolved — Node stripping does no cross-binding analysis (erasable TS requires export { type Foo }), so only directly-exported declaration forms are marked.

Smaller items

• isTsTypeKeyword follower set misses import type/*c*/{ A } from 'm' (verified: Node strips it; the lexer reports tp: false) and import type* as ns from 'm'. Accepting / and * as followers is safe — the callers' savePos/restore logic already disambiguates.
• The single-token as lookahead in the export brace loop mishandles export { type as/*c*/T } and export { type as as X } (type modifier on a specifier named as). Pathological, but worth a deliberate call — at minimum a comment.
• README opening ("lexes the erasable TypeScript syntax that Node.js type stripping accepts") overstates this increment; scope it to the type-only subset until the annotation work lands.
• Untested: export type * from 'x' (traced as correct — import edge tp: true — but pin it), and the tp: false additions in the JS builds once the harness is env-parameterized.
• Nit: js-build-unchanged.cjs runs against the wasm build; js-superset.cjs would name what it actually asserts.

[BridgeAR]

This is still a WIP PR with some issues and not yet ready. It should move in the right direction though.

[guybedford]

This is really nicely done — pinning the accept/reject boundary to Node's stripTypeScriptTypes and backing it with a differential fuzzer is exactly the right approach for an opaque skipper, and it clearly earned its keep (several tests are annotated "fuzzer-found"). The minimal build staying zero-cost behind #ifdef LEX_TS and the JS-superset guarding are both thorough.

Two things I'd like addressed before merge:

1. import type from'x' (no space before the quote) is mis-marked type-only. In the import type detection:

if (!typeIsBinding && nextCh == 'f' && memcmp(pos + 1, &ROM[0], 3 * 2) == 0 && isBrOrWs(*(pos + 4))) {

The from-followed-by-quote check requires whitespace after from. import type from'x' is valid JS (type default binding, from keyword, 'x' with no separating space), but it fails isBrOrWs(*(pos + 4)), so typeIsBinding stays false and the import is reported tp: true — dropping a real runtime edge from a file that is also valid JavaScript, which breaks the "no JS consumer observes a change" invariant. The fuzzer misses it because the import-type-from form always emits a space before the specifier. Fix is to also accept a quote here, e.g. isBrOrWs(c) || isQuote(c), and add the no-space case to the fuzzer's form.

2. Type/interface declarations nested in a block still leak their import(...) types. Both bare triggers gate on openTokenDepth == 0:

if (*(pos + 1) == 'y' && openTokenDepth == 0 && keywordStart(pos))
  tryTsTypeDeclaration(true);

So function f() { type X = import('m').T; } — which Node strips cleanly — reports import('m') as a runtime edge. This is a remaining gap rather than a regression (everything leaked before), and the openTokenDepth == 0 guard is a reasonable defense against expression-context misfires, so I'm fine leaving the behaviour for a follow-up. But the README currently says type and interface declarations are skipped "whether exported or not" with no nesting caveat — please add a one-line scope note there so the limitation is documented, and ideally extend the fuzzer to emit nested declarations (it only generates top-level statements today, so it can't surface this class).

3. Prefer forcing the keyword dictionary contiguous over reconstructing it from a gappy span. The gen-asm-in.mjs change reconstructs the dictionary across the whole first-to-last span, filling gaps with NUL, because adding the new tables made the compiler place nterface at a non-adjacent offset. That works, but scanning first-to-last across the entire memory image is sensitive to any unrelated printable-char16 constant landing between the tables (it would widen the span and bloat words with a long NUL run).

The split happens only because each keyword is a separate static const char16_t[] object, which the compiler/linker may reorder or interleave. Merging them into one array and offsetting into it keeps the object contiguous on both toolchains (no linker script or section attributes needed — those don't actually guarantee intra-section ordering):

static const char16_t KEYWORDS[] = {
  'x','p','o','r','t',              // xport
  'm','p','o','r','t',              // mport
  /* ... */
  'y','p','e',                      // ype
  'n','t','e','r','f','a','c','e',  // nterface
};
#define XPORT (KEYWORDS + 0)
#define MPORT (KEYWORDS + 5)
/* ... */

The call sites already pass explicit lengths (memcmp(pos + 1, &MPORT[0], 5 * 2)) and never use sizeof, so only &NAME[0] becomes NAME. That lets the asm extraction go back to "grab the single contiguous run", drops the NUL-gap span logic and its end - start fragility, and the contiguous blob (no gap NULs) is likely smaller than today's span.

[guybedford]

Review — type-only TypeScript import/export lexing

The architecture is sound and test coverage is reasonable, but the new speculative TS-skipping paths in src/lexer.c have a cluster of real bugs. The common thread: when the lexer opaquely skips an erased declaration, it leaves lastTokenPos, pos, and openTokenStack in states the rest of the tokenizer doesn't expect. Two of these are memory-safety / DoS class and were reproduced against a native build. I'd block on the five correctness findings below.

Correctness (all confirmed)

1. src/lexer.c:821 — wild pointer read after skipping a {}-terminated declaration (memory safety). Skipping interface A {} (or type X = {a:1}) leaves lastTokenPos on the } but never pushes onto openTokenStack, so openTokenDepth stays 0. A following / — e.g. interface A {}\n/import('m')/.test(x); — makes handleSlash read openTokenStack[0].pos (uninitialized stack garbage) and dereference it in isExpressionTerminator. SEGV under ASan; in Wasm it reads arbitrary linear memory and may leak a bogus import edge.

2. src/lexer.c:816 — enclosing block's } consumed as a type operator, corrupting brace depth. For valid TS function f() { type X = A }\nimport { v } from 'runtime';, the bare alias-RHS loop treats the function's closing } as an operator (operandPending = true; pos++), then swallows the following import. openTokenDepth is left at 1 and parse() throws PARSE_ERROR on input Node's stripTypeScriptTypes accepts. Also repros with switch/case bodies and object-method bodies.

3. src/lexer.c:161/:821 — regex after an erased bare type alias misread as division, leaking a phantom import. type X = A\n/import('m')/; erases to a regex statement with no imports, but the skip leaves lastTokenPos on A (an expression token), so handleSlash picks division and lexes the regex body as code — the built lexer reports a dynamic import of 'm' that doesn't exist after stripping.

4. src/lexer.c:692 — interface rejects an intervening comment before the name, unlike type. The guard !isBrOrWs(*(pos + 9)) fails on /, so export interface/*c*/Foo { m(): import('m').T } (valid TS, accepted by Node) isn't recognized: its body is tokenized, import('m') is recorded as a real dynamic import, and the Foo type-only export is dropped. isTsTypeKeyword (src/lexer.c:572) already accepts / as a follower for type — the interface check should match.

5. src/lexer.c:717 / skipTsBalanced — O(n²) on unbalanced < (DoS on untrusted source). In speculative mode, type a< triggers skipTsBalanced on the <, which scans to EOF when no > matches; the missing = then restores pos fully and the main loop re-lexes one token forward, re-triggering at the next type a<. Input "type a<\n" repeated N times is quadratic. ; and line breaks don't bound the angle scan.

These five share a root cause worth fixing at altitude: after an opaque declaration erasure, the skipper should restore tokenizer state to what an erased statement looks like — leave lastTokenPos on an expression-terminator sentinel (like ;) and never let an enclosing closer fall through the RHS operator path — rather than each site patching pos-- individually. Fixing that consistently likely closes 1, 2, and 3 together.

Efficiency (confirmed, lower severity)

6. src/lexer.c:147,161 — the "plain JavaScript pays nothing" claim isn't quite right. With LEX_TS on (all three shipped builds), every keyword-start token beginning in (in, instanceof, index…) or ty (typeof) makes a non-inlined call into tryTsTypeDeclaration. It's a handful of comparisons per occurrence, not a deep lookahead — minor — but the description should be corrected, and the gate could be widened inline (mirroring the existing case 'c' LASS pattern) to keep typeof/instanceof inside the switch arm.

7. src/lexer.ts:300,309 (and src/lexer.asm.js:75,84) — one extra wasm boundary call per specifier on every module. itp()/etp() exist solely to move the type_only bit across the boundary, including for plain JS where it's always false (~9% more calls per import, ~14% per export). Nearly free to fold in: it() returns enum 1–7 so bit 3+ is free (import_ty | (type_only << 3)); for exports, re()'s bool can carry it as value 2.

Minor

• test/typescript/regex-division.cjs:4 — the // Increment 1 adds … comment narrates the PR's development staging rather than code semantics; goes stale once later increments land.
• The triple-duplicated contextual-type disambiguation (src/lexer.c:361/997/1026) looked worth factoring, but on inspection the shared skeleton is only ~5 lines and the follower logic is genuinely site-specific — I'd leave it. (asm.js has no TS logic, so it's three copies, not six.)

[guybedford]

LGTM, we must just make sure to land this on v3 not main.