Lexer overhaul: rewrite the pipeline around the logos vocabulary (post-0.11.0)

[tobert]

Trigger

Two cross-model batch reviews of crates/kaish-kernel/src/lexer.rs (whole file, holistic, no diff) converged hard:

• gemini-pro: gemini/batches/1xtxjpdweuxa2hfm6hyo3tdmoj23e9ad6ojk
• fable: anthropic/msgbatch_01HqpFLAG39L3irq6XGeyHtx

Shared verdict: the architecture (logos + post-pass fusion) is conceptually sound, but the file has become a four-stage mini-compiler (two string-rewriting preprocessors → logos → marker re-threading → three fusion passes) whose stages communicate through three unchecked channels: in-band marker strings, span adjacency, and pass ordering. compute_value_context is a stack machine disguised as counters. Both reviews independently recommended refactoring before the next feature touches this file.

Decision (2026-07-04, Amy): full rewrite of the pipeline, single PR, no phasing — the ~4,100-test suite is the spec. Lands after 0.11.0 ships (opens the 0.12.0 cycle).

Confirmed bugs (all corruption-class, all die by construction in the rewrite)

1. Heredoc-blind arithmetic — preprocess_arithmetic runs first with zero heredoc awareness. An apostrophe in a heredoc body (don't) opens quote mode and poisons all later $(( )) in the buffer; a literal $(( in a doc body false-positives UnterminatedArithmetic.
2. Quote/comment-blind heredocs — preprocess_heredocs tracks no quote or comment state: echo "a << b" misfires heredoc collection; # see <<EOF in a comment does too.
3. Marker leak on glued arithmetic — echo $((1+2))abc preprocesses to __KAISH_ARITH_{id}__abc, which lexes as ONE Ident; the ends_with("__") check fails and raw marker text reaches the parser/user. Bash prints 3abc; kaish prints garbage.
4. Numeric re-stringification in fusion — mergeable_text/glob_mergeable_text rebuild text via Int(n) => n.to_string(): a:007 fuses to Ident("a:7"), 007* globs as 7*. Contradicts the file's own DashNumWord verbatim-slice principle.
5. Positionally-blind Eq — any top-level = opens value position for the next token (lexer.rs:1841), whitespace or not: cmd a = [x] / grep -E = [a-z]* suppress glob fusion where the grammar's notion of assignment requires span adjacency.
6. No statement-boundary reset / no scope isolation in compute_value_context — an unterminated x=[a b changes the tokenization shape of every following line; [[ -n $(x=[a]) ]] leaks test-depth state into the subshell.

Also noted: heredoc rewrites produce no SpanReplacement records at all — every token after a heredoc carries spans in the wrong coordinate system (body_start_offset documents its own drift); tokenize_with_comments is a divergent second pipeline (no preprocessing, no merges) with zero callers outside tests.

Contract inventory (what survives, verified against current code)

Surface	Consumers	Disposition
Token enum (~90 variants) + Display/category()/is_*	parser.rs (sole kernel caller of tokenize)	Untouched — the parser's vocabulary
tokenize() signature, Spanned, LexerError	parser, REPL	REPL matches UnterminatedHeredoc for continuation prompts (kaish-repl/src/lib.rs:200) — preserve
HereDocData.body_start_offset	parser interpolation spans (parser.rs:1906)	Becomes exact original-source offset (fixes documented drift)
${__ARITH:expr__} inside String token values	parse_interpolated_string (parser.rs:446, 680)	Keep — computed directly at scan time, never string-fished
__KAISH_ESCAPED_DOLLAR__	parser.rs:75 also generates it	Keep; in-band collision (a user literally typing the marker) filed as follow-up below
parse_string_literal / parse_var_ref / parse_int / parse_float	parser	Keep
tokenize_with_comments	none (tests only)	Fold into the one pipeline as a keep_comments flag; divergent dialect dies

Architecture (decisions locked)

Keep logos as the primitive word classifier; rewrite everything around it. The regex vocabulary encodes every shipped idiom decision (DashNumWord ISO dates, AtWord, PlusBare, ShortFlag/Int/MinusBare priority games) and none of the confirmed bugs touch it. Hand-rolling it would re-derive ~60 tested decisions for zero defect payoff.

Replace the ~1,400-line pipeline with:

1. One composed source-order scanner — a single quote/escape/comment state machine that extracts heredocs AND arithmetic in one pass (no mutual blindness possible). Heredoc bodies sliced raw at collection time; for interpolated (non-literal) heredocs, arithmetic-in-body is rewritten to ${__ARITH:expr__} body-locally at collection. Produces a complete replacement table (heredocs included — kills the span drift).
2. Positional marker matching — placeholder→token swap keyed by span/table position, never by fishing Ident string names. Glued $((1+2))abc becomes detectable: emit Arithmetic + adjacent word so the parser's no-pasting guard makes it a loud error with a quote hint (decision: crash-over-corrupt; bash-style 3abc interpolation explicitly rejected as scope creep).
3. One context pass — explicit Vec<Ctx> stack (Test/List/Record), computed once; span-adjacent = discrimination for assignment vs. comparison; scope isolation at $() boundaries; mismatched closers detectable.
4. One fusion pass — single run collector; flush decision tree covers colon-fuse, glob-fuse, flag-metachar absorption, lvalue & value-position suppression; fused text is a verbatim source slice (leading zeros survive). Assert no replacement boundary inside a fused run.

Estimated ~1,000 new lines replacing ~1,400; Token enum and helpers (~1,100 lines) retained.

Process

1. Census first (TDD) — characterization tests for the six bugs as should-be behavior (failing today), plus two seam traces that must be understood before writing the engine:
   • $(echo $((1+2))) — the outer arithmetic preprocessor deliberately skips inner arithmetic via skip_command_substitution; trace what the parser actually receives and pin it.
   • Whether multiline list literals are legal (parser grammar + docs/arrays-and-hashes.md) — this decides the statement-boundary-reset rule for the context stack (a blind reset at Newline would break literals that span lines).
   • Audit existing tests that may pin corrupt behavior (span assertions after heredocs; fused-number text).
2. Engine swap, single commit arc, full gates (cargo test --all, clippy --all-targets, cargo insta test --check).
3. kaibo review pair per house style: deepseek consult + gemini-pro batch, whole file attached, no diff.
4. CHANGELOG Fixed bullets — corrupt→loud is agent-visible behavior change (echo "a << b" starts working; glued arith goes from garbage to loud error).

Follow-ups (not in scope)

• __KAISH_ESCAPED_DOLLAR__ in-band collision (fable finding G): fixed string, not uniquified, parser-side contract — needs a parser change, separate issue when picked up.
• Speculative fable finding F severity (argv = observable effects) resolves itself in the rewrite via span-adjacency.

References

• Full review texts: collect batch handles above with kaibo job_get (durable).
• Both reviews' "not recommended": pushing [[/]] into logos as compound tokens — the two-token choice is correct; the pairing logic just belongs in the explicit stack.

[tobert]

Dep-audit follow-up (2026-07-04): logos is already at the latest published version (0.16.1), so this plan needs no dependency work. The workspace's one real prerelease is the parser's chumsky 1.0.0-alpha.8 — a dead-ended line upstream — filed separately as #98. The two are deliberately independent (the seam is pure Vec<Spanned<Token>> data); keep them in separate PRs, either order.