refactor(lexer): rewrite the pipeline around one composed scanner (GH #95)

[tobert]

Closes #95.

What this is

The full lexer-pipeline rewrite locked in #95 (2026-07-04, after two cross-model
batch reviews converged on it): the four-stage pipeline — two string-rewriting
preprocessors → logos → marker re-threading → three fusion passes, communicating
through in-band marker strings, span adjacency, and pass ordering — is replaced by:

1. scan() — one composed source-order pass with explicit quote/escape/comment
   state extracts heredocs AND arithmetic together, emitting a rewritten buffer plus
   a complete replacement table in both coordinate systems (the old pipeline
   recorded no replacements for heredocs; every span after one drifted).
2. logos — the regex vocabulary and Token enum are untouched (they encode ~60
   shipped idiom decisions; none of the bugs lived there).
3. Positional marker resolution — replacement-table ranges, never
   identifier-name fishing. A word glued onto a marker splits into Arithmetic +
   re-lexed fragments so the parser's no-pasting guard rejects it loudly.
4. Exact span mapping — every token span, HereDoc included, is an exact
   original-source byte range; body_start_offset is exact (was documented drift).
5. Fusion with verbatim slices — fused text is sliced from the source
   (a:007 stays a:007; 007* globs as 007*), suppression decided by a new
   compute_value_context: an explicit frame stack (Test/List/Record/Subst/Paren)
   plus a statement-head DFA modeling the real assignment grammar.

Bugs fixed (all corrupt→loud or broken→works)

• Apostrophe in a heredoc body no longer poisons later arithmetic; literal $(( in
  a <<'EOF' body is prose (was a false UnterminatedArithmetic).
• echo "a << b" and # see <<EOF no longer misfire heredoc collection.
• echo $((1+2))abc — loud no-pasting error with a quote hint (was raw
  __KAISH_ARITH_…__abc leaked to the user); echo "$((1+2))abc" prints 3abc.
• Leading zeros survive fusion (a:007, 007*).
• Argv = no longer suppresses glob fusion (grep -E = [a-z]*); real assignments
  (glued/spaced/local/subscripted/env-prefix chains) still put their RHS at value
  position — census found spaced x = [a b] is a legal assignment, so the fix is a
  statement-shape DFA, not span adjacency as Lexer overhaul: rewrite the pipeline around the logos vocabulary (post-0.11.0) #95 sketched.
• $( ) bodies get fresh context: [[ -n $(x=[a]) ]] no longer leaks test depth.
• Census bonus finds: echo $(echo $((1+2))) now prints 3 (substitution
  bodies are lexed inline — the old preprocessor skip left inner arithmetic raw and
  unparseable); ${X:-$((1+2))} is a loud new ArithmeticInVarRef error (was a
  silent marker leak into the AST); $# no longer opens comment state
  (echo $# $((1+2)) works); two heredocs on one line both collect (the parser
  then rejects ambiguous stdin with its own message).

Deviation from the locked plan

The three fusion passes stayed three passes sharing the one new context walker,
rather than collapsing into a single pass: their alphabets differ deliberately
(Float colon-fuses but doesn't glob-fuse — 1.5:x* fuses, 1.5* stays split),
and a union-alphabet pass would change tokenization in corners the suite pins.

Testing & review

• cargo test --all: 4,606 passed — the entire pre-existing suite passed the
  new engine after exactly one fix (bare-CR heredoc terminators). 25 new
  characterization tests in lexer_pipeline_tests.rs pin all eight corruption
  classes dead, plus heredoc-introducer line continuation (verified bash-parity
  end-to-end: cat <<EOF \ + | tr a-z A-Z uppercases the body).
• clippy --all --all-targets 0 warnings; cargo insta test --check clean;
  no-default-features check; wasm32-wasip1 build.
• kaibo review pair on the worktree: deepseek consult (its HIGH didn't reproduce —
  continuation semantics are bash's; its four cleanups landed) + gemini-pro batch
  (truncated output; visible finding was the documented spaced-lvalue trade-off).
• Docs: CHANGELOG bullets; LANGUAGE.md's "context-unaware preprocessing" Known
  Limitation replaced by the two loud errors; devlog entry.

🤖 Generated with Claude Code