WIP: Parser#34
Draft
TheLazyCat00 wants to merge 149 commits into
Draft
Conversation
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
TheLazyCat00
force-pushed
the
essential/parser
branch
from
May 25, 2026 21:27
0605f46 to
b7b556d
Compare
…alls, kill method-call ambiguity (#35) * parser: add reference expr and immutable method call, fix warnings - add `&expr` reference-to-a-value expression (uses the AND token) - add `obj:method()` immutable method call alongside `obj!method()` mutable call; Verb_call.Meth now carries an is_mut flag - drop the redundant ERROR token; the lexer raises Lexing_error on an unrecognised character, formatted through the same parse-error path - remove the never-useful IF/ELSE precedence levels and the redundant %prec DOT on dot-access Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01PkF51jD8vywTCqcP4UNmXq * parser: add @ intrinsic namespace, remove class token - add `@package$name` intrinsic-namespace form to both name_type and name_expr (e.g. @primitives$I32, @funcS$strToI32("30")), modelled as an Intrinsic constructor mirroring Qualified; uses the AT token - remove the unused CLASS token and its lexer rule Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01PkF51jD8vywTCqcP4UNmXq * parser: stratify expr and unify calls to kill method-call ambiguity Split expr into primary / app (postfix chain) / expr (binary), so a call's receiver and a method name can no longer swallow a trailing binary operator. `a!b(c) * d(e)` now parses unambiguously as `(a!b(c)) * (d(e))` -- calls bind tighter than arithmetic -- rather than forking into two trees. Method calls `!`/`:` share one production (identical shape; the marker only selects the is_mut payload), and function and method calls are unified into a single form: a flexible `app` receiver with an optional (marker, primary-name) method part. The method name is a primary; the receiver is a full postfix expr. Removes the now-structural LPAREN/DOT precedence levels. Reduce/reduce conflict states drop 14 -> 6 and shift/reduce 54 -> 40; the whole method-call conflict family is gone. No node changes -- primary/app/expr all still yield Expr.t. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01PkF51jD8vywTCqcP4UNmXq --------- Co-authored-by: Claude <[email protected]>
* Add bounded ambiguity search recipe * Add bounded Menhir ambiguity finder * Prune inactive ambiguity lookaheads * Configure optimized ambiguity search executable * Add optimized parallel OCaml ambiguity search * Use parallel OCaml ambiguity search by default * Address ambiguity search review feedback * Report multiple ambiguity families * Expose ambiguity witness limit * Bound queued ambiguity frontiers * Add controlled syntax experiment harness * Test syntax experiment transformations * Document syntax experiment workflow * Add syntax experiment recipes * Respell known witnesses in each variant's syntax (#37) * Harden worker waits, file reads, and search invocation - retry Unix.waitpid on EINTR so signal delivery cannot abort result collection - close the read_lines channel with Fun.protect on any exception - strip trailing carriage returns before parsing token declarations - run syntax experiments against a prebuilt ambiguity_search.exe instead of concurrent dune exec, validating the executable up front with a build hint - build the executable in the syntax-experiment justfile recipes Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR * Respell known witnesses in each variant's syntax Each grammar transformation now registers a matching spelling update, and every known case builds its intended program from the composed spelling profile. A rejection now means the variant genuinely cannot parse the intended program, never that the old spelling became illegal; cases a variant cannot express at all are labeled explicitly and counted as rejected. Adds a plain named-call compatibility probe (print("hello"), spelled print["hello"] under bracket-calls) that every variant must parse exactly once, plus focused spelling tests. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR * Add anchored-abort-handle experiment variants A deep 10M-frontier search found 49 complete ambiguity families, of which 47 pivot on abort-handler attachment: every operator and call production carries its own optional handler suffix, so ?? and ? handlers can attach at multiple nesting depths of one undelimited expression. The new anchored-abort-handles transform removes the per-production slots and allows handlers only at delimited boundaries (statement calls, declaration values, return/resolve/abort values, call arguments, and grouping parentheses). Witness spellings are unchanged, so the spelling update is the identity. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR * Document the ambiguity policy for the GLR grammar Zane's grammar is deliberately not LR(1) and stays GLR-parsed, but must remain provably unambiguous: every LR conflict state carries either a documented precedence resolution, a transience argument, or a tracked open obligation that the bounded ambiguity search continuously attacks. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR * Add a conservative unambiguity prover (--prove) Abstract GLR stacks to their top-K states, with reductions that pop into the unknown region re-entering through every goto edge on the reduced nonterminal. The abstract configuration space is finite, so a pair search over runs consuming the same input terminates: if no pair of runs diverges and still accepts, the grammar is proven unambiguous with no sentence-length bound. Reduction chains advance in lockstep so shared cycles cancel, divergence is recognized only where two parses of one sentence can first part ways (different productions, or reducing against shifting), and a non-diverged pair with equal suffixes resolves unknown-base gotos identically on both sides because it is still a single run. An abstract candidate is concretized with the existing bounded search: witnesses mean ambiguous (exit 1), otherwise the verdict is not-proven (exit 3); a proof exits 0. Validated on known-ambiguous, LR(1), precedence-resolved, and unambiguous non-LR grammars (palindromes), on the Zane baseline (ambiguous, real witnesses), and on the anchored-handles variant (not proven at 10 tokens, ambiguous with witnesses at 12). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR * Check search executables portably and hoist joint move lookups Use os.path.dirname to detect a path-like search command so the existence check also runs on Windows, and bind both sides' move lists outside the joint pairing loops instead of re-looking one side up per inner iteration. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01QR6jhw2CXW3oQbBTT4C1aR --------- Co-authored-by: Claude <[email protected]> * Address ambiguity search review feedback --------- Co-authored-by: Claude <[email protected]>
'just ambiguities' (and 'just prove' / 'just syntax-experiments') never ran: they build via 'dune exec', whose default dev profile treats warning 34 as an error, and the 'signature' type was declared but never referenced because the function of the same name shadowed every use. Annotate that function with the type so the dev-profile build succeeds. Also flush stdout/stderr before forking search workers so buffered output (e.g. the --prove candidate report) is not replayed once per worker at exit. Claude-Session: https://claude.ai/code/session_01BmybBm68SKByiXbHszu2Cn Co-authored-by: Claude <[email protected]>
* Fix ambiguity tool build failure in dev profile 'just ambiguities' (and 'just prove' / 'just syntax-experiments') never ran: they build via 'dune exec', whose default dev profile treats warning 34 as an error, and the 'signature' type was declared but never referenced because the function of the same name shadowed every use. Annotate that function with the type so the dev-profile build succeeds. Also flush stdout/stderr before forking search workers so buffered output (e.g. the --prove candidate report) is not replayed once per worker at exit. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BmybBm68SKByiXbHszu2Cn * Bound search memory instead of stopping at the frontier limit The searcher previously halted outright when the dedup table reached --max-frontiers, and its stack pool and closure cache grew without bound for as long as the search ran, which is how long runs ended in OOM kills. Deduplication is pruning, not correctness, so the table is now a bounded two-generation cache that forgets stale entries instead of stopping the search. The same budget caps the number of queued frontiers - a full queue drops new discoveries and the run reports itself as interrupted - and the stack pool is periodically compacted against the live queue (clearing the closure cache) on a geometric schedule. Resident memory is now roughly 2 KB per --max-frontiers unit per worker and stays flat over time, so the depth reached is bounded by --timeout and --max-tokens rather than by RAM; on the current grammar the same budget that previously stopped at 11-token witnesses reaches 15-token ones. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BmybBm68SKByiXbHszu2Cn * Key the dedup cache on 124-bit digests; drop the Python search Storing full frontier signatures cost 100-200 bytes per dedup entry; hashing them through two independently seeded 62-bit mix lanes stores a 16-byte digest instead, so the same memory budget remembers four times as many frontiers and the search re-explores less (about a third more frontiers per second at deep settings). A digest collision could only skip a frontier wrongly, never fabricate a witness; the docs now state the completed-bound theorem is up to that astronomically unlikely case. tools/find_ambiguity.py was the slower Python reference implementation of the old algorithm and no longer matches the OCaml search's behavior; remove it and its ambiguities-python recipe. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BmybBm68SKByiXbHszu2Cn * Address review: exact unique count, document 64-bit requirement Rediscovering a cached frontier at a shallower depth went through Seen_cache.insert and inflated the unique-frontier statistic; split the match so only genuinely new digests count as insertions. The digest mixer's constants need OCaml's 63-bit native int; state that the tool requires a 64-bit platform rather than boxing the hot path with Int64. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01BmybBm68SKByiXbHszu2Cn --------- Co-authored-by: Claude <[email protected]>
Since the memory-bounding change, --max-frontiers did double duty: it sized the dedup cache (memory) and also capped the work queue (search reach). Lowering it to fit RAM therefore shrank the queue, so deep frontiers were dropped and the depth-bucket BFS terminated early, well before the timeout. Split the queue cap into its own --max-queue knob (defaulting to --max-frontiers, so existing invocations are unchanged). The queue is the live set that drives stack-pool memory, so the stack budget now tracks --max-queue; --max-frontiers is left as the independent dedup-cache budget, where a smaller table simply prunes less. This lets a run hold a deep queue without paying the 4x dedup over-provision, or shrink the dedup table without capping reach. Thread --max-queue through the just recipe and document both budgets. Claude-Session: https://claude.ai/code/session_016ratmfeMyTD7pMxaHDMiR6 Co-authored-by: Claude <[email protected]>
* Decouple ambiguity search reach from dedup memory budget Since the memory-bounding change, --max-frontiers did double duty: it sized the dedup cache (memory) and also capped the work queue (search reach). Lowering it to fit RAM therefore shrank the queue, so deep frontiers were dropped and the depth-bucket BFS terminated early, well before the timeout. Split the queue cap into its own --max-queue knob (defaulting to --max-frontiers, so existing invocations are unchanged). The queue is the live set that drives stack-pool memory, so the stack budget now tracks --max-queue; --max-frontiers is left as the independent dedup-cache budget, where a smaller table simply prunes less. This lets a run hold a deep queue without paying the 4x dedup over-provision, or shrink the dedup table without capping reach. Thread --max-queue through the just recipe and document both budgets. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_016ratmfeMyTD7pMxaHDMiR6 * add first run --------- Co-authored-by: Claude <[email protected]>
* Simplify ambiguity tooling configuration * Reserve just for parameterless commands * Read machine settings from the environment * Use one ambiguity tool for proof mode * Require explicit ambiguity configuration * Keep ambiguity search within its memory budget * Keep memory pressure within a narrow plateau * add new run
* Add targeted ambiguity search controls * Document targeted ambiguity searches * Use a function prefix in ambiguity example * try find f()(x)() * Make depth search rotate sibling nodes * Document nodes-per-depth scheduling * Add ambiguity search profiles * Add ambiguity search profiles * Add ambiguity search profiles * Add ambiguity search profiles * Add ambiguity search profiles * Add ambiguity search profiles * Add ambiguity search profiles * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * Use singular tool names * fix file permissions * make reports/ * Show live ambiguity search progress * Show live ambiguity search progress * Fix live progress record typing * Fix live progress record typing * Fix progress record type inference * Fix progress depth type inference * new report
* ambiguity: search by terminal equivalence class, not per token The bounded search branched on every terminal at each frontier, so interchangeable tokens (STRING/FLOAT/TRUE/FALSE as a primary, +/- as an operator, ...) each spawned a full isomorphic subtree that only reconverged after the token reduced to a nonterminal. The dedup cache could not collapse them because their post-shift frontiers sit on distinct LR states. Compute terminal equivalence classes up front: a two-step partition refinement (a state bisimulation over recognition behaviour, then a grouping of terminals by their per-state action up to that partition) puts two terminals in one class exactly when swapping them is an automorphism of the recognition relation. The search, the parallel split, and the prover then explore one representative per class. Because class members generate isomorphic parse forests, an ambiguous sentence exists with one iff it exists with every member, so a completed bound remains a theorem up to renaming terminals within their class and no obligation is lost. On the current grammar this collapses 56 terminals to 45 classes and roughly halves the frontiers explored at a given depth; exhaustive runs agree with the previous behaviour up to the representative substitution. Add `ambiguity classes` (engine flag --dump-terminal-classes) to list the classes, and document the change. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01NkitqNXZZsXuEKfwK3ajMc * ambiguity: test the prove path through a non-representative terminal Add an engine-level regression test for the terminal equivalence-class machinery, per CodeRabbit review on #45. A tiny ambiguous grammar with two interchangeable atoms A and B (one class, A as representative) checks that: the class is detected, an all-B sentence is still recognized as ambiguous even though the search only ever shifts A, and prove concretizes the ambiguity via the representative. Skips when the engine binary or menhir is unavailable so the pure-Python suite still runs without the OCaml toolchain. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01NkitqNXZZsXuEKfwK3ajMc * ambiguity: harden the terminal-class engine tests Address CodeRabbit review on #45: give the engine subprocess a process-level timeout so a hung binary or menhir cannot block the suite, and assert the prove witness is spelled with the class representative A (never the non-representative B) so the test would catch concretization drifting off representatives. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01NkitqNXZZsXuEKfwK3ajMc --------- Co-authored-by: Claude <[email protected]>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: zane-lang/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parser for Zane from scratch.