v0.8 generalization round: broad-distribution measurement + admit-nothing recommendation#94
Merged
Conversation
…on baseline
Replace the spent, co-evolved dev corpus with a generator-based, train/dev-split
task distribution and measure the CURRENT-language out-of-sample static
compression across it (the v0.8 round's optimization target).
- families.rs: add two oracle-verified families covering the sealed-flagged
uncovered shapes (BASELINE-SEALED §6):
* `bitdigit` (tier-0 scalar): popcount, digit-sum base b (2..16),
digit-product base b (8/10/16) — seeded from seal_count_set_bits /
seal_digit_product, generalized over the base literal.
* `scan` (tier-2 list): alternating sum, count local maxima, max adjacent
diff, dedup adjacent, RLE flatten, min running balance — each seeded
verbatim from the matching bench/sealed/corpus solution, re-verified
against a fresh (non-sealed) checked_* reference grid. No Value::Vec /
Value::Str; lists via existing fold/linrec/cons machinery.
- bin/broad.rs: shape emitter. Sweeps seeds, gates every tier-0/2 instance,
dedups by canonical program, assigns a reproducible seed-independent
TRAIN/DEV split (canonical-SHA-256 nibble parity), tags each shape with a
per-family template key + args. 1145 distinct shapes, 0 unclassified.
- design-v0.8/measure.py: per-family idiomatic-Python templates (v0.2 fairness
bar; scan templates byte-identical to the sealed idiomatic refs), same o200k
tokenizer both sides. Method labels: MTL=synth, Py=template.
- Result: new uncovered shapes compress ~1.7x (TRAIN 1.70x / DEV 1.72x),
reproducing the sealed held-out 1.67x; full-distribution micro 3.25x/3.24x is
an arithmetic-domination artifact (968/1145 affine one-liners), not
generalization. Per-shape ratios reproduce the sealed numbers exactly
(alt_sum 4.33x, dedup 1.85x, min_running_balance 1.86x, popcount 0.68x).
List-output running max/min recorded as an expressibility edge finding.
- oracle_gate.rs: v08_new_families_all_gate asserts every new-family instance
passes the real oracle. Workspace: 311 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_0124dqBiHMYYZMGoRvx3bSjx
…commendation Simulator-verified windowed-fold rewrites of the two windowed scan tasks (local_maxima 43->8/10 tok, max_adj_diff 42->17/19 tok; juggle 20->3, 21->7), projecting new-families out-of-sample 1.70/1.72x -> ~2.0/1.92x. State-machine (rle) and accumulator (min_running) sinks unaddressed. popcount = benchmark-fit; doc-fixes = 0 compression. Primary recommendation: admit nothing, ceiling ~1.7x. No proven crate touched; cargo test --workspace 311/0. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0124dqBiHMYYZMGoRvx3bSjx
Round writeup for the generalization round: diagnoses the out-of-sample compression gap (sealed 1.67x o200k / 1.72x cl100k vs in-sample 3.87x), apportions the 141-token shortfall via the sink taxonomy (stack-juggling ~59%, missing-idiom ~23%, accumulator ~10%, residual ~8%), corrects the input-encoding premise (label with 0 solution-token effect), and folds in the generator-based train/dev broad-distribution baseline (new families 1.70x TRAIN / 1.72x DEV, reproducing the sealed number). Recommendation: admit nothing; restate thesis at the measured out-of-sample ceiling ~1.7x, with a conditional windowed-fold aperture `w` fallback (verified, checker-typeable, ~2.0x) reserved for a scan-heavy niche. Specifies (does not execute) the fresh re-seal protocol for the next round, encoding the PR #91 lessons (clean barrier, no committed solutions, taxonomy-balanced authoring, ~1.7x pass/fail band, mkseal salted manifest). No proven mtl-core/mtl-syntax/mtl-check modules touched; changes confined to docs/. Workspace: 311 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_0124dqBiHMYYZMGoRvx3bSjx
madmax983
marked this pull request as ready for review
July 15, 2026 20:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this round found
The ≥3× static-compression gate was only ever measured in-sample. It hit 3.87× on the tier-2 dev corpus, but that corpus had co-evolved with primitive admission and the number does not survive out-of-sample: the blind sealed set (#91) came in at 1.67× o200k / 1.72× cl100k.
This round diagnoses why, builds a generalization-hygienic measurement corpus to replace the spent dev corpus, evaluates candidate fixes, and recommends how to restate the thesis.
Before / After
Before. The language thesis claimed ≥3× static token compression. The dev corpus that "proved" it had co-evolved with primitive admission, so the gate was measuring memorization, not generalization.
After. A generator-based TRAIN/DEV-split corpus (1145 deduped shapes) measures the honest out-of-sample compression at ~1.7× on genuinely-uncovered task shapes — TRAIN 1.70× / DEV 1.72×, agreeing to ≤0.02×, and reproducing the sealed 1.67×. The recommendation is to admit no new primitives and restate the thesis at the measured ceiling.
Sink taxonomy
Decomposition of the 141-token shortfall from the 3× budget:
The post-mortem's "input-encoding artifact" premise is corrected: it is a text-feedability label with zero solution-token effect. Against fair terse Python, MTL compresses only ~1.03×. Its real out-of-sample value is agent writability (100% pass@5) and marginal per-solution economics (CSPM 2.124×), not static compression.
Candidates evaluated & measured
w(literal-k, per Admit pick/roll stack-juggling primitives via token accounting #50) — oracle-verified. Cuts windowed-scan juggle glyphs (20→3, 21→7) and disproves the docs: v0.6 indexed-access design — measured decision (decline core change) #41 "juggling-bound" hypothesis for windowed scans. But it only lifts 2/1145 shapes to ~2.0× and does not amortize its cold-quickref + proof cost.Recommendation: admit nothing; out-of-sample ceiling ~1.7×. A conditional
wfallback applies only if the maintainer re-scopes to a scan/window-heavy niche.Re-seal protocol (specified, not executed)
The next round runs this:
How (implementation)
bitdigit+scan) inbench/dataset/src/families.rs, honoring the decision record (noValue::Vec/Value::Str).bench/dataset/src/bin/broad.rs+measure.py) with a canonical-SHA-nibble-parity TRAIN/DEV split.bench/design-v0.8/.docs/design/v0.8-generalization.md.mtl-core/mtl-syntax/mtl-checkmodule touched.cargo test --workspacegreen at 311/0 (+1 new oracle-gate test).Generated by Claude Code