Runnable int8 demo (demo/): mechanism + corpus build handoff

[lukefwalton]

What this is

A thin scaling/ module that makes the paper's §6 claim runnable: the same gold suite that owns grounding and refusal also adjudicates the int8 cost lever. It quantizes the embedding index to int8, re-ranks, and runs the full gold suite (including must-refuse and must-route) against the quantized index.

The result it is built to produce is a caught failure: at --bits 4 a route case flips (the private note loses the top slot to a public record) and the gold suite catches it. "int8 held" alone proves little; the gate saying no when pushed is the point.

What is committed vs pending a build run

This session's egress allowed only GitHub (api.openai.com, Gutenberg, archive.org all returned host_not_allowed) and no OPENAI_API_KEY was set. So:

• Committed and green now: the quantizer, the harness, the keyless runner, the keyed build script, the gold set, the corpus provenance manifest, and deterministic fixture tests — including the int4 route-flip payload proven on synthetic geometry.
• Pending a build run (local agent with network + key): the real text bodies and the committed vectors (index.json, query-vectors.json). Exact steps in docs/scaling-demo/build-handoff.md. Nothing is faked; the demo halts honestly at the network boundary.

Budget rule: held, no halt

The int8 path is a quantize wrapper plus a re-rank. It reuses src/retrieve.ts (retrieve/cosine), the no-leak boundary, the gold judges, src/store.ts, and the corpus loaders untouched — no second pipeline, no core type changes. quantize.ts is the public twin of the production vector-quant.ts.

Two design points worth a look (both in the delta log):

• The keyless headline needs committed gold-query vectors, not just FP vectors, because the core eval CLI embeds queries at run time (so it always needs a key). query-vectors.json + a thin runner solve this; the eval CLI is not reused verbatim.
• judgeRetrieval checks presence only, so a top-slot route flip where the note stays retrieved would slip through; the harness adds a keyless route-selection check to catch it.

What to review

• scaling/ — the module (README, config, quantize.ts, harness.ts, run.ts, build.ts, gold, corpus manifest, the quarantined spire).
• scaling/quantize.test.ts — the mechanism + the deliberate failure, offline.
• docs/scaling-demo/scaling-demo-delta-log.md — what's settled vs pending, the divergences, and the prepared NEXT-STEPS C-intro/C1 reconciliation (held until scaling:run confirms the headline, so "runnable" is verified not asserted).
• docs/scaling-demo/build-handoff.md — the brief for the local agent.

Test plan

• npm test → 36 pass (25 core + 11 scaling); npm run typecheck clean.
• npm run scaling:run degrades cleanly with a pointer to the handoff when vectors aren't built.
• Local agent runs build-handoff.md to generate vectors, confirm the int8 headline, fire the int4 break, then apply the NEXT-STEPS reconciliation.

Base / scope note

main is at v1.2.0, so this branch is 58 commits ahead (the prior v1.3–v1.5 work plus these 5 scaling commits). The scaling contribution is the scaling/ directory and the two docs/scaling-demo/ files above (plus tsconfig.json/package.json wiring); the rest of the diff is the accumulated work main has not yet received. Retarget the base if you want a scaling-only diff.

https://claude.ai/code/session_01EhtDe3ZQnv6vx2qSLy8qQ1

---

Generated by Claude Code

[surmado-code-review]

Automated Checks (advisory, non-blocking)

• ❌ Hallucinated package — node:test used in demo/quantize.test.ts does not exist on npm. This package appears to be invented by AI.
  ✅ No other issues detected.

---

Standards Compliance

One repo-standards issue still looks unresolved after the scaling/ → demo/ rename, and it’s the main thing I’d want the reviewer to settle before merge:

• demo/config.ts + demo/build.ts still produce commit-intended artifacts from the private layer, which conflicts with the repo’s artifact boundary rule.
  Relevant code:

  privateNotesDir: './demo/corpus/private',

  const naturalNotes = buildPrivateNotes(config);

  const naturalEntries = [...recordEntries(config, vectors), ...noteEntries(naturalNotes, vectors)]
  writeIndexFile(naturalEntries, NATURAL_INDEX);

  console.log('Done. Commit the *.json artifacts, then `npm run demo:run`.');

  and the note entries themselves are serialized as:

  sourceType: 'note',
  note,

  The loaded standards are explicit in §5: “Don't leak private embeddings/text into committed artifacts. (The index is gitignored for a reason.)”
  This PR still builds demo/corpus/index.json / index.synthetic.json from privateNotesDir and explicitly instructs committing those JSON files. Even if runtime answering still respects assembleEvidence and never sends note prose to the model, this is a standards conflict at artifact creation time.

  I don’t see a standards change in this diff that authorizes that exception, so as written this is still a repo-policy violation, not just a docs choice.

Summary

This PR adds a standalone demo/ flow for quantizing the corpus index, re-ranking against the quantized vectors, and adjudicating a gold suite keylessly via committed query vectors, with an optional keyed --full answer-mode pass. The implementation mostly stays thin and reuses core retrieval/eval/store logic; the main review decision is whether demo/build.ts is allowed to create committed note-derived artifacts from demo/corpus/private.

Reviewer: most of the risk is in demo/build.ts / demo/config.ts and the standards exception they imply; the quantization harness itself looks comparatively low-risk.

What to pay attention to

• demo/build.ts + demo/config.ts: this is where the standards boundary is actually enforced or broken. The concern is not “does runtime leak private prose?”; it’s “are we committing note-derived artifacts at all?”
• demo/run.ts --full: worth a quick sanity check in a good way — it now passes requantizeIndex(index, args.bits) into the answer pass, which avoids masking a lossy-index flip with full-precision retrieval.
• demo/harness.ts top-slot logic: this is the substantive new gate behavior. judgeRetrieval only checks presence, so this extra top-slot check is what makes route/disambiguation flips visible.

Things I noticed

🔴 Red flags — fix before merge:

• Committed private-layer artifacts still violate the repo standards.
  In demo/build.ts, the natural index is built from both public records and private notes:

  const naturalEntries = [...recordEntries(config, vectors), ...noteEntries(naturalNotes, vectors)]
  writeIndexFile(naturalEntries, NATURAL_INDEX);

  combined with:

  privateNotesDir: './demo/corpus/private',

  and:

  console.log('Done. Commit the *.json artifacts, then `npm run demo:run`.');

  That will produce commit-intended JSON derived from the repo’s private layer, which conflicts directly with §5.

🟡 Yellow flags — consider for this PR or a follow-up:

• There’s still no hard guard around that exception path. Even if the team decides this demo deserves a narrow carve-out, demo:build writes note-derived commit artifacts as the default path. An explicit opt-in or hard fail would make accidental boundary drift much less likely.

Good patterns

• demo/run.ts keeps the keyed answer-mode pass aligned with the same quantized retrieval surface by calling:

  await runAnswerPass(gold, requantizeIndex(index, args.bits), qv.byId, config);

  That avoids a subtle false-pass where retrieval is judged on lossy vectors but answers are generated from full-precision evidence.
• demo/query-vectors.ts handles malformed or mismatched artifacts as loud failures with rebuild guidance, which matches the repo’s error-handling standard well.

Suggested improvements

1. Make demo:build refuse to write commit-intended artifacts containing note-derived entries unless the repo standards are explicitly updated to allow that case.
2. If no standards exception is intended, keep committed demo artifacts public-only and make note-derived vectors local/uncommitted.
3. Once policy is settled, add a regression guard around the artifact boundary so future demo changes can’t silently reintroduce committed private-layer outputs.

Questions for the author

• Is the intent here to establish a repo-level exception for demo/corpus/*.json containing private-layer embeddings, or should the demo be reshaped so committed artifacts stay public-only?

---

Surmado Code Review (v1.2-mt) is an automated review, designed to work alongside human judgment.

Want to change your STANDARDS.md or YML? Edit it directly, or tune it with our AI agent Scout.

Comment /rerun-review on this PR to refresh the review — costs 1 additional PR credit.