Mac K2.A reviewer: split research A/B vs product-shape smoke (per user directive)

Per user directive 2026-06-09:
  '在Mac mini上的k2.a.1的测试脚本严重浪费时间。跑大量无用用例，
   而且完全不顾及真是的用户对时延的体验。按照上边的建议进行修改。'

Diagnosis the user reached:
  K2.A.1's product question is binary correctness — does KL ON
  preserve recall? Is the bf16 dtype crash fixed? — not a
  statistical research question. The Mac M4 A/B reviewer
  (review_pr_k2a1_integration_on_mac.sh) takes 7-9h to answer
  the wrong question for the product:

    * 20 samples ≠ 1 user request
    * KL OFF / oracle / v0.3 baselines are research-only — they
      are NOT on the production path (KL ON + K2.A.2 stateful)
    * mean throughput across 20 samples masks first-token latency
      that users actually feel

User's product-aligned shape (verbatim):
    * Don't run oracle / v0.3 / A/B; only one production arm
    * Don't run 20 samples; single request
    * K2.A.2 stateful caching, no per-step recompression
    * Eventually MLX/Metal — not the current PyTorch MPS research
      harness (acknowledged this is a K3 deliverable)
    * Long history: incremental update only; do not rebuild
      full context every step

Three changes in this PR:

1. NEW scripts/review_pr_k2a_production_smoke_on_mac.sh
   ~120 LOC. Single-request K2.A production-shape smoke.

   Wraps scripts/research/k1e_niah_validation.py with the
   product-aligned constraint set:

     --n-samples 1            single request, NOT 20
     --skip-oracle            no oracle arm
     --skip-v03               no v0.3 arm
     --kl-on                  KL compression on (production path)
     --stateful               K2.A.2 incremental decode (default ON)

   Default 5.6k context (CTX_LINES=280); ~3-5 min on Mac M4
   24 GB. Reports recall hit/miss, prefill latency, first-token
   latency, mean tok/s, peak resident memory. Time budget
   listed for 1.4k / 5.6k / 16k context rungs so the user
   can pick what matches their request distribution.

   Explicitly documents what it does NOT validate:
     - statistical recall delta (that's the A/B's job)
     - throughput parity vs full-attention oracle (gate (c) work
       for K2.A.2 + K3)
     - MLX/Metal native latency — this is PyTorch MPS, the
       conservative upper bound; K3 MLX path will be faster

2. AMENDED scripts/review_pr_k2a1_integration_on_mac.sh
   Two banner additions:

   (a) Header docstring ('READ THIS BEFORE RUNNING'): explains
       this is a 7-9h RESEARCH A/B for PR-K2.A.1 binding gate
       (b) recall delta evidence — NOT a product-latency signal.
       Points users to review_pr_k2a_production_smoke_on_mac.sh
       if that's what they wanted.

   (b) Runtime banner with 10-second sleep: prints the same
       warning as the script starts running, with a Ctrl-C
       window so users who launched the wrong script can abort
       before they've burned hours of M4 time.

   The A/B itself is unchanged in semantics — it remains the
   binding evidence collector for PR-K2.A.1 merge (gate (b)
   recall delta ≤ 1pp at every rung, ADR §11.8 criterion 1a).
   Cannot be deleted; can be made impossible to run by accident.

3. AMENDED docs/adr/0008-session-bound-runtime-and-grpc-protocol.md
   §11.11 K2.A.1 reviewer-scripts subsection now lists three
   reviewer scripts (vast A/B, Mac A/B, Mac product smoke) with
   a 'scope split' table making explicit:

     A/B script              binding gate (b) evidence    ~7-9h
     production smoke        user-facing latency + recall  ~3-5min

   Plus a paragraph (recorded as the user's directive)
   stating that the A/B's KL OFF / oracle / v0.3 arms are
   research-only and 'must not be conflated' with product
   validation. Future contributors can read this section
   instead of relearning the lesson.

Tests:
  tests/inference_engine/v04/: 253 passed (no v04 code changes).
  bash -n syntax check on both Mac reviewer scripts: OK.

Compat:
  * No code-path changes. K1.E runner (k1e_niah_validation.py)
    is untouched; both scripts use its existing flags
    (--n-samples / --skip-oracle / --skip-v03 / --kl-on /
    --stateful / --kl-lattice / --kl-q-range).
  * The A/B reviewer's default behaviour is unchanged — same
    flags, same time budget. Only the banner differs.
  * The new product smoke script is additive; it cannot affect
    the A/B's binding gate evidence path.

Stack:
  Off main, parallel to (does not depend on, does not block):
    AgentMemory/v04-pr-k3-block-b-prereqs-1-4-dflash-loader-8e7f
    (K3 prereq 1+4 work, currently stashed; will resume after this)

Net effect:
  Mac M4 reviewers now answer two distinct questions cleanly.
  Users wanting the 'does this work for my latency' answer get
  it in ~3-5 min instead of being told to run a 7-9h script
  whose stated purpose is statistical research evidence.

Co-authored-by: FluffyAIcode <[email protected]>

Author: cursoragent

docs/adr/0008-session-bound-runtime-and-grpc-protocol.md

@@ -2293,10 +2293,40 @@ scripts + tests).** What it delivers:
   schema bumps 4 → 5 to record the KL config block.
 * Reviewer scripts:
   - `scripts/review_pr_k2a1_integration_on_vast.sh` — vast.ai
-    CUDA A/B at the §11.12 ladder.
+    CUDA A/B at the §11.12 ladder. **Research evidence
+    collector** (statistical, ~hours).
   - `scripts/review_pr_k2a1_integration_on_mac.sh` — Mac M4
     (PyTorch MPS) A/B at the small-end §11.12 rungs (1.4k +
-    5.6k by default).
+    5.6k by default). **Research evidence collector**
+    (statistical, ~7-9h). Banner at runtime warns users who
+    ran it expecting product-shape latency.
+  - `scripts/review_pr_k2a_production_smoke_on_mac.sh` —
+    **product-shape smoke** (added 2026-06-09 per user
+    directive). Single request, KL ON + K2.A.2 stateful only,
+    no oracle / v0.3 / KL OFF arms, no statistical averaging.
+    Reports first-token latency, recall hit/miss, peak resident
+    memory. ~3-5 min @ 5.6k context on Mac M4 24 GB.
+
+**Scope split (recorded 2026-06-09)**: research A/B and
+product-shape smoke answer different questions and **must
+not be conflated**:
+
+| Script | Question | Time |
+|---|---|---|
+| `..._k2a1_integration_on_mac.sh`   | Statistical recall delta (ADR §11.8 1a binding gate)        | ~7-9h  |
+| `..._k2a_production_smoke_on_mac.sh` | User-facing first-token latency + recall hit + dtype crash | ~3-5min |
+
+The A/B is necessary for PR-K2.A.1 merge evidence (binding
+gate (b) recall delta ≤ 1pp at every rung needs sample
+distribution). The product smoke is necessary for honest
+release-readiness signal — it answers "if a user sends one
+request through this stack on Mac, what do they wait for".
+Mean throughput across 20 samples masks first-token latency
+that users actually feel; the A/B's KL OFF / oracle / v0.3
+arms are not on the production path; running them as a
+proxy for product validation **wastes time and does not
+produce the answer the question is asking**. Per the user's
+directive: do not use the A/B as a product-experience signal.
 
 K2.A.1 acceptance gates (per §11.11.5 above): **gate (a)
 round-trip identity** is closed by the K2.A.0 Mac smoke

scripts/review_pr_k2a1_integration_on_mac.sh

@@ -1,6 +1,36 @@
 #!/usr/bin/env bash
-# Mac M4 reviewer aid for PR-K2.A.1 — KakeyaLattice integration
-# A/B at the §11.12 ladder rungs Mac M4 24 GB can handle.
+# Mac M4 RESEARCH A/B for PR-K2.A.1 KakeyaLattice integration —
+# §11.12 ladder rungs Mac M4 24 GB can handle.
+#
+# ============================================================
+# READ THIS BEFORE RUNNING (added 2026-06-09 per user directive)
+# ============================================================
+#
+# This is a research evidence collector with a 7-9h time budget.
+# It runs 20 samples × 2 context rungs × {KL OFF, KL ON} × {oracle,
+# v0.3, v0.4} arms because the BINDING gate for PR-K2.A.1 merge
+# (recall delta ≤ 1pp at every rung — see ADR §11.8 criterion 1a)
+# needs statistical signal across a sample distribution.
+#
+# It deliberately does NOT reflect user-facing latency:
+#   * 20 samples ≠ 1 user request
+#   * KL OFF / oracle / v0.3 baselines are research-only — the
+#     production path is KL ON + K2.A.2 stateful (incremental
+#     decode), not stateless A/B
+#   * mean throughput across 20 samples masks first-token latency
+#
+# If you want to measure user-facing latency for the K2.A
+# production path, run instead:
+#
+#   bash scripts/review_pr_k2a_production_smoke_on_mac.sh
+#
+# (single request, KL ON + stateful, ~3-5 min on Mac M4 24 GB at
+# 5.6k context. Report includes first-token latency and dtype-crash
+# regression check.)
+#
+# Continue reading only if you actually need the A/B research
+# evidence for PR-K2.A.1 merge (gate (b) recall delta).
+# ============================================================
 #
 # Mirror of scripts/review_pr_k2a1_integration_on_vast.sh, but
 # scoped to Mac M4 budget: default ladder '70 280' (~1.4k +
@@ -166,7 +196,20 @@ run_one_rung() {
     fi
 }
 
-echo "==> PR-K2.A.1 KakeyaLattice integration A/B — Mac M4"
+cat <<'BANNER'
+==> PR-K2.A.1 KakeyaLattice integration A/B — Mac M4
+    +------------------------------------------------------+
+    | This is a RESEARCH A/B (~7-9h on Mac M4 24 GB).      |
+    | It does NOT reflect user-facing latency.             |
+    |                                                      |
+    | For product-shape latency on Mac, run instead:       |
+    |   bash scripts/review_pr_k2a_production_smoke_on_mac.sh |
+    |                                                      |
+    | Press Ctrl-C in the next 10 s to abort if you meant  |
+    | the production smoke.                                |
+    +------------------------------------------------------+
+BANNER
+sleep 10
 echo "    Model:           google/gemma-3-1b-it"
 echo "    Samples / arm:   $N_SAMPLES"
 echo "    Sink x window:   ${SINK} x ${WINDOW}"

scripts/review_pr_k2a_production_smoke_on_mac.sh

@@ -0,0 +1,216 @@
+#!/usr/bin/env bash
+# Mac M4 K2.A production-shape smoke (per user directive 2026-06-09).
+#
+# This script answers the **product-aligned** question: under a
+# realistic single-user single-request shape, does the K2.A KL ON +
+# K2.A.2 stateful path deliver acceptable behaviour on Mac M4?
+#
+# Specifically:
+#
+#   * Recall hits the needle once per request (not statistical
+#     averaging over 20 samples).
+#   * Per-step dtype crash (the bf16 / fp32 round-trip failure
+#     fixed by PR #87) does NOT recur on MPS bf16.
+#   * First-token latency is recorded so we can see what the user
+#     actually waits for, not just a research mean throughput.
+#
+# What this script does NOT validate (deliberately):
+#
+#   * Statistical recall delta over a sample distribution. That's
+#     a research A/B question, answered by
+#     scripts/review_pr_k2a1_integration_on_mac.sh (~7-9h, BINDING
+#     gate evidence for PR-K2.A.1 merge).
+#   * Throughput parity vs full-attention oracle (no oracle arm
+#     run here — gate (c) work that K2.A.2 + K3 close).
+#   * MLX/Metal native path latency. v0.4 today is PyTorch MPS
+#     research harness; the user-facing latency floor will only
+#     drop when K3 ships the MLX path. This smoke nonetheless
+#     records the PyTorch MPS number as a CONSERVATIVE upper bound
+#     on what users experience: anything K3 ships will be faster.
+#
+# Usage
+# -----
+#
+# Default — single 5.6k-context request, KL ON + stateful, ~3-5 min
+# on Mac M4 24 GB:
+#
+#   bash scripts/review_pr_k2a_production_smoke_on_mac.sh
+#
+# Single 1.4k request (~30-60 s):
+#
+#   CTX_LINES=70 bash scripts/review_pr_k2a_production_smoke_on_mac.sh
+#
+# Wider context (slower, may OOM on 24 GB):
+#
+#   CTX_LINES=800 bash scripts/review_pr_k2a_production_smoke_on_mac.sh
+#
+# Env knobs (defaults):
+#
+#   CTX_LINES         (280)       padding-line count; 70=1.4k, 280=5.6k,
+#                                 800=16k tokens
+#   SINK              (4)
+#   WINDOW            (64)
+#   MAX_NEW_TOKENS    (24)
+#   SEED              (42)
+#   KL_LATTICE        (D4)        D4 (v1.4) or E8 (v1.5)
+#   KL_Q_RANGE        (38)        canonical D4 operating point
+#   ATTN_IMPL         (sdpa)      memory-efficient; required for >=4k Mac M4
+#   STATEFUL          (1)         1 = K2.A.2 (default; production path);
+#                                 0 = K2.A.1 stateless (each-step recompute,
+#                                     research-only — DO NOT ship to users)
+#
+# Output
+# ------
+#
+# Single JSON report at:
+#
+#   results/research/k2a_production_smoke_mac_ctx<CTX_LINES>_<stamp>.json
+#
+# JSON contains: recall (hit/miss for the single request), prefill
+# latency, first-token latency, mean tok/s, peak resident memory,
+# effective attention window, dtype/numerics warnings if any.
+#
+# Pre-flight
+# ----------
+#
+# kakeyalattice must be installed (for KL ON arm). PyTorch with MPS
+# support must be importable. HF_TOKEN must be exported (Gemma 3-1B
+# is gated):
+#
+#   export HF_TOKEN=hf_xxx
+#   pip install kakeyalattice
+#
+# Lineage (why this script exists alongside the A/B script)
+# ---------------------------------------------------------
+#
+# The A/B reviewer scripts/review_pr_k2a1_integration_on_mac.sh is a
+# **research evidence collector** for PR-K2.A.1 merging — it takes
+# 7-9h, runs 20 samples × 2 context rungs × {KL OFF, KL ON} × {oracle,
+# v0.3, v0.4} arms because the BINDING gate for that PR (recall delta
+# ≤ 1pp at every rung) needs statistical signal.
+#
+# That A/B shape is wrong for the **product** question — users send
+# one request, expect one answer, and care about the latency of that
+# one request. Per the user's 2026-06-09 directive ("严重浪费时间..."):
+# do not run oracle / v0.3 / A/B for product-experience verification;
+# run a single production arm (KL ON + stateful), single request,
+# report user-facing latency.
+#
+# Both scripts have a place — research signal vs product signal.
+# This file is the product-signal entry point.
+
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+
+CTX_LINES="${CTX_LINES:-280}"
+SINK="${SINK:-4}"
+WINDOW="${WINDOW:-64}"
+MAX_NEW_TOKENS="${MAX_NEW_TOKENS:-24}"
+SEED="${SEED:-42}"
+ATTN_IMPL="${ATTN_IMPL:-sdpa}"
+KL_LATTICE="${KL_LATTICE:-D4}"
+KL_Q_RANGE="${KL_Q_RANGE:-38}"
+STATEFUL="${STATEFUL:-1}"
+
+stamp="$(date +%s)"
+out_dir="results/research"
+log_dir="${out_dir}/logs"
+mkdir -p "$out_dir" "$log_dir"
+
+# ±15 % range around the target line count — same formula as A/B
+# script so the haystack length distribution matches research evidence.
+lo=$(( (CTX_LINES * 85 + 50) / 100 ))
+hi=$(( (CTX_LINES * 115 + 50) / 100 ))
+if [[ $lo -lt 10 ]]; then lo=10; fi
+if [[ $hi -lt $((lo + 1)) ]]; then hi=$((lo + 1)); fi
+
+report="${out_dir}/k2a_production_smoke_mac_ctx${CTX_LINES}_${stamp}.json"
+log="${log_dir}/k2a_production_smoke_mac_ctx${CTX_LINES}_${stamp}.log"
+
+echo "==> Mac M4 K2.A production-shape smoke (single request)"
+echo "    Model:           google/gemma-3-1b-it"
+echo "    Context:         ~$(( CTX_LINES * 20 )) tokens (lines [$lo, $hi])"
+echo "    Sink x window:   ${SINK} x ${WINDOW}"
+echo "    Attn impl:       $ATTN_IMPL"
+echo "    Lattice:         $KL_LATTICE  (Q=$KL_Q_RANGE)"
+echo "    Stateful (K2.A.2): $STATEFUL"
+echo "    Samples:         1   (product shape, NOT 20)"
+echo "    Arms:            v0.4 KL ON only (NO oracle, NO v0.3, NO KL OFF)"
+echo "    Time budget:     ~30 s @ 1.4k, ~3-5 min @ 5.6k, ~12-18 min @ 16k"
+echo "    Report:          $report"
+echo
+
+echo "==> Pre-flight: kakeyalattice availability"
+if PYTHONPATH=.:sdks/python python3 -c "import kakeyalattice; print(' kakeyalattice', kakeyalattice.__name__, 'OK')" 2>/dev/null; then
+    echo "    kakeyalattice installed."
+else
+    echo "    ERROR: kakeyalattice NOT installed. Install:"
+    echo "        pip install kakeyalattice"
+    exit 1
+fi
+echo
+
+# Build the K1.E runner invocation. Production-shape constraints:
+#
+#   --n-samples 1        single request, not 20
+#   --skip-oracle        no oracle arm (research-only baseline)
+#   --skip-v03           no v0.3 arm (research-only baseline)
+#   --kl-on              production arm uses KL compression
+#   (no --kl-on toggle for KL OFF)
+#   --stateful           K2.A.2 incremental decode path (production
+#                        path; without this each step recompresses
+#                        the whole resident window, which is what
+#                        the user's directive specifically forbids)
+#
+flags=(
+    --model google/gemma-3-1b-it
+    --device auto
+    --attn-impl "$ATTN_IMPL"
+    --n-samples 1
+    --haystack-min-lines "$lo"
+    --haystack-max-lines "$hi"
+    --sink-size "$SINK"
+    --window-size "$WINDOW"
+    --max-new-tokens "$MAX_NEW_TOKENS"
+    --seed "$SEED"
+    --skip-oracle
+    --skip-v03
+    --kl-on
+    --kl-lattice "$KL_LATTICE"
+    --kl-q-range "$KL_Q_RANGE"
+    --output "$report"
+)
+[[ "$STATEFUL" == "1" ]] && flags+=(--stateful)
+
+echo "==> Running v0.4 KL ON $([[ "$STATEFUL" == "1" ]] && echo "+ stateful (K2.A.2)") on a single request"
+PYTHONPATH=.:sdks/python python3 scripts/research/k1e_niah_validation.py \
+    "${flags[@]}" 2>&1 | tee "$log"
+exit_code=${PIPESTATUS[0]}
+
+echo
+if [[ "$exit_code" -eq 0 ]]; then
+    echo "==> Production smoke OK."
+    echo "    Report:  $report"
+    echo "    Log:     $log"
+    echo
+    echo "Inspect first-token latency / recall:"
+    echo "    python3 -c 'import json,sys;r=json.load(open(\"$report\"));"
+    echo "        v=r[\"per_config\"].get(\"v0.4\");"
+    echo "        print(\"recall:\", v[\"recall\"]);"
+    echo "        print(\"first_token_s:\", v.get(\"first_token_seconds\"));"
+    echo "        print(\"mean_tok_s:\", v.get(\"mean_throughput_tokens_per_sec\"));"
+    echo "        print(\"peak_mem_GB:\", v.get(\"peak_memory_bytes\",0)/1e9)'"
+    echo
+    echo "Commit (only if recall=hit AND no dtype crash):"
+    echo "    git add $report $log"
+    echo "    git commit -m 'Mac M4 K2.A production-shape smoke evidence'"
+    echo "    git push"
+else
+    echo "==> Production smoke FAILED (exit=$exit_code)."
+    echo "    Report (partial): $report"
+    echo "    Log:              $log"
+fi
+
+exit "$exit_code"