feat(fold): Tier D1 — bf16 merged-QKV default-ON across dense/coder/dflash/Gemma

Executes Tier D1 of the cross-arch merged-GEMM fold plan
(.agents/specs/arch-fusion-fold-plan-2026-07-30.md). The q/k/v projections that
SHARE the layer input collapse to ONE bf16 vt::MatmulBT over the merged
[Nq+Nk+Nv,H] owner + a contiguous vt::QkvSplit (the OLMo-2/Granite/StableLM
exemplar), replacing three per-shard GEMMs (folds the tiny GQA k/v GEMMs into one
wide tensor-core GEMM, -2 launches/layer).

The shared gate Qwen3QkvMergeEnabled() (alias MergedQkvEnabled(), env
VT_QWEN3_QKV_MERGE, dense_attn_block.h) FLIPPED default-ON. Consumers folded:
qwen3_dense (dense_attn_block.h), qwen3_coder (inherits via the shared dense
AttnBlock, qwen3_moe.cpp), qwen3_dflash (qwen3_dflash.cpp), Gemma-1/2/3/4
(gemma*.cpp). No loader concat — every consumer already packs the merged
qkv_proj owner. RoPE UNCHANGED (no RopeNeox->RopeFromCache swap; the compounding
second-1-ULP-flip hazard avoided). dflash_gguf (keep-quant merged instance) is
OUT of D1 scope. VT_QWEN3_QKV_MERGE=0 restores the byte-identical 3-shard A/B in
the same binary.

The merge is bit-exact GEMM math (wider N): tests/vt/test_ops_qkv_merge.cpp (CPU,
RED-first) byte-compares merged MatmulBT+QkvSplit == three separate MatmulBT
across GQA/MHA/MQA/wide-head shapes (a wrong split offset fails the check). The
ONLY byte effect is the downstream FA2 1-ULP on the 0.6B genuine bf16 near-tie,
so the whole family batched behind ONE 0.6B near-tie golden regen.

Golden regen (0.6B ONLY): the default flip shifted the 0.6B anchor at 2
divergence points (p5 tok11, p10 tok10; greedy cascades) -> regenerated via
scripts/qwen3-neartie-gap.py teacher-forcing the vLLM oracle, all tokens within
the near-tie band (max gap 0.125 nats). Qwen3-4B our_ids.npy UNCHANGED (0 diffs,
measured-neutral) + qwen3coder UNCHANGED (0 diffs).

GATED (DGX GB10 sm_121a, RelWithDebInfo + cutlass-4.5.0 + triton, one flock):
test_ops_qkv_merge 2/2*14; test_qwen3_paged_engine 0.6B 16/16 + 4B 16/16;
test_qwen3coder_paged_engine 6/6; test_gemma2_forward SACRED 48/48
(global+sliding); test_gemma4_paged_engine STRICT 32/32; shared-header canary
test_qwen27_paged_engine 235/235 UNCHANGED. All four record checkers rc=0.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

Author: mudler

.agents/specs/arch-fusion-fold-plan-2026-07-30.md

@@ -97,10 +97,12 @@ Extend the non-Marlin `MoeGroupedGemmNvfp4` path to the already-tuned fused-w13
 
 ### TIER D — reordering near-ties: consistency-only, gated, do AFTER the bit-exact tiers
 
-**D1. bf16 merged-QKV default flip → shared `MatmulBT`+`QkvSplit`** *(shared-op: merged-QKV, bf16 instance)*
-**Measured NEUTRAL on Qwen3-4B** (decode compute-bound) — this is a pure launch-count/consistency fold, not a speed win. Wider merged-N K-reduction flips one 0.6B near-tie token → must regen the SACRED near-tie golden. Same near-tie effect across all consumers, so batch them and regen once.
-- qwen3_dense `dense_attn_block.h:352-380` (`VT_QWEN3_QKV_MERGE`) · qwen3_coder `qwen3_moe.cpp:79` · qwen3_dflash `qwen3_dflash.cpp:293-308` · dflash_gguf `qwen3_dflash_gguf.cpp:235` · Gemma×4 (`gemma.cpp:59-65`, `gemma2.cpp:142-148`, `gemma3.cpp:130-136`, `gemma4.cpp:215-221`)
+**D1. bf16 merged-QKV default flip → shared `MatmulBT`+`QkvSplit`** *(shared-op: merged-QKV, bf16 instance)* — ✅ **DONE 2026-07-31** (branch `fold/d1-bf16-merged-qkv` off `08eec2a8`, NOT pushed).
+**Measured NEUTRAL on Qwen3-4B** (decode compute-bound) — a pure launch-count/consistency fold, not a speed win. The six bf16 consumers now default to ONE `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2/Granite/StableLM exemplar). The shared gate `Qwen3QkvMergeEnabled()` (alias `MergedQkvEnabled()`, env `VT_QWEN3_QKV_MERGE`) FLIPPED default-ON; `=0` restores the byte-identical 3-shard A/B in the same binary. **No loader concat needed** — every consumer already packs the merged `qkv_proj` owner. RoPE handling UNCHANGED (no RopeNeox→RopeFromCache swap — the second-1-ULP-flip hazard avoided). The merge is bit-exact GEMM math (wider N; per-output-row reduction identical — `tests/vt/test_ops_qkv_merge.cpp` CPU byte-identity, RED-first, 14/14); the ONLY byte effect is the downstream FA2 1-ULP on the 0.6B genuine bf16 near-tie, so the whole family batched behind ONE 0.6B near-tie golden regen.
+- ✅ qwen3_dense `dense_attn_block.h` (`Qwen3QkvMergeEnabled` default flipped ON) · ✅ qwen3_coder `qwen3_moe.cpp` (inherits via the shared dense `AttnBlock`) · ✅ qwen3_dflash `qwen3_dflash.cpp` · ✅ Gemma-1/2/3/4 (`gemma.cpp`, `gemma2.cpp`, `gemma3.cpp`, `gemma4.cpp`)
+- **NOT in D1 scope:** dflash_gguf `qwen3_dflash_gguf.cpp` is the KEEP-QUANT merged instance (routes via `kMatmulBTQuantGrouped`, S2), a separate follow-up — not the bf16 path this batch flips.
 - Reference: OLMo-2 `olmo2.cpp:98-103` shows the target form (already merged).
+- **GATED (DGX GB10 sm_121a, RelWithDebInfo + cutlass-4.5.0 + triton):** A/B unit `test_ops_qkv_merge` 2/2·14 (merged == 3-shard byte-identical, RED-first wrong-split fails); **golden regen shifted the 0.6B ANCHOR at exactly 2 divergence points** (prompt[5] tok11, prompt[10] tok10, then greedy cascades) — regenerated via `qwen3-neartie-gap.py` teacher-forcing the vLLM oracle, ALL our tokens within the near-tie band (max gap **0.125 nats** ≪ 0.5); `our_ids.npy` md5 `c849abce`→`13b59dd6`, `neartie_gap_mnats.npy` `f716ecce`→`b79989c5`. **Qwen3-4B our_ids.npy md5 UNCHANGED (0 token diffs — measured-neutral) + qwen3coder UNCHANGED (0 diffs).** Re-gate clean: `test_qwen3_paged_engine` 0.6B **16/16** (11 strict + 5 near-tie, max 0.125) + 4B **16/16** (11 strict + 5 near-tie, max 0.25) · `test_qwen3coder_paged_engine` **6/6** · `test_gemma2_forward` SACRED **48/48** (global+sliding) · `test_gemma4_paged_engine` STRICT **32/32** · shared-header canary `test_qwen27_paged_engine` **235/235** UNCHANGED (27B resident nvfp4 QKV — this bf16 path untouched).
 
 **D2. GDN in_proj merge (qkv+z along N)** *(shared-op: merged-QKV, GDN instance)*
 Near-tie, distributionally sensitive; GDN layers are the majority of the 35B hybrid so it compounds — but verify token-exact gate. `qwen3_5.cpp:2952-2958`.

.agents/state.md

@@ -34351,3 +34351,5 @@ this branch, so this branch adds zero device leakage.
 Evidence: [../docs/bench-evidence/qwen35-4b-postrebase-20260729.md](../docs/bench-evidence/qwen35-4b-postrebase-20260729.md).
 
 **Laguna-S-2.1 W1 oracle-decision + W2 structural bring-up (2026-07-30, `CLAIM-LAGUNA-W1W2`, base `main` `5c3da2f1`, CPU-only, foreground, NOT pushed).** `MODEL-TEXT-laguna-laguna-for-causal-lm` INVENTORIED→ACTIVE. Landed the ADDITIVE structural bring-up for Poolside `LagunaForCausalLM`/`laguna` (118B/8B MoE): registry TU (`laguna_registry.cpp`, one `REGISTER_VLLM_MODEL` line, KV-cache spec = one full-attention group with sliding masked at the kernel, gemma3 topology), `ParseLagunaParams` (`laguna_weights.cpp`) resolving every consumed field incl. the NESTED per-layer-type `rope_parameters` (the OLMo-3 KeyError hazard, handled) and the per-layer VARIABLE Q-head array, the GGUF `blk.N.*` name-map + the UD-Q4_K_XL per-role quant-mix enumeration (Q4_K/Q5_K/Q6_K/Q8_0 ALL already decoded ⇒ ZERO new decode kernel), and the per-layer forward COMPOSITION documented with reuse citations (ds2 MoE `deepseek_v2.cpp:340-365`, gemma3 sliding `gemma3_registry.cpp:103-121`, olmo3 YaRN `olmo2_weights.cpp:198-217`, `Softplusf` `gemma4_audio.cpp:20`) — the forward + loaders are honest `VT_CHECK(false)` W3 residuals. **W1 oracle DECISION (no big run):** vLLM native `laguna.py` is in the pin ⇒ config CONSTRUCTS (nested rope handled, transformers 5.14.1); dual-oracle viable — vLLM-NVFP4/-FP8 (fits GB10 119 GiB; BF16 235 GiB does NOT) for behavior + llama.cpp Poolside-fork `laguna` on the identical UD-Q4_K GGUF for the same-quant token-exact gate. Gates: `test_laguna_scaffold` 3/3·40 (arch resolves + config descends + loaders throw) + `test_model_registry` 24/24 (Laguna in the sorted supported set + error message); CPU full-library `-DVLLM_CPP_CUDA=OFF` Release `-Werror` build CLEAN; all four record checkers rc=0. RESIDUALS (W3/W4): device tower materialization (GGUF keep-quant + dual RoPE caches) + the real `LagunaModel::Forward` composing the reuse + the 3 new host ops (per-head softplus attn gate, ungrouped sigmoid-noaux router, variable-Q-head runner wiring) + the strict dual-oracle greedy gate on a fetched 73 GB checkpoint. No model ran, no download. Spec: [specs/laguna-s21-w1w2-2026-07-30.md](specs/laguna-s21-w1w2-2026-07-30.md). Not pushed; FULL SHA reported.
+
+**Tier-D1 bf16 merged-QKV default-ON across dense/coder/dflash/Gemma×4 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`, branch `fold/d1-bf16-merged-qkv` off `main` `08eec2a8`, foreground, NOT pushed).** Executed Tier-D1 of the cross-arch merged-GEMM fold plan (`.agents/specs/arch-fusion-fold-plan-2026-07-30.md`). The q/k/v projections that SHARE the layer input now collapse to ONE bf16 `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (the OLMo-2/Granite/StableLM exemplar), replacing three per-shard GEMMs. The shared gate `Qwen3QkvMergeEnabled()` (alias `MergedQkvEnabled()`, env `VT_QWEN3_QKV_MERGE`, `dense_attn_block.h`) FLIPPED default-ON; the six bf16 consumers fold behind it: qwen3_dense (`dense_attn_block.h`), qwen3_coder (inherits via the shared dense `AttnBlock`, `qwen3_moe.cpp`), qwen3_dflash (`qwen3_dflash.cpp`), Gemma-1/2/3/4 (`gemma*.cpp`). `=0` restores the byte-identical 3-shard A/B in the same binary. NO loader concat (every consumer already packs the merged `qkv_proj`); RoPE UNCHANGED (no RopeNeox→RopeFromCache swap — the compounding-second-1-ULP hazard avoided). dflash_gguf (keep-quant merged instance) is OUT of D1 scope. **A/B unit** `tests/vt/test_ops_qkv_merge.cpp` (CPU, RED-first): merged `MatmulBT`+`QkvSplit` == three separate `MatmulBT` BYTE-IDENTICAL across GQA/MHA/MQA/wide-head shapes (a wrong split offset fails the check) — 2/2·14. **The merge is bit-exact GEMM math (wider N); the ONLY byte effect is the downstream FA2 1-ULP on the 0.6B genuine bf16 near-tie, so the whole family batched behind ONE 0.6B golden regen.** Golden regen (0.6B ONLY): flipping the default shifted the 0.6B anchor at 2 divergence points (p5 tok11, p10 tok10; greedy cascades) — regenerated via `scripts/qwen3-neartie-gap.py` teacher-forcing the vLLM oracle, ALL tokens within the near-tie band (max gap 0.125 nats ≪ 0.5); `our_ids.npy` md5 `c849abce`→`13b59dd6`, `neartie_gap_mnats.npy` `f716ecce`→`b79989c5`. **Qwen3-4B our_ids.npy md5 UNCHANGED (0 diffs — measured-neutral, stays STRICT) + Qwen3-Coder UNCHANGED (0 diffs).** GATED on DGX GB10 sm_121a (RelWithDebInfo + cutlass-4.5.0 + triton, one `flock`): `test_ops_qkv_merge` 2/2·14; `test_qwen3_paged_engine` 0.6B 16/16 (11 strict + 5 near-tie, max 0.125) + 4B 16/16 (11 strict + 5 near-tie, max 0.25); `test_qwen3coder_paged_engine` 6/6; `test_gemma2_forward` SACRED 48/48 (1065 assertions, global+sliding); `test_gemma4_paged_engine` STRICT 32/32; shared-header canary `test_qwen27_paged_engine` 235/235 UNCHANGED (27B resident-nvfp4 QKV — this bf16 path untouched; 35B structurally inert, not run). Consistency/launch-count fold (measured NEUTRAL on 4B decode, per the pre-existing `dense_attn_block.h` measurement); no new throughput owed. All four record checkers rc=0. Not pushed; FULL SHA reported. RESIDUAL (fold plan): D2 (GDN in_proj merge), D3 (OLMo-2 full-width qk-norm), C2/C3 (MM/nvfp4 towers), E-tier.

docs/BENCHMARKS.md

@@ -106,6 +106,18 @@ See `.agents/specs/laguna-s21-w4-2026-07-31.md`.
 
 ## Laguna-S-2.1 (`LagunaForCausalLM`) W3 real forward + 3 new ops (2026-07-31, `CLAIM-LAGUNA-W3`) - no throughput owed yet (gate PENDING W4)
 
+## bf16 merged-QKV default-ON — Tier-D1 (dense/coder/dflash/Gemma×4, 2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`) - CONSISTENCY/LAUNCH-COUNT fold, CHARACTERIZED near-tie (0.6B golden regenerated), correctness-gated (no new throughput ratio owed)
+
+Tier-D1 of the cross-arch merged-GEMM fold plan (`.agents/specs/arch-fusion-fold-plan-2026-07-30.md`). The q/k/v projections that SHARE the layer input collapse to ONE bf16 `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (the OLMo-2/Granite/StableLM exemplar), replacing three per-shard GEMMs (folds the two tiny GQA k/v GEMMs into one wide tensor-core GEMM, −2 launches/layer). The shared gate `Qwen3QkvMergeEnabled()` (alias `MergedQkvEnabled()`, env `VT_QWEN3_QKV_MERGE`) FLIPPED default-ON across all six bf16 consumers: qwen3_dense (`dense_attn_block.h`), qwen3_coder (inherits the shared dense `AttnBlock` via `qwen3_moe.cpp`), qwen3_dflash (`qwen3_dflash.cpp`), and Gemma-1/2/3/4 (`gemma*.cpp`). `=0` restores the byte-identical 3-shard path in the same binary. **No loader concat needed** — every consumer already packs the merged `qkv_proj` owner. RoPE handling UNCHANGED (no RopeNeox→RopeFromCache swap — the second-1-ULP-flip hazard from the fold plan §gaps.2 avoided). 27B/35B are UNTOUCHED (resident nvfp4/fp8 QKV, not this bf16 path).
+
+**The merge is bit-exact GEMM math (wider N), PROVEN.** `tests/vt/test_ops_qkv_merge.cpp` (CPU, RED-first) byte-compares (raw uint16 memcmp) the merged `MatmulBT`-over-`[wq;wk;wv]` + `QkvSplit` against three separate `MatmulBT`, BYTE-IDENTICAL across GQA/MHA/MQA/wide-head_dim shapes — because `MatmulBT` reduces each output ROW independently, so widening N changes neither the per-row operand nor the per-row reduction; a deliberately-wrong k/v split offset fails the byte-check (the check has teeth). The ONLY byte-affecting part is downstream on hardware — cuBLASLt picks a different K-reduction algo for the wider merged N — which is the CHARACTERIZED 0.6B near-tie, NOT this op-level identity.
+
+**Near-tie golden regen (0.6B only), honest evidence.** Flipping the default shifted the Qwen3-0.6B SACRED anchor at exactly 2 divergence points (prompt[5] tok11 61347 vs 68723, prompt[10] tok10 6851 vs 8193; greedy then cascades). Regenerated via `scripts/qwen3-neartie-gap.py` teacher-forcing the vLLM oracle on our new sequence: **every one of our tokens is within the near-tie band, max gap 0.125 nats ≪ the 0.5-nat bar** (all divergences are genuine bf16 near-ties in vLLM's own logits). `our_ids.npy` md5 `c849abce`→`13b59dd6`, `neartie_gap_mnats.npy` `f716ecce`→`b79989c5`. **Qwen3-4B UNCHANGED (0 token diffs, our_ids.npy md5 stable — measured-neutral, stays STRICT) and Qwen3-Coder UNCHANGED (0 diffs).** So exactly one golden pair moved, batched behind one regen, as the plan required.
+
+- DGX (GB10 sm_121a, RelWithDebInfo + cutlass-4.5.0 + triton): A/B unit `test_ops_qkv_merge` 2/2·14 (byte-identical + RED-first). Re-gate clean: **`test_qwen3_paged_engine` 0.6B 16/16** (11 STRICT + 5 near-tie, max 0.125 nats @ p5 tok11, 0 forward-divergent) **+ 4B 16/16** (11 STRICT + 5 near-tie, max 0.25 nats, 0 divergent); **`test_qwen3coder_paged_engine` 6/6** (5 STRICT + 1 near-tie); **`test_gemma2_forward` SACRED 48/48** (1065 assertions; global+sliding both, teacher-forced near-tie membership); **`test_gemma4_paged_engine` STRICT 32/32**; shared-header canary **`test_qwen27_paged_engine` 235/235** UNCHANGED (27B resident-nvfp4 QKV, this bf16 path never runs). 35B not run (structurally inert — same resident-fp8 QKV path, header change is a getenv default only reachable from the bf16 dense `AttnBlock`).
+
+Disposition: a launch-count/consistency fold measured NEUTRAL on Qwen3-4B (c8 decode is compute-bound; the −2 launches/layer are negligible against ~82k decode launches, per the pre-existing `dense_attn_block.h` measurement), so no isolated tok/s A/B is claimed; the same-binary `VT_QWEN3_QKV_MERGE=0` rollback is available for an A/B if ever wanted. The binding proofs are the byte-exact op unit + the SACRED gates holding (0.6B re-anchored within the near-tie band, 4B/coder/27B byte-stable). OLMo-2/Granite/StableLM already shipped this form; the whole bf16 dense/coder/dflash/Gemma family is now consistent on the one merged-QKV descriptor.
+
 ## bf16 grouped-MoE gate+up+SwiGLU fold — Tier-A4 (Qwen3-Coder + DeepSeek-V2, 2026-07-31, `CLAIM-A4-BF16-MOE-FOLD`) - CONSISTENCY/LAUNCH-COUNT fold, bit-exact, correctness-gated (no new throughput ratio owed)
 
 Tier-A4 (last A-tier) of the cross-arch merged-GEMM fold plan. The bf16 grouped-MoE archs' `{gate grouped-GEMM (f32); up grouped-GEMM (f32); MoeSiluMul}` triplet collapses to ONE NEW shared op `vt::MoeGroupedGemmBf16GateUpSilu` (`kMoeGroupedGemmBf16GateUpSilu`, CUDA). **Open-question resolved:** qwen3_coder ALREADY routes to task #90's fast `vt::MoeGroupedGemmBf16` (`MoeBlockBf16Cuda`), NOT the reference loop — so A4 is a launch-reduction/consistency fold for BOTH consumers (qwen3_coder `qwen3_5.cpp` `MoeBlockBf16Cuda`, deepseek_v2 `deepseek_v2.cpp` `MoeBlock`). Decode/non-WMMA fuses to a `MoeGroupedGemmBf16NaiveSplitK` partials launch + a fused reduce+SwiGLU launch (**5→3 launches**, drops the two f32 [P,I] reduce round-trips); prefill/WMMA reuses `LaunchGroupedBf16` twice + the identical silu-mul. Default-ON behind `VT_MOE_BF16_FUSED_GATEUP` (`=0` = composite everywhere, same-binary rollback). Route-weight stays in `moe_combine` post-down (invariant); no loader change.