perf(laguna): byte-exact attention-glue fusions on NVFP4 decode (VT_LAGUNA_GLUE_FUSED)

Fold the tiny under-occupied glue kernels the decode CUDA graph does NOT hide
into fewer, larger nodes on the Laguna-NVFP4 resident decode — BYTE-EXACT,
default-ON, VT_LAGUNA_GLUE_FUSED=0 opts back out for a same-binary A/B.

- L1: fold the per-head softplus out-gate into the attention-combine store
  (DecodeAttnCombineKernel + the SPLIT<=1 DecodeAttnGqa{,G}Kernel fallbacks).
  The combine stored acc*inv (f32, lossless) and a separate SoftplusHeadGate
  pass reloaded it x softplus(g); folding computes the same (acc*inv)*softplus(g)
  in registers. gate==nullptr keeps the un-gated store bit-for-bit. -1 kernel/layer.
- L4: route each residual-Add + STANDARD RMSNorm pair through the shared
  vt::FusedChain(kFusedAddRmsNormStd) seam (res += x; out = rms_norm(res)*w).
  Byte-exact to vt::Add + rms_norm_seq (commutative f32 add + ResRound<f32>
  identity + the SAME 256-thread strided-sum/tree/1-over-sqrt reduction as
  RmsNormSeqKernel); the default Tier-0 composite already collapses it to ONE
  RmsNorm(residual) launch. The post-attn norm folds in place; the last MLP
  residual add folds with the next layer's input norm (final layer's with the
  final norm), so both loose per-layer Adds disappear into fused nodes. ~-2/layer.
- C (shared-into-MoeCombine) SKIPPED: Laguna's MoeCombine writes bf16 then casts
  to f32, so folding the f32 shared term before the bf16 store rounds
  (routed+shared) to bf16 -> not byte-exact (unlike qwen3_5's f32 combine).
- L2 (fused qk-norm+RoPE preamble) DEFERRED: needs a capture-safe device-position
  kernel variant (graph decode indexes cos/sin by *pos_buf, half-split layout).

DGX gate (GB10, laguna-xs-nvfp4, VT_LAGUNA_RESIDENT_DECODE=1 MARLIN_MOE=1
DECODE_GRAPH=1, ids 2,785,9626,377,15360,395, max-tokens 160): =0 and =1 emit
byte-identical 159-token id streams (PASS). Paging-immune nsys cuda_gpu_kern_sum
2-length diff (160-20)/140 = steady decode/step: GPU-busy 28.90 -> 27.69 ms/step
(-4.2%), graph nodes 1195 -> 1075/step (-120, -10%); drop_caches wall tied
(4.48 vs 4.49s/159). Byte-exact + neutral-to-positive -> landed default-ON.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

Author: mudler

docs/BENCHMARKS.md

@@ -95,6 +95,27 @@ Built clean on GB10 sm_121a (incremental, **7 TUs, 0 warnings**). DGX-gated on `
 
 **WASH on speed** (bf16 3.93s vs base 3.84s; spreads base 3.84–4.03 / bf16 3.93–4.07 overlap → indistinguishable) — did NOT move the residual. At the XS gate's short context (~130 tok) the decode-attention KV read is a small share of decode time; `lm_head_gemv` (streams the ~600 MB tower) + Marlin MoE dominate — so this **confirms the ~17%-to-vLLM-42.46 residual is GEMV/MoE, not KV/attention traffic**, at this context. **Near-tie BREAK on correctness**: bf16 output is coherent (re-emits the golden prefix from ~token 14) but flips the step-2 argmax 268→22345 — our device regime is not bit-identical to vLLM, so a near-tie flips even when moving TOWARD vLLM's bf16 dtype (vLLM itself uses bf16 KV yet emits 268). Two strikes → **NOT landed**; default stays f32 KV (base prefix + ~33 tok/s re-verified after DGX restore). Tested diff + analysis: `.agents/specs/laguna-kv-attn-port-2026-08-02.md`.
 
+## Laguna decode attention-glue fusion — BYTE-EXACT, −4.2% GPU-busy, −120 graph nodes/step (2026-08-02, `CLAIM-LAGUNA-GLUE-FUSED`)
+
+The last real Laguna-NVFP4 decode speed lever + a frameworkization win: fold the tiny, under-occupied attention-glue kernels the decode CUDA graph does NOT hide (device-side ramp/drain of the ~1,195 sequential kernel-nodes/token) into fewer, larger nodes — BYTE-EXACT. Gated behind `VT_LAGUNA_GLUE_FUSED` (default ON, `=0` opts out) for a same-binary A/B. Two folds landed on the resident decode-graph (`LagunaGraph::RunChain`) + the eager resident fallback:
+
+- **L1 — softplus out-gate folded into the attention-combine store.** `SoftplusHeadGateKernel` was a separate pass doing `attn[h,d] *= softplus(g[h])` after the combine wrote `o=acc*inv`. Folding computes the same `(acc*inv)*softplus(g[h])` in registers inside `DecodeAttnCombineKernel` (+ the SPLIT≤1 fallbacks `DecodeAttnGqa{,G}Kernel`) — BYTE-EXACT because the store+reload of the f32 `acc*inv` is lossless. −1 kernel/layer.
+- **L4 — residual-Add + STANDARD RMSNorm pairs routed through the shared `vt::FusedChain(kFusedAddRmsNormStd)` seam** (the same machinery qwen3/gemma use): `res += x; out = rms_norm(res)*w`. BYTE-EXACT to `vt::Add + rms_norm_seq`: the f32 residual add is commutative (+ `ResRound<f32>` is identity) and the norm uses the SAME 256-thread strided-sum + shared-tree reduction + `1/sqrtf` as `RmsNormSeqKernel`. NODE-COUNT WIN in the DEFAULT Tier-0 composite (no `VT_FUSED_TIER` needed): the composite collapses `kAdd`+`kRmsNorm` to ONE `vt::RmsNorm(residual)` launch (`RmsNormRowKernel` does the add inline). The post-attn norm folds in place; the last MLP residual add folds with the NEXT layer's input norm (last layer's with the final norm), so both loose per-layer Adds disappear into fused nodes. ≈−2 kernels/layer.
+- **C (shared-into-MoeCombine) SKIPPED — provably NOT byte-exact in Laguna.** Unlike qwen3_5 (f32 combine out), Laguna's `MoeCombine` writes **bf16** `dout` then casts to f32, so folding the f32 shared `so` before the bf16 store rounds `(routed+shared)` to bf16 vs the current `bf16(routed)+so_f32` — a bit change. Left as its separate f32 add.
+- **L2 (fused qk-norm+RoPE preamble) DEFERRED.** The shared `AttnQkNormRopeGate` indexes cos/sin by token, but Laguna's graph decode needs position-from-DEVICE (`*pos_buf`, half-split layout) — a new capture-safe kernel variant; deferred to avoid correctness risk.
+
+**BYTE-EXACT GATE (RED-first, `~/laguna-xs-nvfp4`, token-ids `2,785,9626,377,15360,395`, `VT_LAGUNA_RESIDENT_DECODE=1 VT_LAGUNA_MARLIN_MOE=1 VT_LAGUNA_DECODE_GRAPH=1`, max-tokens 160, SAME binary):** `=1` (fused) and `=0` (baseline) produce **byte-identical** 159-token id streams (first-20 `22345 83 268 33586 81 855 397 874 367 6376 815 340 9626 377 15360 83 1729 756 1205 565`). PASS.
+
+**SPEED — paging-immune nsys `cuda_gpu_kern_sum`, `--cuda-graph-trace=node`, 2-length diff `(160−20)/140` = steady decode/step (the GB10 "e2e-wall-is-useless / GPU-busy is the anchor" rule):**
+
+| arm | GPU-busy ms/step | graph nodes/step | wall decode/step (drop_caches, best-of-4) |
+|---|---|---|---|
+| `=0` baseline | 28.90 | 1,195 | 4.49s/159 = 35.4 tok/s |
+| `=1` fused | 27.69 | 1,075 | 4.48s/159 = 35.5 tok/s |
+| **Δ** | **−4.2%** | **−120 (−10%)** | **tied (no regression)** |
+
+The **−120 nodes/step** confirms the folds took effect (48× softplus removed + ~2 norm/add folds/layer collapsed); GPU-busy device time drops **4.2%** — beyond the ~1.5-2.5% ramp/drain estimate, because the fused `RmsNorm(residual)` is itself cheaper than the loose `Add + RmsNorm`. At the wall the two arms are tied (`drop_caches`-standardized 4.48 vs 4.49s; the earlier un-standardized 5.95s reading was pure GB10 reload/paging noise) — the recovered device-time sits inside a graph replay still bounded by the fixed per-step host embed-gather + step-boundary sync. BYTE-EXACT + neutral-to-positive → **landed default-ON**. Repro: `~/laguna_glue_run.sh` (byte-exact) + `~/laguna_glue_nsys.sh` (nsys/wall).
+
 ## DeepSeek-V4-Flash — decode nsys attribution: 13.0 tok/s, Q8_0 GEMV is 53.5% (2026-08-01, `CLAIM-DSV4-DECODE-NSYS`)
 
 Re-measured the GGUF decode from scratch (`deepseek-v4-gen --gpu --kv-cache`, `ds4flash.gguf` = `IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8`, GB10, tmux drop-proof capture). **Decode = 13.0 tok/s** (0.076 s/step, dead-consistent over 23 steps) vs **ds4 16.5 → 79% of ds4, ~1.27× behind.** The prior-recorded **8.0 baseline is STALE** (device-resident decode + tuned glue landed since; superseded). Prefill 0.47s, PEAK RESIDENT 86.3 GiB.

docs/STATUS.md

@@ -45,9 +45,13 @@ struct LagunaDeviceKernels {
   // Physical cache row r has GLOBAL position kv_pos=first_pos+r (matches the host's
   // sliding-window eviction bookkeeping); causal (skip kv_pos>q_pos) + per-layer window
   // (skip if window>0 and q_pos-kv_pos>=window); NO attn-sink. scale=1/sqrt(Dh).
+  // L1 (VT_LAGUNA_GLUE_FUSED): `gate` (or nullptr) folds the per-head softplus out-gate
+  // into the normalized attention store — byte-exact vs the separate softplus_head_gate
+  // pass, one fewer kernel/layer. nullptr keeps the un-gated store bit-for-bit.
   void (*decode_attn_gqa)(vt::Queue&, float* o, const float* q, const float* k, const float* v,
                           int64_t Hq, int64_t Hkv, int64_t Dh, int64_t group, int64_t kv_rows,
-                          int64_t q_pos, int64_t first_pos, int64_t window, float scale);
+                          int64_t q_pos, int64_t first_pos, int64_t window, float scale,
+                          const float* gate);
   // Per-head softplus OUT-gate in place: attn[h,d] *= softplus(gate_logits[h]),
   // softplus(x)=(x>20)?x:log1p(exp(x)) in f32. Bit-exact to LagunaSoftplusHeadGate:25.
   void (*softplus_head_gate)(vt::Queue&, float* attn, const float* gate_logits, int64_t Hq,
@@ -70,10 +74,13 @@ struct LagunaDeviceKernels {
   // Hq / Hkv / Dh / group / scale are per-layer constants baked at capture. The key
   // set == decode_attn_gqa's cache[0..rows) AFTER the between-replay append, so the
   // replayed output is bit-identical to the eager decode_attn_gqa. NO attn-sink.
+  // L1 (VT_LAGUNA_GLUE_FUSED): `gate` (or nullptr) folds the softplus out-gate into the
+  // combine store (see decode_attn_gqa); capture-safe (glp is a persistent qkvg[l] slice).
   void (*decode_attn_gqa_g)(vt::Queue&, float* o, const float* q, const float* k, const float* v,
                             const float* knew, const float* vnew, int64_t Hq, int64_t Hkv,
                             int64_t Dh, int64_t group, int64_t first_pos, int64_t window,
-                            float scale, const int* len_dev, const int* pos_dev);
+                            float scale, const int* len_dev, const int* pos_dev,
+                            const float* gate);
   // Laguna lm_head M=1 decode GEMV: out[N] (f32) = W[N,K] (bf16, row-major) · x[K]
   // (f32), M=1. A dedicated one-block-per-row coalesced kernel that streams the
   // ~616 MB [vocab,hidden] weight ONCE at ~roofline — cuBLASLt mis-routes this

include/vllm/model_executor/models/laguna_device.h

@@ -161,4 +161,5 @@ VT_LAGUNA_RESIDENT_MOE
 VT_NVFP4_FP4_GEMV
 VT_LAGUNA_DECODE_GRAPH
 VT_LAGUNA_MARLIN_MOE
+VT_LAGUNA_GLUE_FUSED
 VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH

scripts/env-doc-allowlist.txt

@@ -49,6 +49,7 @@
 #include "vt/backend.h"  // vt::GetBackend (device drain for the keep-quant GEMMs)
 #include "vt/dtype.h"    // vt::IsBlockQuant / RowSizeBytes
 #include "vt/ops.h"      // vt::MatmulBT (dispatches kMatmulBTQuant on block weights)
+#include "vt/recipes.h"  // vt::kFusedAddRmsNormStd (L4 residual-add + RMSNorm fusion)
 #include "vt/tensor.h"
 
 namespace vllm {
@@ -1501,6 +1502,24 @@ inline bool LagunaDecodeGraphEnabled() {
   return on;
 }
 
+// ── Attention-glue fusions on the resident decode (VT_LAGUNA_GLUE_FUSED, default ON).
+// Recovers the device-time the CUDA graph does NOT hide (ramp/drain of ~450 tiny
+// under-occupied sequential kernel-nodes/token) by BYTE-EXACT folds that shrink the
+// captured node count: L1 folds the per-head softplus out-gate into the attention
+// combine store (one fewer kernel/layer); L4 folds each residual-Add + RMSNorm pair
+// into ONE vt::FusedChain(kFusedAddRmsNormStd) call (two fewer kernels/layer with the
+// Tier-1 interpreter, VT_FUSED_TIER=1). Both are bit-for-bit identical to the toggle-
+// off path (same f32 arithmetic, same 256-thread RMSNorm reduction order) — a same-
+// binary A/B: VT_LAGUNA_GLUE_FUSED=0 restores the separate kernels. inline so a CPU
+// build (VT_MARLIN_NVFP4 off) does not -Wunused.
+inline bool LagunaGlueFusedEnabled() {
+  static const bool on = [] {
+    const char* e = std::getenv("VT_LAGUNA_GLUE_FUSED");
+    return !(e != nullptr && e[0] == '0');  // default ON; =0 opts out
+  }();
+  return on;
+}
+
 bool LagunaCanRunResidentDecode(const LagunaParams& p, vt::Queue& q, const LagunaWeights& w,
                                 int64_t T) {
 #ifndef VT_MARLIN_NVFP4
@@ -1675,13 +1694,16 @@ std::vector<float> LagunaForwardResidentDecode(const LagunaWeights& weights, vt:
     vt::GetBackend(dev).Copy(q, vdev + dr * kvdim, vnp, rowbytes);
     const int64_t rows = dr + 1;
     const int64_t fp = cache.dev_first_pos[static_cast<size_t>(l)];
+    // LEVER 1: the fused GEMV already produced g (glp -> qkvg_buf); only the split
+    // fallback still needs a dedicated g_proj GEMV. L1 folds the softplus out-gate into
+    // decode_attn_gqa's store, which reads g — so produce it BEFORE attention (reorder of
+    // two independent same-stream ops => byte-exact).
+    if (!fused_qkvg) GemmBf16Into(gl.data(), lw.attn.g_proj, hn.data(), Hq, H);
+    const bool glue_fused = LagunaGlueFusedEnabled();
     LAG->decode_attn_gqa(q, attn.data(), qvp, kdev, vdev, Hq, Hkv, Dh, group, rows, pos, fp,
-                         window, scale);
+                         window, scale, glue_fused ? glp : nullptr);
     cache.dev_rows[static_cast<size_t>(l)] = rows;  // advance this layer's cached-row count
-    // LEVER 1: the fused GEMV already produced g (glp -> qkvg_buf); only the
-    // split fallback still needs a dedicated g_proj GEMV here.
-    if (!fused_qkvg) GemmBf16Into(gl.data(), lw.attn.g_proj, hn.data(), Hq, H);
-    LAG->softplus_head_gate(q, attn.data(), glp, Hq, Dh);
+    if (!glue_fused) LAG->softplus_head_gate(q, attn.data(), glp, Hq, Dh);  // L1 off => separate pass
     GemmBf16Into(o.data(), lw.attn.o_proj, attn.data(), H, qdim);
     {
       vt::Tensor ht = vt::Tensor::Contiguous(hidden.data(), DType::kF32, dev, {1, H});
@@ -1958,6 +1980,28 @@ struct LagunaGraph {
       vt::Tensor at = DevT(a, H);
       vt::Add(q, at, at, DevT(b, H));
     };
+    const bool glue_fused = LagunaGlueFusedEnabled();
+    // L4: fold a residual-Add + STANDARD RMSNorm pair into ONE vt::FusedChain
+    // (kFusedAddRmsNormStd): res += x; out = rms_norm(res)*w. BYTE-EXACT to
+    // AddInto(res,x) + rms_norm_seq(out,res,w): the f32 residual add is the same
+    // (commutative + ResRound<f32> is identity), and the RMSNorm uses the SAME 256-thread
+    // strided sum + shared-tree reduction + 1/sqrtf as RmsNormSeqKernel. NODE-COUNT WIN in
+    // BOTH tiers: the default Tier-0 composite collapses the chain to ONE
+    // vt::RmsNorm(residual) launch (RmsNormRowKernel does the add inline), and Tier-1
+    // (VT_FUSED_TIER=1) to one interpreter kernel — either way 2 kernels -> 1. All
+    // operands are persistent member buffers => capture-safe (same temporary-Tensor
+    // pattern as the AddInto lambda already captured in this graph).
+    auto FusedAddNorm = [&](float* out, float* x, const float* wgt, float* res) {
+      vt::Tensor o2 = DevT(out, H), x2 = DevT(x, H), r2 = DevT(res, H);
+      vt::Tensor w1 = vt::Tensor::Contiguous(const_cast<float*>(wgt), vt::DType::kF32, dev, {H});
+      vt::FusedChain(q, o2, x2, w1, &r2, vt::kFusedAddRmsNormStd, eps);
+    };
+    // L4: when fused, `hn` always holds rms_norm(hidden, input_norm[l]) on entry to layer
+    // l — layer 0 is seeded here (no preceding residual add to fold), and every later
+    // layer inherits it from the previous layer's fused Pair-2 tail (last MLP add folded
+    // with the next input norm). When NOT fused, the input norm runs at each loop top.
+    if (glue_fused)
+      LAG->rms_norm_seq(q, hn.data(), hidden.data(), input_norm_f[0].data(), 1, H, eps, true);
     for (int64_t l = 0; l < nlayers; ++l) {
       const LagunaLayerWeights& lw = w->layers[static_cast<size_t>(l)];
       const LayerC& c = lc[static_cast<size_t>(l)];
@@ -1972,8 +2016,9 @@ struct LagunaGraph {
       // LEVER A: the position-indexed full RoPE table for this layer's regime; the graph
       // RoPE indexes row *pos_buf on-device (no per-step host cos/sin rebuild).
       const float* rcache = (c.global ? yarn_full : slide_full).data();
-      LAG->rms_norm_seq(q, hn.data(), hidden.data(), input_norm_f[static_cast<size_t>(l)].data(), 1,
-                        H, eps, true);
+      if (!glue_fused)  // L4: fused path carries hn in from the previous layer's Pair-2 tail
+        LAG->rms_norm_seq(q, hn.data(), hidden.data(),
+                          input_norm_f[static_cast<size_t>(l)].data(), 1, H, eps, true);
       GemmBf16(base, lw.attn.qkvg_proj, hn.data(), c.qdim + 2 * kvdim + c.Hq, H);
       if (p->has_qk_norm && !lw.attn.q_norm.Empty()) {
         LAG->rms_norm_seq(q, qvp, qvp, q_norm_f[static_cast<size_t>(l)].data(), c.Hq,
@@ -1986,7 +2031,8 @@ struct LagunaGraph {
       // len/pos read from the DEVICE buffers. first_pos=0 (uniform, P<512).
       LAG->decode_attn_gqa_g(q, attn.data(), qvp, cache_k[static_cast<size_t>(l)].data(),
                              cache_v[static_cast<size_t>(l)].data(), kn, vn, c.Hq, Hkv, Dh, c.group,
-                             /*first_pos=*/0, c.window, scale, len_buf.data(), pos_buf.data());
+                             /*first_pos=*/0, c.window, scale, len_buf.data(), pos_buf.data(),
+                             glue_fused ? glp : nullptr);  // L1: fold softplus out-gate
       // LEVER A: append this token's post-RoPE K (kn) and raw V (vn) into the growing
       // cache at the DEVICE-read slot *len_buf, IN-GRAPH — folds the 2×nlayers between-
       // replay host Copy launches into the captured graph. Runs after decode_attn_gqa_g
@@ -1995,17 +2041,29 @@ struct LagunaGraph {
       LAG->append_kv_row(q, cache_k[static_cast<size_t>(l)].data(),
                          cache_v[static_cast<size_t>(l)].data(), kn, vn, kvdim, len_buf.data());
       // g was produced by the fused GEMV (glp -> qkvg[l]); untouched until here.
-      LAG->softplus_head_gate(q, attn.data(), glp, c.Hq, Dh);
+      if (!glue_fused)  // L1 off => separate softplus out-gate pass
+        LAG->softplus_head_gate(q, attn.data(), glp, c.Hq, Dh);
       GemmBf16(o.data(), lw.attn.o_proj, attn.data(), H, c.qdim);
-      AddInto(hidden.data(), o.data());
-      LAG->rms_norm_seq(q, hn.data(), hidden.data(), post_norm_f[static_cast<size_t>(l)].data(), 1,
-                        H, eps, true);
+      // L4 Pair 1 (post-attn): hidden += o; hn = rms_norm(hidden, post_norm[l]).
+      if (glue_fused) {
+        FusedAddNorm(hn.data(), o.data(), post_norm_f[static_cast<size_t>(l)].data(), hidden.data());
+      } else {
+        AddInto(hidden.data(), o.data());
+        LAG->rms_norm_seq(q, hn.data(), hidden.data(), post_norm_f[static_cast<size_t>(l)].data(), 1,
+                          H, eps, true);
+      }
+      // L4 Pair 2: the LAST residual add of this layer folds with the NEXT layer's input
+      // RMSNorm (or the post-loop final RMSNorm on the last layer) — computed into hn.
+      const float* next_norm = (l + 1 < nlayers)
+                                   ? input_norm_f[static_cast<size_t>(l + 1)].data()
+                                   : final_norm_f.data();
       if (c.is_dense) {
         GemmBf16(dg.data(), lw.mlp.gate_proj, hn.data(), dense_I, H);
         GemmBf16(du.data(), lw.mlp.up_proj, hn.data(), dense_I, H);
         SiluMul(dact.data(), dg.data(), du.data(), dense_I);
         GemmBf16(fdn.data(), lw.mlp.down_proj, dact.data(), H, dense_I);
-        AddInto(hidden.data(), fdn.data());
+        if (glue_fused) FusedAddNorm(hn.data(), fdn.data(), next_norm, hidden.data());
+        else AddInto(hidden.data(), fdn.data());
       } else {
         // LEVER 2: fused router|shared_gate|shared_up GEMV -> rsg slices.
         GemmBf16(rsg.data(), lw.moe.router_shared_gu, hn.data(), E + 2 * moe_I, H);
@@ -2021,8 +2079,10 @@ struct LagunaGraph {
                                     topk, doutb.data());
         SiluMul(dact.data(), sgp, sup, moe_I);
         GemmBf16(so.data(), lw.moe.shared_down, dact.data(), H, moe_I);
-        AddInto(hidden.data(), doutb.data());  // hidden += routed
-        AddInto(hidden.data(), so.data());     // hidden += shared
+        AddInto(hidden.data(), doutb.data());  // hidden += routed (always a plain add)
+        // hidden += shared, folded with the next layer's input (or final) RMSNorm.
+        if (glue_fused) FusedAddNorm(hn.data(), so.data(), next_norm, hidden.data());
+        else AddInto(hidden.data(), so.data());
       }
     }
     // final RMSNorm + lm_head INSIDE the captured region (device), into persistent
@@ -2034,7 +2094,9 @@ struct LagunaGraph {
     // roofline, the measured #1 decode GPU cost). Fixed grid=Vsz + fixed pointers ⇒
     // capture-safe. (f32/quant lm_head would fall back to GemmBf16, but the nvfp4 arm
     // is always bf16 here.)
-    LAG->rms_norm_seq(q, hn.data(), hidden.data(), final_norm_f.data(), 1, H, eps, true);
+    if (!glue_fused)  // L4: fused path already produced hn=norm(hidden,final_norm) in the
+                      // last layer's Pair-2 tail (next_norm==final_norm_f when l+1==nlayers)
+      LAG->rms_norm_seq(q, hn.data(), hidden.data(), final_norm_f.data(), 1, H, eps, true);
     if (w->lm_head.dtype == vt::DType::kBF16)
       LAG->lm_head_gemv(q, logits.data(),
                         reinterpret_cast<const void*>(w->lm_head.bytes.data()), hn.data(), Vsz, H);