spike(laguna): W10 re-profile — decode 89.7% host-sync, device-residency confirmed top lever

Re-profiled current main (W8+W9) to re-rank levers with real data (the W7 attribution
was of the pre-W8/W9 0.66 path). nsys: decode 0.13 s/tok is 89.7% cudaStreamSynchronize
(14,034 calls ~610/step; down from W7's 22k via W9), launches only 2.6%. Confirms lever
#1 device-resident decode (defer per-GEMM DrainQueue to one DrainDevice/step, glue
on-device) as the top remaining lever. Measurement-only, no code change.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

Author: mudler

.agents/specs/laguna-s21-w7-speed-2026-07-31.md

@@ -178,3 +178,30 @@ This routes through the SHARED `vt::MatmulBTQuantGrouped` (fold policy: no per-m
 and proves the grouped-consumption pattern qwen3_5's A3 reuses (after its loader-stacking —
 qwen3_5 stores experts per-`OwnedTensor`, not stacked). Remaining Laguna lever: #1
 device-resident decode (the 22k-syncs kill) toward the ~13–20 tok/s ceiling.
+
+---
+
+## W10 — RE-PROFILE after W8+W9 (2026-07-31, measurement-only, `CLAIM-LAGUNA-W10-REPROFILE`)
+
+The W7 attribution profiled the pre-W8/W9 0.66 s/tok path; re-profiled current main
+(`a402eb6b`, W8+W9) to re-rank levers with real data. nsys `--trace=cuda`, real
+UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok, TPOT 0.13 s/tok):
+
+| CUDA API | Time% | calls | reading |
+|---|---|---|---|
+| **`cudaStreamSynchronize`** | **89.7%** (2.59 s) | **14,034 (~610/step)** | the per-GEMM `DrainQueue` — STILL the dominant cost |
+| `cudaStreamCreate` | 7.4% | 1 | one-time queue create (213 ms, amortized, not per-step) |
+| `cudaLaunchKernel` | 2.6% | 28,068 (~1,220/step) | launch overhead is NOT the bottleneck |
+
+**VERDICT: lever #1 (device-resident decode) CONFIRMED as the top remaining lever.**
+Syncs dropped 22,115 → 14,034 (W9's grouped-MoE effect) but decode is STILL 89.7%
+host-sync-bound: every `LqGemm`/`LqGemmGrouped` drains the stream so the host reads the
+output for the next f32 glue op (RmsNorm, RoPE, attention, SwiGLU, router, softplus
+out-gate — all host today). The fix (mirror ds4 `ForwardResidentDecodeGguf` + decode
+CUDA-graph): keep intermediates ON-DEVICE and run the glue on-device so the whole
+step defers to ONE `DrainDevice` (610 → ~1 sync/step). This is a MULTI-BRICK campaign
+(port each Laguna glue op to a device kernel + chain them), NOT a bit-exact host fix
+like W8/W9 — governed by the CUDA-graph-capture-safety hazards (see
+[[cudagraph-capture-bakes-stack-addresses]]). GPU-kernel breakdown (post-residency
+residual: `QuantizeQ8K` dedup, weight-GEMV BW) owed on the next profile. Reachable
+still ~13–20 tok/s; killing ~600 syncs/step is the path there.

docs/BENCHMARKS.md

@@ -10876,3 +10876,5 @@ speed lever. Gated microbench `VT_FP4_M1_BENCH` (test-only).
 **Laguna-S-2.1 decode: 0.18 → 0.13 s/tok (1.38×), W9 grouped-expert MoE (2026-07-31, GB10 sm_121a).** Real UD-Q4_K_XL keep-quant GGUF, `--gpu`, W6 cached, drop_caches cold, 24 tok. Same-binary A/B (`VT_LAGUNA_GROUPED_MOE=1` vs `=0`): BYTE-IDENTICAL ids (md5 `754728c6`, == W6 golden), so bit-exact both ways. The top_k per-expert gate/up/down GEMVs collapse to 3 grouped `vt::MatmulBTQuantGrouped` launches/token over the stacked tower. Cumulative Laguna decode 0.66 → 0.13 s/tok (5.1×; 1.5 → 7.7 tok/s; 18× → 3.6× vs llama.cpp 27.8). Prefill 0.99 → 0.70s. Next: device-resident decode (#1).
 
 _(Laguna W8+W9 campaign detail + repro in `.agents/state.md` and `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`.)_
+
+_(Laguna W10 re-profile (2026-07-31, GB10): decode 0.13 s/tok = 89.7% `cudaStreamSynchronize` (610/step), 2.6% launches → device-resident decode is the confirmed next lever toward ~13-20 tok/s. Measurement-only. See spec §W10.)_

docs/STATUS.md

@@ -2024,3 +2024,5 @@ The gate models' W4A4 decode is at the achievable roofline. See `.agents/specs/n
 **Laguna W9 grouped-expert MoE LANDED + GATED (2026-07-31).** The 30 un-grouped per-expert GEMV launches/step fold onto the shared `vt::MatmulBTQuantGrouped` (3×top_k → 3 launches/token, no loader change). Same-binary A/B on real UD-Q4_K_XL (GB10, `--gpu`): grouped==per-expert BYTE-IDENTICAL (md5 `754728c6`, == W6 golden) + decode **0.18 → 0.13 s/tok (1.38×)**. Cumulative with W8: **0.66 → 0.13 s/tok (5.1×, 1.5 → 7.7 tok/s)**. See spec §W9.
 
 _(Laguna W8+W9 decode-speed campaign handoff recorded in `.agents/state.md`; cumulative 0.66 → 0.13 s/tok, next lever #1 device-resident decode.)_
+
+_(Laguna W10 re-profile: current 0.13 s/tok decode is 89.7% host-sync (14,034 `cudaStreamSynchronize`/24tok ≈ 610/step); lever #1 device-resident decode confirmed top. See spec §W10.)_