perf(laguna): embed gather reads only T rows, not the full 1.23 GB table/token — 3.9x decode

The W7 speed profile measured Laguna decode at 32.7% GPU-active, 67% host-orchestration.
LagunaEmbed dequantized the ENTIRE [Vsz,H] embed table to f32 (~311M element-converts,
~1.23 GB) on every token just to gather T rows — the DOMINANT decode cost, not the minor
"#5 cleanup" the profile filed it as. Now gathers only the T needed rows directly from
the (f32/bf16) table bytes.

BIT-IDENTICAL by construction: same per-element f32/bf16->f32 conversion, same rows
selected. GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10 sm_121a, --gpu, W6 cached
path, drop_caches cold, prompt "The capital of France is", 24 tokens):
- TOKEN-IDENTICAL to the W5/W6 golden (byte-equal ids `22345 83 350 785 ...`, coherent
  " Paris." continuation).
- decode 0.66 -> 0.17 s/tok = 3.9x (Laguna 1.5 -> 5.9 tok/s; 18x -> 4.7x vs llama.cpp
  27.8). Prefill 0.98s, peak 69.96 GiB.

Syntax-verified with production build flags; CUDA laguna-gen build clean on GB10.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]

Author: mudler

.agents/model-matrix.md

@@ -79,7 +79,7 @@ Engaged architectures (the 43 non-`INVENTORIED` rows):
 | ✅ | `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash (31.2B MLA + GLM MoE) | SACRED gate 8/8 vs vLLM 0.25.0 (STRICT token-exact 1/8 + near-tie-band 7/8, 69/128 tokens strictly exact, max teacher-forced gap 0.0 nats, 0 forward-divergent; vLLM K=5 self-deterministic → STRICT bar); FIRST e2e coverage of the q_lora query branch AND the noaux_tc sigmoid router (closes the MLA campaign's two gaps, C2); speed pending | `MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm` |
 | 📋 | `KimiLinearForCausalLM` | Kimi-Linear | MLA half unlocked by the shared MLA campaign; the full model is not gated, row stays `SPIKE` | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` |
 | 📋 | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12×)** and NOT in the pinned oracle ⇒ no on-box golden — DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` |
-| ✅ | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute — GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(n²) STATELESS recompute — TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05× faster per token: decode 3.33 → 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent → GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 — bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) — "The capital of France is" → " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition — variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head — replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin ⇒ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward — the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factor·(1+0.1·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) — resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` |
+| ✅ | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **FASTER DECODE (W8, 2026-07-31, `CLAIM-LAGUNA-W8-EMBED`): `LagunaEmbed` no longer converts the whole 1.23 GB embed table to f32 every token (it gathered T rows out of the whole [Vsz,H] table via `ReadF32` — ~311M host element-converts/token, the DOMINANT decode cost the W7 profile under-filed as "#5"); now gathers only the T needed rows directly (BIT-IDENTICAL — same per-element conversion, same rows). GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok): TOKEN-IDENTICAL to the W5/W6 golden (`22345 83 350 785 …`, coherent " Paris.") + decode 0.66 → 0.17 s/tok = 3.9× (1.5 → 5.9 tok/s; 18× → 4.7× vs llama.cpp 27.8). See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` §W8. Next: grouped-expert GEMM (=A3) then device-resident decode.** Prior **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute — GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(n²) STATELESS recompute — TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05× faster per token: decode 3.33 → 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent → GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 — bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) — "The capital of France is" → " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition — variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head — replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin ⇒ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward — the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factor·(1+0.1·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) — resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` |
 | 🚫 | `DeepseekV3ForCausalLM` / `DeepseekV32ForCausalLM` | DeepSeek-V3 / V3.2 | HW-blocked (671B, ~642 GiB fp8 vs 119 GiB unified memory); V3.2 additionally DEP-blocked (DSA indexer) | `MODEL-TEXT-deepseek-v2-deepseek-v3-for-causal-lm` |
 | 🚫 | `GlmMoeDsaForCausalLM` | GLM-5 (DSA) | HW-blocked (1404 GiB bf16) and DEP-blocked (GLM-5.x is DeepSeek-V3.2 verbatim) | `MODEL-TEXT-deepseek-v2-glm-moe-dsa-for-causal-lm` |
 | 🚫 | `MiniMaxM2ForCausalLM` | MiniMax-M2 | HW-blocked (~230B / ~428 GiB bf16, ~4x over unified memory) | `MODEL-TEXT-minimax-m2-mini-max-m2-for-causal-lm` |

.agents/specs/laguna-s21-w7-speed-2026-07-31.md

@@ -127,3 +127,27 @@ GEMVs at 22% of BW peak** — NOT kernel compute. The ranked levers (device-resi
 expert GEMM first, then decode-graph + tuned MMVQ) are the DeepSeek-V4 playbook, all in-tree. Honest
 reachable ~13–20 tok/s, 27.8 a stretch. Numbers are measured (nsys) or explicitly computed (roofline);
 hypotheses are labelled. No code changed.
+
+---
+
+## W8 — lever #5 (embed gather) LANDED + GATED (2026-07-31, `CLAIM-LAGUNA-W8-EMBED`)
+
+The W7 profile filed `LagunaEmbed`'s full-table copy under "#5 free host cleanups", but
+it was the **dominant** decode cost, not minor: `LagunaEmbed` called `ReadF32(embed_t)`
+which converted the ENTIRE `[Vsz,H]` f32/bf16 embed table (~311M element-converts,
+~1.23 GB) on the HOST **every token** just to gather T rows. Fixed: gather only the T
+needed rows directly from the table bytes (bit-identical — same per-element f32/bf16→f32
+conversion, same rows; the untouched rows never affected the output).
+
+**GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10 sm_121a, `--gpu`, W6 cached path,
+drop_caches cold, prompt "The capital of France is", 24 tokens):**
+- **TOKEN-IDENTICAL PASS** — `generated ids` byte-equal to the W5/W6 golden
+  (`22345 83 350 785 989 395 13259 330 4159 9431 377 340 4328 377 444 136 22029 9626 71 493 6396 565 7760 10291`),
+  coherent " Paris." continuation. Bit-exact confirmed on the real model.
+- **SPEED: decode 0.66 → 0.17 s/tok = 3.9×** (Laguna **1.5 → 5.9 tok/s**; 18× → 4.7× vs
+  llama.cpp 27.8). Prefill 0.98s, peak 69.96 GiB. Single bit-exact host fix.
+
+The measured 0.49 s/tok saved per token ≈ the host cost of the eliminated 311M-element
+embed conversion — consistent with the host-orchestration attribution. Remaining levers
+unchanged (grouped-expert GEMM = A3, then device-resident decode); the per-token
+RoPE-cos/sin rebuild + norm/router `ReadF32` are the smaller residual #5 items.

docs/BENCHMARKS.md

@@ -10852,3 +10852,5 @@ speed lever. Gated microbench `VT_FP4_M1_BENCH` (test-only).
 **D3 GPU gate CLOSED (2026-07-31, GB10 sm_121a, main `ace8c210`).** The OLMo-2 full-width qk-norm fold (`1a614a9f`, bit-exact-by-construction, was CPU-gated only) is now empirically confirmed on hardware: `test_olmo2_paged_engine` **16/16** (92 assert; 13 strict + 3 near-tie, max 0.094 nats, 0 forward-divergent) + `test_ops_fused_chain` **10/10** (583) + canaries `test_qwen3_paged_engine` 0.6B/4B **16/16** + `test_qwen27_paged_engine` **235/235** — all UNCHANGED vs committed goldens (the `norm_full_width` shape-param is inert on the per-head `kAttnQkNormRope` path). Zero regressions. `test_olmo3_paged_engine` remains checkpoint-blocked (owed if OLMo-3 fetched).
 
 **A3 keep-quant grouped MoE fold — SCOPED, not yet gated (2026-07-31).** No benchmark yet (impl pending): the fold removes E host round-trips + E×2 tiny keep-quant GEMM launches → ~3 grouped launches on the qwen3_5 GGUF MoE path. Bit-exact by construction (same block-quant vec_dot). Perf outcome PENDING; DGX SACRED token-exact gate OWED.
+
+**Laguna-S-2.1 decode: 0.66 → 0.17 s/tok (3.9×), W8 embed-gather (2026-07-31, GB10 GB10 sm_121a).** Real 3-shard UD-Q4_K_XL keep-quant GGUF, `--gpu`, W6 cached path, drop_caches cold, 24 tokens: TOKEN-IDENTICAL to the W5/W6 golden (byte-equal ids, coherent " Paris."). 1.5 → 5.9 tok/s (18× → 4.7× vs llama.cpp 27.8). `LagunaEmbed` stopped ReadF32-converting the entire [Vsz,H] embed table (~311M host element-converts, ~1.23 GB) every token to gather T rows — now gathers only the T rows directly (bit-identical). Prefill 0.98s, peak 69.96 GiB. Next levers: grouped-expert GEMM (=A3) + device-resident decode.

docs/STATUS.md

@@ -2003,3 +2003,5 @@ The gate models' W4A4 decode is at the achievable roofline. See `.agents/specs/n
 **D3 GPU gate CLOSED (2026-07-31, GB10).** OLMo-2 full-width qk-norm fold empirically confirmed: olmo2 16/16 + fused_chain 10/10 + qwen3/27B canaries unchanged. See BENCHMARKS.
 
 **A3 fold SCOPING VERIFIED (2026-07-31, pending impl).** qwen3_5 GGUF keep-quant grouped MoE fold: premise confirmed bit-exact (ResidentWeight keeps weights quantized → per-expert loop already runs the ds4 vec_dot). Remaining: loader stacking + a keep-quant grouped MoE block (ds4 MoeFusedResident analog) + RED-first byte test. DGX SACRED gate OWED. See `.agents/specs/arch-fusion-fold-plan-2026-07-30.md` A3.
+
+**Laguna W8 embed-fix LANDED + GATED (2026-07-31).** `LagunaEmbed` gathers only the T needed rows instead of converting the whole 1.23 GB embed table to f32 every token. GATED on real UD-Q4_K_XL GGUF (GB10, `--gpu`): TOKEN-IDENTICAL to the W5/W6 golden + decode **0.66 → 0.17 s/tok (3.9×; 1.5 → 5.9 tok/s)**. Bit-exact by construction. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` §W8.

src/vllm/model_executor/models/laguna.cpp

@@ -234,19 +234,37 @@ std::vector<float> LqGemmRowSlice(vt::Queue& q, const OwnedTensor& w,
 //    KV-cached incremental forward, so the two paths are bit-identical BY SHARING
 //    the exact same float ops — the moved code is verbatim from the W5 forward). ──
 
-// Embed gather: hidden[T,H] = embed_table[token_ids].
+// Embed gather: hidden[T,H] = embed_table[token_ids]. Gathers ONLY the T needed
+// rows directly from the (f32/bf16) table bytes — BIT-IDENTICAL to the prior
+// ReadF32(whole-table)-then-gather (same per-element f32/bf16→f32 conversion,
+// same rows), but avoids materializing the full [Vsz,H] table (~1.23 GB, ~311M
+// element-converts) on EVERY decode token — the dominant host-orchestration
+// waste measured in the W7 speed profile (laguna-s21-w7-speed-2026-07-31.md #5).
 std::vector<float> LagunaEmbed(const OwnedTensor& embed_t,
                                const std::vector<int32_t>& token_ids, int64_t H,
                                int64_t Vsz) {
-  const std::vector<float> embed = ReadF32(embed_t);
   const int64_t T = static_cast<int64_t>(token_ids.size());
   std::vector<float> hidden(static_cast<size_t>(T * H));
+  const uint8_t* raw = embed_t.bytes.data();
+  const bool is_bf16 = embed_t.dtype == vt::DType::kBF16;
+  VT_CHECK(embed_t.dtype == vt::DType::kF32 || is_bf16,
+           "laguna embed: table dtype must be f32/bf16 (matches ReadF32)");
   for (int64_t t = 0; t < T; ++t) {
     const int64_t tok = token_ids[static_cast<size_t>(t)];
     VT_CHECK(tok >= 0 && tok < Vsz, "laguna: token id out of range");
-    std::memcpy(hidden.data() + static_cast<size_t>(t * H),
-                embed.data() + static_cast<size_t>(tok * H),
-                static_cast<size_t>(H) * sizeof(float));
+    float* dst = hidden.data() + static_cast<size_t>(t * H);
+    if (is_bf16) {
+      const auto* b =
+          reinterpret_cast<const uint16_t*>(raw) + static_cast<size_t>(tok * H);
+      for (int64_t i = 0; i < H; ++i) {
+        const uint32_t bits = static_cast<uint32_t>(b[i]) << 16;
+        std::memcpy(&dst[i], &bits, sizeof(float));
+      }
+    } else {
+      std::memcpy(dst,
+                  reinterpret_cast<const float*>(raw) + static_cast<size_t>(tok * H),
+                  static_cast<size_t>(H) * sizeof(float));
+    }
   }
   return hidden;
 }