vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement#176
Open
raviguptaamd wants to merge 4 commits into
Open
vllm_dissag: GLM-5.1/5.2-FP8 (MLA+DSA) MoRI-EP WideEP disaggregated enablement#176raviguptaamd wants to merge 4 commits into
raviguptaamd wants to merge 4 commits into
Conversation
…ement Adds GLM-5.1-FP8 (GlmMoeDsaForCausalLM = MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the ROCm#171 unified launcher. Fully isolated from DeepSeek-V3/R1: GLM gets its own image + a MODEL_NAME-gated runtime path, so existing models are byte-identical to develop. Defects fixed (validated 1P/1D EP8 + 2P/2D EP16, NIAH 2k-35k = 10/10, no crash): - Long-context accuracy collapse: vLLM #47766 cache-key fix keeps the persistent sparse-MLA kernel ON (keys metadata on per-request context+query len). - 8k disagg prefill crash: DSA adds a 2nd (indexer) KV cache per layer that the single-geometry MoRIIO connector never transferred; paired + shipped prefill-> decode. Plus DSA invalid-token kernel fix (#45324) and shik-latest DP-notify. Changes: - docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile: NEW per-model image (raviguptaamd/vllm glm5.1-dsa-wideEP_on_shik_latest + aiter e03fa6040 + mori 42e895472b08 + router). The base vllm_disagg_inference Dockerfile (DSV3/R1) is left untouched. Future models add their own Dockerfile the same way. - models.json: card pyt_vllm_disagg_mori_glm-5.1-fp8 (GLM_SKIP_PATCHERS=1: image carries the DSA fixes in-source). - models.yaml: GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, mori backends). DeepSeek-V3 dp: caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode logits workspace (isolated to the DSV3 entry). - connectors/moriio.sh: MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models); GLM_SKIP_PATCHERS switch for baked-fix images. - 9 idempotent, anchor-based, self-skipping GLM DSA patcher scripts. KNOWN OPEN DEFECT (future work): 4P/4D EP32 emits corrupted tokens at all context lengths (suspect moriep all-to-all combine at scale); use 1P/1D and 2P/2D. Co-Authored-By: Claude <[email protected]>
raviguptaamd
requested review from
Rohan138,
amathews-amd,
coketaste,
gargrahul and
ppalaniappan-amd
as code owners
July 8, 2026 06:43
benchmark_niah.py mis-scored thinking models: it never disabled thinking and read only content + reasoning_content. GLM-5.1 emits chain-of-thought into the `reasoning` field and leaves `content` empty until the final answer, so with a small max_tokens the answer never lands in content -> a false 0/10 even when generation is correct. - Add chat_template_kwargs.enable_thinking=false so the answer goes to content. - Also score the `reasoning` field as a fallback. Verified against GLM-5.1-FP8: correct 9-10/10 retrieval across 2k-35k on all tested topologies (EP8/EP16/EP32) after the fix. Co-Authored-By: Claude <[email protected]>
…accuracy Needle layout is seeded, so a single run is deterministic (bit-exact on the same stack) but can't tell a real accuracy dip from single-needle variance. Add NIAH_SEEDS (default 0,1,2) to run each context length across multiple needle layouts; the summary now reports mean/min/max across seeds. Backward compatible: NIAH_SEEDS=0 reproduces the prior single-seed behavior. Co-Authored-By: Claude <[email protected]>
…robe) On a freshly-booted node the first request of each context shape pays the full JIT/kernel-autotune compile (minutes). The NIAH harness scored the FIRST request, so cold compile landed on a scored/gated request -> false 0/10 or timeout, failing the accuracy gate and skipping the perf sweep. Root-caused by reproducing on a cold boot (0 results) vs a warm server (all pass) on the same image. Fixes: - benchmark_niah.py: add a warmup pass (NIAH_WARMUP=1 default) — one throwaway request per context length before scoring, with a generous timeout, failures tolerated. Scored requests are then always warm. - benchmark_niah.py: distinguish TIMEOUT/ERROR from a wrong answer. Timeouts return a sentinel (excluded from mean, never counted as 0/10); summary flags NO-RESULT with guidance instead of silently reporting 0. - benchmark_niah.sh: replace the blind `sleep 10` with a /v1/models readiness poll (up to 5 min), and forward NIAH_WARMUP. Verified: patched harness on the warm server passes 10/10; cold-boot repro no longer produces false 0/10 because compile happens in the warmup pass. Co-Authored-By: Claude <[email protected]>
raviguptaamd
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds GLM-5.1-FP8 (
GlmMoeDsaForCausalLM= MLA + DeepSeek Sparse Attention) to the MoRI-EP WideEP disaggregated serving path, stacked on the #171 unified launcher. Fully isolated from DeepSeek-V3/R1 — GLM gets its own image and aMODEL_NAME-gated runtime path, so existing models are byte-identical todevelop.Defects fixed
Validated on 1P/1D (EP8) and 2P/2D (EP16): NIAH 2k–35k = 10/10, no crash; DeepSeek-V3 non-regressed.
unmap MISS→ 60s deferred-write expiry → prefill EngineCore crash. Now the indexer cache is paired with its main cache and shipped prefill→decode. Plus the DSA invalid-token kernel fix (#45324) and the shik-latest DP-notify/hash-failsafe path.Changes
docker/vllm_disagg_inference.glmv5.1.ubuntu.amd.Dockerfile— NEW per-model image (GLM vLLM + aitere03fa6040+ mori42e895472b08+ router). The basevllm_disagg_inferenceDockerfile (DSV3/R1) is left untouched. Future models (e.g. Kimi-2.6) add their own Dockerfile the same way.models.json— cardpyt_vllm_disagg_mori_glm-5.1-fp8→ the new Dockerfile (GLM_SKIP_PATCHERS=1: the image carries the DSA fixes in-source).models.yaml— GLM-5.1-FP8 recipe (block=1, AITER MLA on, eager, per-role mori backends). Also DeepSeek-V3dp:caps (--max-num-seqs 64 --max-model-len 32768) to bound the newer base's decode split-KV logits workspace — isolated to the DSV3 entry.connectors/moriio.sh—MODEL_NAME-gated GLM DSA runtime patchers (pure no-op for other models);GLM_SKIP_PATCHERSswitch for baked-fix images.py_compile-checked.Isolation
Every GLM addition is either a new file, a new catalog entry, or gated on
MODEL_NAME=GLM-5.1-FP8. DeepSeek-V3/R1 run on the unchanged upstream Dockerfile and an unchanged connector code path.Known open defect (future work)
4P/4D EP32 emits corrupted tokens at all context lengths (garbage even at 2k), distinct from the long-context bug; prime suspect is the moriep all-to-all combine at EP32 scale. Use 1P/1D and 2P/2D; EP32 is documented in the GLM Dockerfile STATUS.
Test plan
🤖 Generated with Claude Code