This is the canonical stable-ID inventory for cross-cutting engine behavior at
the parity pin 555967922 (vLLM 0.26.0.dev0 + transformers 5.14.1; advanced
2026-07-26 from the prior e24d1b24/0.25.0 pin, see
specs/pin-advance.md), plus the historical additions from
the v0.25.0 sync target 702f4814fe54. Model, quantization, kernel, and platform support
remain in their own matrices. The inventory audit that established the 26
code-bearing baselines is recorded in
feature-anchor-backfill.md.
Cross-ref (roadmap_v1 ORDER-1): the portable automatic op-fusion framework —
which binds fusion recipes to forward-pass sites at model build — is tracked as
KERNEL-FUSION-FRAMEWORK in kernel-matrix.md (its evidence
is per-backend fused-kernel dispatch), spike
portable-fusion-framework.md. The build-time
recipe→site binding surface touches the engine forward, but the row lives in the
kernel matrix; not duplicated here.
ANCHOR-BACKFILL means the named bounded slice has code and tests but still
lacks its required leaf spike. PARTIAL means the implementation is also
known to omit upstream behavior. Neither state is protocol-complete. A plain
planned: specs/... entry is not an accepted spike and cannot make a row
READY.
Current SERVE-GATE-ONLINE binding: 9ecd9d0: 114/124 (mem 4/4, c1 20/20,
c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20); two-grid totality with f0fb727
(111/124) is 115/124 effective parity against vLLM 0.25.0 (27B). The
bit-identical fast decode-kernel stack (348d12d+9ecd9d0) plus async-default-ON
(a0013a2), the vendored Triton GDN cubin (a321d7c) and packed-decode equivalence
(e47b4d6) closed +62 axes over the prior superseded bindings (3f256ab 55/124,
246a23c 49/124, a875397 52/124, all retained immutable). The residuals are the
low-concurrency-median edge of a net-positive determinism tradeoff (we win the tails
- high concurrency + throughput); no closeable real 27B deficit. Full grid + forensics: roadmap_v1.md and the parity ledger.
| Area | Rows | ANCHOR-BACKFILL |
PARTIAL |
SPIKE |
READY |
ACTIVE |
GATING |
DONE |
INVENTORIED |
|---|---|---|---|---|---|---|---|---|---|
| Engine and scheduling | 27 | 3 | 4 | 1 | 0 | 12 | 2 | 1 | 4 |
| KV cache and memory | 21 | 1 | 2 | 2 | 2 | 7 | 2 | 1 | 4 |
| Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 |
| Sampling and generation | 15 | 0 | 2 | 0 | 0 | 7 | 0 | 1 | 5 |
| Structured output and tools | 7 | 0 | 3 | 0 | 0 | 2 | 0 | 0 | 2 |
| Speculative decoding | 12 | 0 | 0 | 1 | 0 | 4 | 0 | 4 | 2 |
| Serving, API, CLI, library | 22 | 3 | 2 | 2 | 0 | 7 | 2 | 1 | 5 |
| LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
| Long context and attention | 10 | 0 | 0 | 0 | 1 | 5 | 1 | 0 | 3 |
| Loading, tokenizer, config | 9 | 1 | 3 | 0 | 0 | 2 | 1 | 1 | 1 |
| Total | 131 | 8 | 16 | 5 | 4 | 48 | 8 | 9 | 32 |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
ENG-SCHED-CORE |
Text-generation running-first unified scheduler, FCFS, token budget, output update; two budget-fitting prefills co-schedule into one step (c2 parity — mirrors upstream, no divergence) | T0 | vllm/v1/core/sched/scheduler.py:396,640,1501 @ e24d1b24/702f481; tests/v1/core/test_scheduler.py:86,847 |
src/vllm/v1/core/sched/scheduler.cpp:114,234,365 |
tests/vllm/v1/test_scheduler.cpp:143,205,241,416; tests/vllm/v1/test_engine_core.cpp:271 |
planned: specs/unified-scheduler.md; verdict scheduler-prefill-coschedule.md |
ANCHOR-BACKFILL |
- |
ENG-CHUNKED-PREFILL |
Basic token-budget chunked prefill | T0 | vllm/config/scheduler.py:84; vllm/v1/core/sched/scheduler.py:835; tests/v1/core/test_scheduler.py:185,503,903 |
src/vllm/v1/core/sched/scheduler.cpp:225,548 |
tests/vllm/v1/test_scheduler.cpp:192; tests/vllm/models/test_qwen27_paged_forward.cpp:492 |
planned: specs/chunked-prefill.md |
ANCHOR-BACKFILL |
- |
KV-PREFIX-CACHE |
APC hashes, lookup, allocation, partial blocks, eviction, plus explicit/model-default cache policy. W0 ports arbitrary-group no-prefix coordination and makes hybrid/attention-free defaults cache-off. Full-surface re-audit 2026-07-22 (spike) — the ported core is DEEPER than this row read (chain hashing, pool, all three coordinators, the complete hybrid intersection and four single-type managers), and the residual gaps are narrower and DIFFERENT: generate_block_hash_extra_keys: W2 DONE 2026-07-27 (CLAIM-ROADMAP-D4APC) — the hardcoded no-op is replaced by a 1:1 port of kv_cache_utils.py:451-591 (_gen_mm_extra_hash_keys + LoRA name + cache_salt, fixed order lora->mm->salt; prompt_embeds deferred, no prompt-embeds path). Request/EngineCoreRequest carry cache_salt + lora_name; FromEngineCoreRequest sets them BEFORE the first hash (fixed a latent ordering bug: mm_features were assigned after the ctor already hashed). The latent correctness trap is CLOSED and RED-first proven: with the stub, a tenant-B request false-hits tenant-A's 48 cached tokens (n1==48); with extra keys n1==0 (no false-share). This unblocks the MM + LoRA cache consumers. prefix-cache statistics: CLOSED 2026-07-22 (W1) — PrefixCacheStats/CachingMetrics ported 1:1 with log_stats DEFAULTED ON, which unblocks the BACKEND-GATE-CUDA-SGLANG-PREFIX hit-proof requirement; first measured hit rate 0.75 on a repeated-prefix corpus; no cache_salt; 1 of upstream's 4 hash algos; skip_reading_prefix_cache absent; partial-block primitives throw (upstream's own are DEAD CODE — no caller in vllm/ — so they are NOT owed as live behaviour). Also cleared: the "blocked on a supported non-hybrid family" blocker is STALE — dense models default APC ON and five have landed, yet NO gate has ever run cache-ON MLA prefix-cache-hit assert fixed 2026-07-23 (CLAIM-MLA-PREFIX-CACHE-ASSERT): FullAttentionManager::find_longest_cache_hit asserted kind()==kFullAttention, aborting DeepSeek-V2 (MLA group, kind kMlaAttention, APC default-ON) under asserts-enabled builds — latent since ec6f4be, inert under Release/NDEBUG. Relaxed to upstream's precondition isinstance(spec, FullAttentionSpec or ChunkedLocalAttentionSpec) (single_type_kv_cache_manager.py:578-582; MLAAttentionSpec IS-A FullAttentionSpec) ⇒ accept kFullAttention / kMlaAttention / kChunkedLocalAttention; restores DeepSeek-V2 SACRED gate 8/8 asserts-on, full-attention byte-identical, new MLA prefix-cache-hit unit cases. |
T0 | vllm/config/model.py:1805-1860; vllm/engine/arg_utils.py:510,1160-1166,2473-2508; vllm/config/cache.py:39,93,95; extra keys vllm/v1/core/kv_cache_utils.py:539-574; hasher factory :673-730; vllm/v1/core/kv_cache_coordinator.py:377-425,782-834; tests/v1/core/test_prefix_caching.py:225,1475,2781 |
hashes/managers src/vllm/v1/core/kv_cache_utils.cpp:259,291; extra_keys generate_block_hash_extra_keys + _gen_mm_extra_hash_keys src/vllm/v1/core/kv_cache_utils.cpp; cache_salt/lora_name on include/vllm/v1/request.h + include/vllm/v1/engine/types.h, copied in src/vllm/v1/request.cpp FromEngineCoreRequest (fields set before the first hash); src/vllm/v1/core/kv_cache_manager.cpp:124; no-prefix coordinator/factory src/vllm/v1/core/kv_cache_coordinator.cpp:260,273,279,545; model-default/hasher selection src/vllm/entrypoints/model_loader.cpp:109,167,180,191; CLI examples/server/main.cpp:126; statistics include/vllm/v1/metrics/stats.h, recorded src/vllm/v1/core/kv_cache_manager.cpp:139-147, reset flag :270-276, take-and-swap make_prefix_cache_stats(), per-step window fold at the end of Scheduler::schedule(), accessors Scheduler/EngineCore/LLMEngine::prefix_cache_metrics(); Request::num_preemptions un-deferred (include/vllm/v1/request.h, incremented in Scheduler::preempt_request) |
existing APC primitives tests/vllm/v1/test_kv_cache_utils.cpp:411,516,536; no-prefix hybrid allocation/no-hit tests/vllm/v1/test_kv_cache_coordinator.cpp:213; default/override resolution tests/vllm/entrypoints/test_loaded_engine_dense.cpp:343; server help and online cache-off contracts examples/CMakeLists.txt:34; tests/tools/test_online_gate_client.py:582,633; statistics plus the first MEASURED hit rate tests/vllm/v1/test_prefix_cache_stats.cpp 12/12; W2 extra_keys — ported mm/lora/salt cases + ordering + hash-level no-false-share tests/vllm/v1/test_kv_cache_utils.cpp (29/29), manager-level salt-partition no-false-share (RED-proven n1 48->0) tests/vllm/v1/test_kv_cache_manager.cpp (10/10), CPU gate on dgx GB10. W3 DONE 2026-07-27 (CLAIM-ROADMAP-D4APC-W3, dgx GB10, NOT pushed) — the FIRST-EVER cache-ON model gate: tests/parity/test_qwen3_apc_e2e.cpp on Qwen/Qwen3-4B (dense, full-attention, APC-default-ON) 2/2 cases, 84/84 asserts — APC-ON hits 2240/2777 (rate 0.807) / APC-OFF 0; APC-ON == APC-OFF token-exact 5/6 (1 diff a vLLM-confirmed 0.125-nat near-tie); == vLLM-APC-ON teacher-forced (OFF 6/6 gap 0.0, ON 6/6 gap ≤0.125 nats, 0 outside top-20); TTFT 70.1→39.9 ms = 1.76×. NO engine code changed (gate-only over the already-shipped default-ON path); 4B SACRED 16/16 no-regression. Oracle vLLM 0.25.0. Ledger: parity-ledger.md#L746 |
prefix-prompt-caching-parity.md (umbrella); prefix-caching.md (cache-policy leaf) | DONE (dense APC path; W4 events/W5 partial/W6 mamba-align/W7 reset endpoint tracked in KV-EVENTS/KV-MAMBA-ALIGN/own future rows) |
a41af480 |
KV-PREFIX-MATCH-UNIT |
--prefix-match-unit (config prefix_match_unit): the finest token boundary a prefix-cache hit can land on == the hash_block_size/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior e24d1b24/0.25.0 pin). For a HYBRID/multi-group model the resolver resolve_kv_cache_block_sizes computes hash_block_size = prefix_match_unit if set else gcd(group_block_sizes) (scheduler block size = lcm), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from cache_block_size (mamba_cache_mode != "align"); throws on a non-divisible unit. W0 spike + W1 resolver LANDED 2026-07-28 (CLAIM-PREFIX-MATCH-UNIT, NOT pushed): resolve_kv_cache_block_sizes ported 1:1 (explicit-parameter signature vs upstream's VllmConfig, our config surface is threaded), RED-first unit-gated (default gcd != =16 override). PARTIAL: the config/CLI/ABI field (W2), the scheduler threading of a resolved hash_block_size != block_size + mamba partial-tail stop (W3, needs the KV-BLOCK-POOL align path that still throws), and the benchmark (W4) are deferred. Default path byte-identical (single-group inert; scheduler still passes block_size). |
T1 | vllm/engine/arg_utils.py:696,1222,1940; vllm/config/cache.py:56-67; resolver vllm/v1/core/kv_cache_utils.py:626-688; hasher :691-748; call site vllm/v1/engine/core.py:154; scheduler vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318; fine-grained view vllm/v1/core/single_type_kv_cache_manager.py:683,697 |
resolver src/vllm/v1/core/kv_cache_utils.cpp:638 (resolve_kv_cache_block_sizes), decl include/vllm/v1/core/kv_cache_utils.h; hash_block_size already plumbed get_request_block_hasher src/vllm/v1/core/kv_cache_utils.cpp:577; DEFERRED align path throws src/vllm/v1/core/block_pool.cpp:93,220 (shared with KV-BLOCK-POOL) |
tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, =16 override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); parity-ledger.md |
prefix-match-unit.md | PARTIAL |
CLAIM-PREFIX-MATCH-UNIT |
ENG-PREEMPT-RECOMPUTE |
FCFS tail preemption with recompute | T0 | vllm/v1/core/sched/scheduler.py:1142; tests/v1/core/test_scheduler.py:930 |
src/vllm/v1/core/sched/scheduler.cpp:102,157; src/vllm/v1/core/sched/request_queue.cpp:36 |
tests/vllm/v1/test_scheduler.cpp:247,295; tests/vllm/v1/test_request_queue.cpp:91 |
planned: specs/preemption.md |
ANCHOR-BACKFILL |
- |
ENG-CUDAGRAPH |
Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from max_num_seqs mirroring vLLM _set_cudagraph_sizes; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) |
T0 | vllm/config/compilation.py:53,1319,683-684,1438-1444; vllm/config/vllm.py:1667-1770; vllm/v1/worker/gpu/cudagraph_utils.py:116; tests/compile/test_config.py:122,229 |
src/vt/cuda/cuda_backend.cu:76,97,105; include/vllm/model_executor/models/decode_graph_sizes.h; src/vllm/model_executor/models/qwen3_5.cpp:3754,3952; src/vllm/v1/worker/gpu/runner.cpp:577,597; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) src/vt/cuda/graph_safe_scratch.h, src/vt/cuda/cuda_moe_marlin.cu:75, src/vt/cuda/cuda_matmul_nvfp4.cu:766, src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105, src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95 |
tests/vt/test_cuda_backend.cpp:98; tests/vllm/models/test_decode_graph_sizes.cpp; tests/vt/test_graph_safe_scratch.cpp; explicit 35B gate tests/parity/test_qwen36_paged_engine.cpp:140 |
blocktable-host-cluster-cleanup.md; decode-graph-scratch-uaf-2026-07-18.md | PARTIAL |
- |
ENG-BATCH-INVARIANT |
Opt-in deterministic execution across scheduler batch sizes (VLLM_BATCH_INVARIANT=1): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off |
T1 | default/env vllm/envs.py:89,576-578; initialization vllm/v1/worker/gpu_worker.py:1262; NVFP4 dispatch csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220; suite fixture tests/v1/determinism/conftest.py:9-12; operator/e2e tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py, tests/v1/determinism/test_nvfp4_batch_invariant.py @ 702f481 |
- | W3-C3R executed contract: production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | planned: specs/batch-invariant-execution.md |
INVENTORIED |
- |
ENG-ASYNC-SCHED |
Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16): AsyncScheduler placeholder accounting, step_with_batch_queue depth-2, ResolveAsyncScheduling default-ON-when-compatible + MaxConcurrentBatches, VT_ASYNC_SCHED rollback; the runner device-input path combine_sampled_and_draft_tokens; PLUS the sampler-OUTPUT half — vt::Backend event/pinned primitives (AllocPinned/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), AsyncGPUModelRunnerOutput (device sampled-id snapshot → non-blocking D2H on a copy queue + event; get_output() waits only that event; MAIN queue never blocked), Sampler::forward(sampled_ids_out) device-resident greedy, GPUModelRunner::sample_tokens_async + runner_supports_async, and the Executor+step_with_batch_queue seam resolving get_output() at CONSUME time. All behind VT_ASYNC_RUNNER/set_async_input_combine, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; sample_tokens_async degenerates to sync when async off; sampled_ids_out=nullptr). ENABLE-FLIP LANDED + CPU-gated (2026-07-16): (1) LoadedEngine now reorders runner_ before the scheduler and builds an AsyncScheduler + max_concurrent_batches=2 when ResolveAsyncScheduling(runner_.runner_supports_async()) resolves ON (else the byte-identical synchronous Scheduler + depth-1); the resolved mcb threads into AsyncLLM→EngineCoreProc (step_with_batch_queue) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (_combine_sampled_and_draft_tokens_kernel + last_sampled scatter) is ported to CUDA (src/vt/cuda/cuda_combine_tokens.cu), main-stream-ordered on the CUDA async path so it DELETES sample_tokens_async's pre-scatter Synchronize; the CPU backend keeps the host loop. VT_ASYNC_RUNNER=1 engages full W3; VT_ASYNC_SCHED=0 is the same-binary rollback. Production default (no env) stays synchronous byte-identical. FULL W3 DGX proof RAN twice — f086b64 (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + Sampler greedy scratch removing ALL per-step cudaMalloc/cudaFree/cudaHostAlloc/event-create from the sampled-id path, incl. the overlap-killing cudaFree inside get_output; mirrors gpu_model_runner.py:873-878 + async_utils.py:12-70): token-exactness 6/6 PASS, interleaved c16 tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 % — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). DEFAULT FLIPPED ON 2026-07-17 (VT_ASYNC_RUNNER default ON via the pure AsyncRunnerFlagIsOn predicate, mirroring vllm/config/vllm.py:992-1044): the discriminator (6ea7856) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). VT_ASYNC_RUNNER=0 = runner-level rollback, VT_ASYNC_SCHED=0 = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. ROBUSTNESS FIX 2026-07-20 (discard_request_mask): the runner was missing vLLM's discard_request_mask, so GPUModelRunner emitted a sampled token for prefill-CHUNK requests too; under async this drained a num_output_placeholders never reserved (the is_prefill_chunk path adds none) → the async_scheduler.cpp num_output_placeholders >= 0 assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: execute_model computes exec_state_.discard[i] = seq_len < num_tokens (gpu_model_runner.py:2048); sample_tokens clears those rows to empty (outputs.py:303), the async path passes invalid_req_indices to AsyncGPUModelRunnerOutput::get_output (gpu_model_runner.py:3625 + outputs.py:303). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors scheduler.py:1888-1890). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, vllm-bench c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger parity-ledger.md 2026-07-20 row |
T1 | vllm/v1/core/sched/async_scheduler.py:12; vllm/config/vllm.py:490,990,1038; vllm/v1/engine/core.py:519; vllm/v1/worker/gpu/input_batch.py:304-406; vllm/v1/worker/gpu/async_utils.py:12-70; vllm/v1/worker/gpu/gpu_model_runner.py:242-332; vllm/v1/outputs.py:298-307 |
src/vllm/v1/core/sched/async_scheduler.cpp:10,45; placeholder plumbing src/vllm/v1/core/sched/scheduler.cpp:148,164,605; src/vllm/v1/engine/core.cpp:91 (step_with_batch_queue, async-output seam); src/vllm/v1/engine/core_proc.cpp:32,46; config include/vllm/config/scheduler.h:117,165,188, src/vllm/config/scheduler.cpp:12; include/vllm/v1/request.h:187; runner input leaf src/vllm/v1/worker/gpu/prepare_inputs.cpp, src/vllm/v1/worker/gpu/input_batch.cpp; runner output leaf include/vt/backend.h+src/vt/backend.cpp+src/vt/cuda/cuda_backend.cu (event/pinned), include/vllm/v1/worker/gpu/async_output.{h,cpp} (AsyncGPUModelRunnerOutput), src/vllm/v1/sample/sampler.cpp (sampled_ids_out), src/vllm/v1/worker/gpu/runner.cpp (sample_tokens_async/runner_supports_async), src/vllm/v1/executor/executor.cpp+include/vllm/v1/worker/gpu/model_runner_base.h (async seam); enable-flip include/vllm/entrypoints/model_loader.h+src/vllm/entrypoints/model_loader.cpp (runner_ before scheduler, ResolveAsyncEnabled/MakeScheduler, AsyncScheduler+mcb=2, log), include/vllm/v1/engine/async_llm.h+src/vllm/v1/engine/async_llm.cpp (mcb param → EngineCoreProc); device kernel include/vt/cuda/combine_tokens.h+src/vt/cuda/cuda_combine_tokens.cu, wired src/vllm/v1/worker/gpu/runner.cpp (CUDA combine/scatter branch removes the pre-sync) |
tests/vllm/v1/test_async_scheduler.cpp:1 (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle tests/vllm/v1/test_engine_core_proc.cpp:479 (mcb=2, async-output seam); config resolution tests/vllm/test_scheduler_config.cpp:75; enable-flip construction matrix tests/vllm/entrypoints/test_loaded_engine_dense.cpp (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf test_combine_tokens.cpp (RED = stale → 5/7 fail), test_input_batch.cpp, test_runner.cpp (async-ON≡sync); output leaf tests/vt/test_backend.cpp (event/pinned contract), tests/vllm/v1/worker/test_async_output.cpp (materialize/flush/snapshot; RED = +1 splice), test_runner.cpp (sample_tokens_async decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic 3812d8 six-leg control: total 1.002153×, TTFT 0.862159×, no GPU-time reduction (neutral for speed). DEFAULT-FLIP (2026-07-17): new pure CPU flag test test_async_runner_flag.cpp (11 asserts, default-ON/'0'-off); construction matrix test_loaded_engine_dense.cpp INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean -Werror rebuild, full serial ctest 116/116, tools 164/164. DGX re-confirmation (evidence dgx:~/work/vllm.cpp-async-flip, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → 27B 235/235 + 35B 315/315 with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (VT_ASYNC_RUNNER=0, VT_ASYNC_SCHED=0) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record parity-ledger.md#L502 |
async-serving.md | DONE |
6ea7856 |
ENG-PRIORITY-SCHED |
Priority request queue + policy + priority preemption + priority plumbing (Request/EngineCoreRequest/OpenAI field); W4 of the async-serving block. Default stays FCFS. GATING: full CPU tier green (93/93; 12 ported priority-scheduler cases + 14 priority-queue cases incl. the seeded random property test); GPU G1 (both greedy engine gates, priority-vs-fcfs token-exactness) deferred to the next GPU-idle window — GPU held by the SERVE-GATE-ONLINE campaign |
T1 | vllm/v1/core/sched/request_queue.py:131,201; vllm/v1/core/sched/scheduler.py:546; vllm/config/scheduler.py:109; tests/v1/core/test_scheduler.py:2382,2978; tests/v1/core/test_priority_scheduler_random.py:1 |
src/vllm/v1/core/sched/request_queue.cpp:101,186; src/vllm/v1/core/sched/scheduler.cpp:178; src/vllm/v1/request.cpp:92; src/vllm/config/scheduler.cpp:21 |
tests/vllm/v1/test_scheduler.cpp:674,916; tests/vllm/v1/test_request_queue.cpp:238,429 |
async-serving.md | GATING |
- |
ENG-PARTIAL-PREFILL |
Concurrent partial-prefill and long-prompt limits | T1 | vllm/config/scheduler.py:70-80 |
- | - | planned: specs/partial-prefill-concurrency.md |
INVENTORIED |
- |
ENG-BATCH-QUEUE |
Pipelined step_with_batch_queue |
T1 | vllm/v1/engine/core.py:519 |
- | - | planned: specs/batch-queue-step.md |
INVENTORIED |
- |
ENG-CORE-BUSY-LOOP |
Busy loop with input/output queue split (in-proc analog of the ZMQ EngineCoreProc boundary); W1 of the async-serving block. Implemented: EngineCoreProc (queues, run_busy_loop, shutdown drain/abort, WAKEUP, ENGINE_CORE_DEAD) + InprocClient on a dedicated engine thread; sync LLMEngine path untouched; UTILITY/DP/aborts-queue/step_with_batch_queue deferred per spec. GATING: CPU suites green; GPU G1 (token-exact twins) + G4 (no-throughput-regression) deferred to the gating handoff — GPU held by the SERVE-GATE-ONLINE campaign |
T0 | vllm/v1/engine/core.py:915,1259; vllm/v1/engine/core_client.py:467 |
include/vllm/v1/engine/core_proc.h:150; src/vllm/v1/engine/core_proc.cpp:51; src/vllm/v1/engine/core_client.cpp:33 |
tests/vllm/v1/test_engine_core_proc.cpp:188,314,395 (9 cases, 82 asserts; CPU ctest 93/93) |
async-serving.md | GATING |
- |
ENG-SCHED-KNOBS |
Reserve-full-ISL, scheduler class seam, stream interval | T1 | vllm/config/scheduler.py:26,127,140,163 |
include/vllm/config/scheduler.h:71,95; src/vllm/config/scheduler.cpp:43,52; src/vllm/v1/core/sched/scheduler.cpp:237; src/vllm/v1/engine/output_processor.cpp:68 |
tests/vllm/test_scheduler_config.cpp:10,20; tests/vllm/v1/test_kv_cache_manager.cpp:425 |
planned: specs/scheduler-knobs.md |
PARTIAL |
- |
ENG-CASCADE-ATTN |
Cascade attention for shared prefixes. SPIKED 2026-07-22: VERIFIED NOT OWED on the path we mirror, for three independent reasons — (1) disable_cascade_attn defaults True (opt-in per its own docstring), (2) the implementation exists ONLY on the legacy V1 runner gpu_model_runner.py:504,2544; a recursive grep for cascade over the MRV2 tree vllm/v1/worker/gpu/ returns ZERO hits and we port MRV2, (3) FlashAttention is the sole implementing backend while FlashInfer/Triton/Flex/ROCm all hard-return False, and Blackwell resolves FlashInfer FIRST — so it is unreachable on our GB10 gate box. Also forces full CUDA graphs OFF, and the shared-block count is hard-0 for SWA/chunked-local/Mamba (our hybrid gate models). The scheduler-side input is already computed/plumbed, so if upstream moves it into MRV2 only the backend half remains |
T2 | vllm/config/model.py:238,239-244; V1-only impl vllm/v1/worker/gpu_model_runner.py:504,2544,3869; backend support vllm/v1/attention/backends/flash_attn.py:670-671,1307-1319 vs flashinfer.py:1445-1452; Blackwell priority vllm/platforms/cuda.py:145-151; tests/v1/e2e/general/test_cascade_attention.py:20 |
common-prefix input only (not consumed): src/vllm/v1/core/sched/scheduler.cpp:330,381; include/vllm/v1/core/sched/output.h:151 |
per-manager zero policies tests/vllm/v1/test_single_type_kv_cache_manager.cpp:263 |
prefix-prompt-caching-parity.md | SPIKE |
CLAIM-PREFIX-PROMPT-CACHING |
ENG-DBO-UBATCH |
DBO and ubatch overlap | T2 | vllm/config/parallel.py:208,524 |
- | - | planned: specs/dbo-ubatch.md |
INVENTORIED |
- |
ENG-MOE-SHARED-AUX |
MoE shared-expert MLP on an aux CUDA stream concurrent with the routed-expert router/align/grouped-GEMMs (mirror vLLM's decode overlap; the largest remaining 35B c1/c2 engine lever). Fork the shared MLP onto a 2nd persistent per-device stream, join before the combine → byte-identical to serial (independent shared/routed paths both complete before combine; overlap changes WHEN not WHAT). Gated T <= threshold decode + CUDA. The aux stream draws scratch from a SEPARATE AuxPool so the concurrent main-stream routed allocations never share a live block with it (the DevicePool reuse invariant is single-stream ordering; vLLM sidesteps this with its stream-aware caching allocator's record_stream). VT_MOE_SHARED_AUX_STREAM DEFAULT ON (=0 rollback); VT_MOE_SHARED_AUX_THRESHOLD (default 128; GB10 48-SM calibration). Captured in the decode CUDA-graph via the fork/join event edges (ThreadLocal capture, no abort). Only the committed Marlin MoE decode path; wmma fallback/CPU/GGUF and 27B dense unaffected |
T1 | vllm/model_executor/layers/fused_moe/runner/shared_experts.py:99-104,125-142; vllm/utils/multi_stream_utils.py:20-58 (maybe_execute_in_parallel, TRT-LLM port); vllm/utils/torch_utils.py:736-756 (aux_stream); vllm/envs.py:260 (threshold 256) |
fork/join src/vllm/model_executor/models/qwen3_5.cpp:3999,4114 (MoeBlockFusedMarlinCuda); aux stream+events src/vllm/model_executor/models/qwen3_5.cpp:3575,3581 (MoeAuxStream/MoeAuxStreamFor); predicates :3553,3560; aux-pool isolation :496,3538 (AuxPool/ActivePool/ActivePoolScope) + DBuf pool_ routing :645 |
DGX (prod flags, one flock): overlap ON==OFF BYTE-IDENTICAL — tests/parity/test_qwen36_paged_engine.cpp 35B 315/315 + tests/parity/test_qwen27_paged_engine.cpp 27B 235/235 under VT_MOE_SHARED_AUX_STREAM∈{0,1}; captured-vs-eager (VLLM_CPP_CUDAGRAPH=0, ON) 315/315; shipping default (no env) 315/315+235/235, rollback =0 315/315+235/235; compute-sanitizer memcheck (default ON, captured) 0 errors; in-situ interleaved TPOT A/B (drop cold rep1) c1 −5.6% / c2 −2.7% / c4 −3.7% / c8 −3.4% / c16 −1.6% / c32 −1.5% (WINS every conc, zero regression); ledger parity-ledger.md 2026-07-19 row |
moe-shared-aux-stream.md | ACTIVE |
CLAIM-MOE-SHARED-AUX-1 |
ENG-RUNNER-MODELSHAPE |
Runner is model-shape-agnostic over the KV-cache group structure — the extensibility deliverable the first additive-model bring-up (Qwen3 dense) forced. Before W1 the GPUModelRunner had only ever executed the Qwen3.6 HYBRID topology and hardcoded it in two places: (#1) the KV-buffer alloc loop indexed config_.layer_types[l], out-of-bounds on a pure-dense model's EMPTY layer_types; (#2) each execute_model step unconditionally built the GDN metadata (gather_block_table(gdn_group_id_) / remap_gdn_state_slots / GDNAttentionMetadataBuilder), which reads block_table[-1] when there is no mamba group. W1 drives both off the resolved KV-group structure — a model-agnostic has_mamba_group / gdn_group_id_ >= 0 predicate (NOT a model-name check): empty/absent layer_types ⇒ all full-attention; no mamba group ⇒ the whole GDN metadata/state path is skipped and gdn_meta stays default-empty. A full-attention-only KV config (one FA group, no MambaSpec) now allocates + steps cleanly; the hybrid gate models keep their GDN group so their path is BYTE-IDENTICAL. This is a one-time generalization: every future dense/non-hybrid arch (Llama, Mistral) now adds new-files-only, zero further runner edits. PER-LAYER KV head_dim extension (Gemma-4 G1b, 2026-07-28, CLAIM-GEMMA4-G1B): the runner's full-attn alloc/view loops now consume an OPTIONAL KVCacheConfig::per_layer_attn_specs (index == layer) so a HETEROGENEOUS-head_dim model (Gemma-4: sliding 256 / global 512, same num_kv_heads) sizes each non-GDN layer's paged KV + PagedKvCache view from its OWN spec. The field is EMPTY for every uniform-KV model ⇒ the loop collapses to the single group spec ⇒ byte-identical allocation/view/indexing/dispatch (same additive-identical property as the model-shape generalization above). Block table / KV manager / scheduler stay head_dim-independent (num_blocks + block_size, uniform) so no per-group block table is introduced |
T0 | model-agnostic runner drives off kv_cache_config.kv_cache_groups — vllm/v1/worker/gpu/model_runner.py initialize_kv_cache / attention-metadata build (per-group, no hardcoded hybrid) @ e24d1b24 |
src/vllm/v1/worker/gpu/runner.cpp:458-470 (alloc loop: has_mamba_group && !layer_types.empty() gate) + :651-680 (GDN metadata build gated on gdn_group_id_ >= 0, default-empty gdn_meta otherwise); per-layer KV head_dim: include/vllm/v1/kv_cache_interface.h (KVCacheConfig::per_layer_attn_specs) consumed in src/vllm/v1/worker/gpu/runner.cpp initialize_kv_cache (per-layer FaDims alloc+view), published by src/vllm/model_executor/models/gemma4_registry.cpp (MakeGemma4ForConditionalGenerationKVCache); the full-attention-only KV spec that exercises the base path src/vllm/model_executor/models/qwen3_dense.cpp (MakeQwen3ForCausalLMKVCache) |
tests/vllm/v1/worker/test_runner.cpp — "full-attention-only KV config allocates without the GDN path" + "full-attention-only step skips GDN metadata build (no OOB)" (RED→GREEN: both SIGSEGV pre-generalization; GREEN post). Behaviour-preservation gate: DGX 27B 235/235 + 35B 315/315 UNCHANGED under the fix; per-layer-KV inertness: full CPU runner/KV suite green + OLMo-2 SACRED GPU re-gate 16/16 UNCHANGED; heterogeneous path proven by Gemma-4 E4B STRICT 32/32 (tests/parity/test_gemma4_paged_engine.cpp); ASan/UBSan clean on the affected paths |
first-additive-model-qwen3-dense.md §3 (seam gaps #1/#2), §6 (W1); gemma4-multimodal.md §G1b | ACTIVE |
CLAIM-MODEL-QWEN3-DENSE |
ENG-MM-INPUT-PIPELINE |
Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input. The C++ mirror of vllm/multimodal/: MultiModalKwargs/MultiModalFeatureSpec/MultiModalInputs, the MultiModalHasher mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> pixel_values+image_grid_thw) and placeholder-token expansion, plus the EncoderCacheManager (+ComputeMmEncoderBudget) and the LMCache extra_keys seam. Additive mm_features carried on Request/EngineCoreRequest; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). SERVING wiring (ROAD-V1-MM MM-SERVE-ENGINE, 2026-07-28, CLAIM-MM-SERVING-W2): the OpenAI server now carries the parsed MultiModalInputs into the engine — additive LLMEngine/AsyncLLM add_request(MultiModalInputs)+generate(MultiModalInputs) overloads via InputProcessor::process_inputs_mm (mirror input_processor.py:333-379, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (get_placeholder_str/_add_placeholder mirror), and the serving_chat MultiModalChatFn seam (default unset ⇒ text byte-identical). SEAM BODY (ROAD-V1-MM MM-SERVE-E2E W3, 2026-07-28, CLAIM-MM-SERVING-E2E): MakeQwen3VLImageChatFn (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → EncodeWithSpecialTokens (the single image_pad marker → one image_token_id) → RouteImageRgb EXPAND to 196 image tokens + mm_features; wired in examples/server/main.cpp (guarded on preprocessor_config.json; text-only unset ⇒ byte-identical). Gated test_chat_mm 8/8 + test_openai_serving (seam invoked + routed). ENGINE MM-FORWARD LANDED (ROAD-V1-MM MM-SERVE-E2E, 2026-07-28, CLAIM-ENGINE-MM-FORWARD): the engine model runner now HAS an mm forward — ModelForwardInput gains an ADDITIVE default-nullopt std::optional<MultiModalForwardInput> mm (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), Qwen3VLForConditionalGeneration is REGISTER_VLLM_MODEL-registered (qwen3_vl_registry.cpp), and the registered forward FOLDS the M2c decode into ModelRegistry::Forward via the SHARED Qwen3VLForwardStepLastLogits (Qwen3VLGenerateGreedyViaRegistry drives every step through the registry). GPU token-exact gate test_qwen3vl_registry_e2e (image→text THROUGH ModelRegistry::Forward == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness test_runner 16/16 + test_scheduler 36/36 + test_model_registry 24/24 + test_chat_mm 8/8 + test_openai_serving 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server /v1/chat/completions GPU e2e — recipe in specs/mm-serving.md. |
T1 | vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}; vllm/model_executor/models/qwen3_vl.py:{1400,1233}; vllm/v1/core/encoder_cache_manager.py:17; transformers image_processing_qwen2_vl.py:62, image_processing_backends.py:327; tests tests/multimodal/test_processing.py, tests/multimodal/test_hasher.py, tests/v1/core/test_encoder_cache_manager.py @ e24d1b24 |
src/vllm/multimodal/hasher.cpp, src/vllm/multimodal/qwen3vl_processor.cpp, include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}; src/vllm/v1/core/encoder_cache_manager.cpp + include/vllm/v1/core/encoder_cache_manager.h; additive inert fields include/vllm/v1/request.h + src/vllm/v1/request.cpp + include/vllm/v1/engine/types.h; extra_keys seam include/vllm/v1/kv_offload/lmcache/chunked_token_database.h + .cpp; M0 scripts/mm/m0_oracle_capture.py |
tests/vllm/multimodal/test_qwen3vl_processor.cpp (processor-parity 23/23 BIT-identical vs the M0 oracle fixture tests/vllm/multimodal/fixtures/qwen3vl/, RED-first: wrong normalize shift -> 1.2M mismatches); tests/vllm/v1/core/test_encoder_cache_manager.cpp 32/32. Text-inertness: test_request/test_engine_types/test_lmcache_codec/test_lmcache_key_agreement/test_openai_conformance all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; check-device-leakage OK |
multimodal-track.md §3 (M0/M1) | ACTIVE |
CLAIM-MULTIMODAL-M1 |
ENG-MM-VISION-TOWER |
Qwen3-VL vision tower Qwen3_VisionTransformer (M2a), proven faithful vs vLLM 0.25.0 in isolation. The reusable vision half of the whole Qwen3-VL family + Qwen3.6 (27B/35B share this exact tower). Pure-additive C++ forward composed from public vt:: ops: patch-embed (Conv3d-as-matmul + bias), host pos-embed bilinear-interp+spatial-merge-reorder, 24 ViT blocks (LayerNorm + vision attention with partial-rotary NeoX vision RoPE via vt::RopeFromCache + non-causal vt::Attention(causal=false) + tanh-GELU MLP), patch merger (LayerNorm + exact-erf-GELU + 2 FCs), DeepStack 3 post-shuffle-norm mergers at layers 5/11/17 → [196,10240]. Adds 2 additive elementwise vt ops (GeluTanh/GeluErf). NO runner/model/registry edit → text engines byte-identical by construction. Proven faithful in ISOLATION; the merge into input_embeds + the MRoPE/DeepStack text backbone + the e2e image gate are M2b/M2c. |
T1 | vllm/model_executor/models/qwen3_vl.py Qwen3_VisionPatchEmbed:347, Qwen3_VisionBlock:413, Qwen3_VisionPatchMerger:467, Qwen3_VisionTransformer:519, forward:800, pos_embed_interpolate_native:277, rot_pos_emb:667; qwen2_5_vl.py::Qwen2_5_VisionAttention.forward:397; rotary_embedding/common.py::ApplyRotaryEmb.forward_static:151 @ e24d1b24 |
src/vllm/model_executor/models/qwen3_vl_vision.{h,cpp}; 2 vt ops include/vt/ops.h + src/vt/ops.cpp + src/vt/cuda/cuda_layernorm.cu + src/vt/cpu/cpu_layernorm.cpp; dumps scripts/mm/m2a_tower_{ref,weight}_dump.py; fixtures tests/vllm/multimodal/fixtures/qwen3vl_tower/ |
tests/vllm/multimodal/test_qwen3vl_tower.cpp — 4 RED-first tower gates vs the dumped vLLM-0.25.0 reference 348/348 (patch-embed 2.1e-3, block0 6.8e-3, merger 6.5e-2, DeepStack 1.2e-2/3.3e-2/4.4e-2, full tower 5.1e-2; pos-embed 2.5e-3 + rope 1.9e-3 TIGHT); bf16-depth envelope RCA'd; RED = rope disabled → block0 0.149/tower 0.75/6 fails; cutlass-ON+FA2 banner; clean -Werror; compute-sanitizer 0 |
multimodal-track.md §3 (M2a) | ACTIVE |
CLAIM-MULTIMODAL-M2A |
ENG-MM-TEXT-BACKBONE |
Qwen3-VL text-backbone numeric contracts Qwen3VLGetRopeIndex/Qwen3VLMergeMultimodal/Qwen3VLComputeDeepstack (M2b/M2c), unit-green vs vLLM 0.25.0. The deterministic pieces that fork the plain Qwen3-dense text path for a vision-conditioned decode: (1) MRoPE 3-D get_rope_index positions [3,T] (image tokens get (t,h,w) grid positions, text sequential); (2) the 3-section MRoPE APPLICATION — proven to be the EXISTING vt::RopeFromCache mrope path (positions [3,T] + mrope_section=[24,20,20] interleaved), faithful to MRotaryEmbedding.forward_native for Qwen3-VL's exact config; (3) _compute_deepstack_embeds scatter → [L,T,H] decoder-injection tensor; (4) _merge_multimodal_embeddings masked scatter of the tower's [:,:2560] into input_embeds. Pure-additive TU — NO shared dense forward / runner / registry edit → text engines byte-identical by construction. The e2e image forward (VL weight loader + forked MRoPE/DeepStack decode loop) is the remaining M2c wire-up. |
T1 | vllm/model_executor/models/qwen3_vl.py _get_mrope_input_positions:2567, _iter_mm_grid_hw:2482, _compute_deepstack_embeds:2761, Qwen3LLMModel.forward deepstack :1589; vllm/model_executor/models/utils.py::_merge_multimodal_embeddings:524; vllm/model_executor/layers/rotary_embedding/mrope.py MRotaryEmbedding @ e24d1b24 |
src/vllm/model_executor/models/qwen3_vl_text.{h,cpp}; existing vt::RopeFromCache mrope path (src/vt/{cpu,cuda}/*); dump scripts/mm/m2b_text_ref_dump.py; fixtures tests/vllm/multimodal/fixtures/qwen3vl_text/ |
tests/vllm/multimodal/test_qwen3vl_text.cpp — 4 RED-first gates vs the dumped vLLM-0.25.0 reference 85/85 (get_rope_index BIT-exact [3,204], delta −182; MRoPE q rel-L2 1.5e-3 / k 1.5e-3, RED interleaved-off >5e-2; DeepStack + merge BIT-exact); CPU-only, no weights; clean CPU -Werror |
multimodal-track.md §3 (M2b/M2c) | ACTIVE |
CLAIM-MULTIMODAL-M2BC |
ENG-MM-QWEN36-VL-FORWARD |
Qwen3.6-27B (Qwen3_5ForConditionalGeneration) GDN-hybrid VL forward — IMAGE (M3-b) + VIDEO (M3d) BOTH e2e, STRICT gates PASS 32/32. Our own gate model's image+video paths now work end-to-end (speed pending). The genuinely-new integration completing our own gate model's mm paths: fork the landed bf16 Qwen3_5DenseModel GDN-hybrid forward (48 GDN + 16 full-attn) on gated, default-off points so a text-only 27B request stays byte-identical — (a) inputs_embeds entry (embed ids + Qwen3VLMergeMultimodal scatter of the 27B tower merger [N,5120] into the visual-token rows; 27B has EMPTY deepstack_visual_indexes ⇒ NO DeepStack); (b) 3-section MRoPE (mrope_section=[11,11,10] interleaved, rotary_dim 64, theta 1e7) in the 16 full-attn layers only via the proven vt::RopeFromCache mrope path (GDN layers carry no rope); (c) mixed load = the M2a Qwen3_VisionTransformer (27B vision config, empty deepstack) bf16 tower + the bf16 GDN-hybrid LLM via the EXISTING LoadQwen3_5Dense. M3d (2026-07-25) added VIDEO by REUSE: the M3-b image driver refactored into a shared VLGenerateCoreGdn, image+video wrappers differ ONLY in the merge mask (image_token vs video_token across frames) + the get_rope_index (Qwen3VLGetRopeIndex vs Qwen3VLGetRopeIndexVideo); the M3c processor/windowed-tower/video-MRoPE are reused verbatim. |
T1 | vllm/model_executor/models/qwen3_5.py:389 (Qwen3_5ForConditionalGeneration subclasses Qwen3VLForConditionalGeneration; visual = Qwen3_VisionTransformer, modalities {"image","video"}); qwen3_vl.py _process_video_input:2165, _get_mrope_input_positions:2567 video branch, get_video_repl:1479; the 27B config.json (mrope_section=[11,11,10], empty deepstack_visual_indexes) @ e24d1b24 / vLLM 0.25.0 |
M3-b + M3d BUILT + GATED 2026-07-25: vision-only loader LoadQwen3VLVisionWeights (src/vllm/model_executor/models/qwen3_vl.cpp, 27B config) + shared VLGenerateCoreGdn + image driver Qwen3_5VLGenerateGreedy + video driver Qwen3_5VLGenerateGreedyVideo + BuildMropeCosSinHost + the mrope_cos_sin param on DenseForwardLayers (src/vllm/model_executor/models/qwen3_5.cpp, nullptr on every text caller ⇒ byte-identical; the video driver is purely additive, the shared text forward UNTOUCHED per git diff --stat) reusing M2a tower + LoadQwen3_5Dense bf16 LLM |
IMAGE: golden tests/vllm/multimodal/fixtures/qwen3_5_27b/ (STRICT sha256 ead4b484…); STRICT image gate PASS 32/32 (test_qwen3_5_vl_e2e.cpp, 54/54, re-run post-refactor). VIDEO (M3d): oracle scripts/mm/m3d_video_oracle_capture.py on the M3c synthetic clip (raw sha 8a111599…, grid [4,8,8], 64 video tokens) K=5 DETERMINISTIC ⇒ STRICT golden; STRICT video gate PASS 32/32 (test_qwen3_5_vl_video_e2e.cpp, 27/27; near-tie gaps 0.0000 nats everywhere), fixtures tests/vllm/multimodal/fixtures/qwen3_5_27b_video/. Text-inertness 27B 235/235, 35B 315/315, Coder 138/138 (by construction); clean -Werror 0 warn; compute-sanitizer 0 on the 27B video forward. SPEED MEASURED (2026-07-26, CLAIM-MULTIMODAL-SPEED): image c1 vs vLLM 0.25.0 GRAPHED — decode TPOT 225.0 ms/tok vs 226.9 = AT PARITY (0.99×), LLM prefill 326 ms vs vLLM TTFT 321 ms = at parity; vision tower WAS 2114 ms vs vLLM encode ≤~250 ms = ~10× (THE gap). TOWER LEVER EXECUTED (2026-07-26, CLAIM-MULTIMODAL-SPEED-TOWER, multimodal-speed.md §7): nsys cuda_gpu_kern_sum attributed 98.9 % of the tower forward to the naive vt::cuda::AttentionKernel (56 ms/block; NOT QKV/FA2-routing); fixed by a warp-scoped online-softmax op AttentionDenseFast (separate op ⇒ kAttention/text byte-identical) + one-time resident-weight load ⇒ per-image tower 2114 → 148 ms (14.3×), 0.59× vs vLLM eager encode = FASTER. STRICT image/video e2e HELD 32/32 (+4B DeepStack 32/32), test_ops_attention 37239/37239, 27B text SACRED 235/235, compute-sanitizer memcheck 0, clean -Werror. benchmark_binding=false, single-seq driver (no c2+/server). Remaining: batched/graphed mm serving (c2+) + audio our-side — DONE bar not yet met. |
multimodal-track.md §M3 + multimodal-speed.md §7 + §8 (decode lever #2 CLOSED 2026-07-27: on-GPU greedy argmax + decode embed round-trip removed on VLGenerateCoreGdn; bit-exact — image/video STRICT 32/32 held; 27B decode NEUTRAL at the ~222 ms bandwidth floor) + §9 (lever #3 FIRST BRICK 2026-07-27, CLAIM-MULTIMODAL-SPEED-GRAPH: the shared VLGenerateCoreGdn decode step now routes through the production Qwen3_5DenseDecodeGraph cold→warm→replay captured decode — the mm decode is now GRAPH-CAPTURABLE, closing the un-graphed-eager-loop structural gap; S==B==1 bit-identical rebuild; token-exact HELD image/video STRICT 32/32 with 30 graph replays confirmed; A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor; the launch-overhead win + batched c2+ + serving ingestion are the recorded W-plan W1-W3) + §16 (vision-forward flash kernel 2026-07-28, CLAIM-MM-SPEED-QWEN-IMAGE: ATTRIBUTION-FIRST nsys attributed ~85% of the 148 ms tower forward to the dense attention AttentionWarpKernel [4.66 ms/block×27]; routed it to the §14 flash-tiled vt::AttentionDenseFlash [head_dim 72, byte-identical — per-warp math verbatim, only K/V from shared-mem tiles]. STRICT image/video e2e HELD 32/32 [27B+4B], test_ops_attention 37239/37239, goldens md5 UNCHANGED, nsys proof AttentionDenseFlashKernel 24 inst/zero warp, RED 30/46→46/46, sanitizer 0. A/B warp 148.3→flash 142.3 ms = 1.04× — the profile REFUTED a big lever: at t=784 the vision attention is serial-latency-bound not bandwidth-bound [audio §14 was 1.82× at t=1500], flash recovers only ~6 ms. HONEST: the tower ALREADY BEATS vLLM — 142 ms vs ~250 ms eager encode = 0.57×; image/video mm-forward is correctness-DONE + speed-BEATS-vLLM; residual = tensor-core MMA hd-72 attention [not needed for parity] + batched c2+/serving) |
ACTIVE |
CLAIM-MULTIMODAL-SPEED-TOWER + CLAIM-MULTIMODAL-SPEED-DECODE + CLAIM-MULTIMODAL-SPEED-GRAPH + CLAIM-MM-SPEED-QWEN-IMAGE |
ENG-MM-VIDEO-FORWARD |
Qwen3-VL VIDEO understanding (M3c) — preprocessing + full wiring LANDED + unit-gated; e2e token-exact PENDING on tower fidelity. Extends the landed image path to video: the genuinely-new piece is video PREPROCESSING (frame sampling + temporal grid + timestamp-interleaved placeholder); the tower handles temporal patches and MRoPE the temporal axis. NEW (additive to qwen3_vl*/multimodal TUs, ZERO text-path TU ⇒ text SACRED byte-identical): (a) ProcessVideo+VideoSmartResize+ComputeVideoTimestamps+BuildVideoRepl+VideoKwargs — video patchify fuses temporal_patch_size REAL frames/row (source frame = grid_t_idx*tp + t, NOT the image duplicate); BuildVideoRepl = per-frame [ts_ids]+vision_start+video_token*Nf+vision_end interleave; (b) tower per-frame windowed attention (cu_seqlens per frame; grid_t==1 image == byte-identical); (c) Qwen3VLGetRopeIndexVideo (per-frame scan); (d) Qwen3VLGenerateGreedyVideo via a shared VLGenerateCore (image driver unchanged). |
T1 | qwen3_vl.py: _process_video_input:2165 (same self.visual), _iter_mm_grid_hw:2482/_get_mrope_input_positions:2567 video branch, get_video_repl:1479, cu_seqlens per-frame :744; transformers video_processing_qwen3_vl.py:35,249 @ vLLM 0.25.0 |
M3c BUILT + UNIT-GATED 2026-07-25: src/vllm/multimodal/qwen3vl_processor.cpp (+inputs.h), qwen3_vl_vision.cpp (windowed attn), qwen3_vl_text.{h,cpp} (Qwen3VLGetRopeIndexVideo), qwen3_vl.{h,cpp} (Qwen3VLGenerateGreedyVideo+VLGenerateCore) |
Video-processor UNIT gate test_qwen3vl_video_processor 41/41, pixel_values_videos BIT-exact 0/393216 (RED-first: image-duplicate mapping → 195838 mismatch); video MRoPE BIT-exact vs vLLM (m3c_mrope_check.py, delta −48); video tower rel-L2 0.072 (within bf16 envelope, m3c_video_tower_ref_dump.py); video e2e test_qwen3vl_video_e2e NEAR-TIE-ROBUST PASS (gate form selected BY MEASUREMENT 2026-07-25, CLAIM-MULTIMODAL-TOWER-FIDELITY): teacher-forcing vLLM 0.25.0 on OUR exact sequence proves the sole divergence is ONE genuine bf16 near-tie at tok22 (' colorful' 33866 vs vLLM ' static' 1099, gap 0.125 nats, our token vLLM's 2nd of 4 tokens tied within 0.25 nats) and EVERY downstream token (tok23-31) IS vLLM's teacher-forced argmax at gap 0.0000 — 22/32 vs greedy is the one-token shift from that single tie. Tower accumulation ALREADY f32 everywhere (cuBLASLt CUBLAS_COMPUTE_32F GEMMs + f32 online-softmax attn + f32 LayerNorm) = matches vLLM's cuBLAS/FlashAttention; the residual rel-L2 is the irreducible inter-op bf16 rounding envelope, NOT a fixable numeric choice — so NO kernel change (methodology fix, mirrors the olmo2/qwen3-dense/glm4 near-tie gates). Gate: our_ids_i32.bin anchor + neartie_gap_mnats_i32.bin (from scripts/mm/m3c_video_neartie_gap.py), max gap 0.125 << 0.5-nat band. NO REGRESSION image e2e 4B STRICT 32/32 (the deterministic strict-pass proof); fixtures tests/vllm/multimodal/fixtures/qwen3vl_video/ + scripts/mm/m3c_*.py |
multimodal-track.md §M3 (M3c) | ACTIVE |
CLAIM-MULTIMODAL-TOWER-FIDELITY |
ENG-MM-AUDIO-PIPELINE |
AUDIO INPUT pipeline (audio-track A0+A1), the genuinely-new AUDIO modality on the modality-agnostic mm spine; INERT when no audio input. Stands audio up on the smallest oracle-runnable vehicle openai/whisper-small (native WhisperEncoder; transformers 5.13.1 constructs it — unlike Gemma-4 which is oracle-blocked). The C++ Whisper-class audio processor WhisperAudioProcessor: canonical PCM16-mono WAV decode (int16/32768.0), identity resample at 16 kHz (genuine windowed-sinc DEFERRED, mirrors the image SmartResize/bicubic deferral), log-mel input_features [80,3000] (pad/truncate 480000 → torch.stft-equiv: reflect-pad n_fft/2, periodic Hann, hop 160, drop last frame, direct DFT over 201 bins → abs(stft)^2 → mel_filters.T@mag → log10(clamp 1e-10) → max(x,x.max()-8) → (x+4)/4), audio placeholder expansion ([0]→[0]*1500, num_audio_tokens = max_source_positions = encoder output length), and MultiModalHasher::HashAudioF32 (float32 1-D ndarray "<f4"/(N,) byte stream, the audio analogue of the image u1/(H,W,3)). REUSE the M1 MultiModalKwargs/MultiModalFeatureSpec(modality)/EncoderCacheManager/LMCache extra_keys seam; adds a default-null AudioKwargs audio_data on MultiModalFeatureSpec. NO audio input ⇒ every field empty, text/image/video byte-identical. Does NOT build the audio ENCODER tower (A2) or the e2e audio→text (A3). CPU-only (no CUDA kernel). |
T1 | transformers feature_extraction_whisper.py (_torch_extract_fbank_features — the torch STFT path that runs when torch is installed; __init__ mel params), audio_utils.mel_filter_bank (slaney/slaney, dumped as a golden constant); vllm/model_executor/models/whisper.py:{103,469-476,656,738,740-753} (WhisperEncoder conv stride, get_num_audio_tokens=max_source_positions, _get_prompt_updates [0]→[0]*N); vllm/multimodal/hasher.py:{50,108-127} + processing/inputs.py::get_mm_hashes @ e24d1b24 / transformers 5.13.1 |
A0+A1 BUILT 2026-07-25: src/vllm/multimodal/audio_processor.{h,cpp}, include/vllm/multimodal/audio_processor.h; AudioKwargs+audio_data include/vllm/multimodal/inputs.h; MultiModalHasher::HashAudioF32 include/vllm/multimodal/hasher.h + src/vllm/multimodal/hasher.cpp (shared refactor: extracted FinalizeHex, byte-identical); CMakeLists.txt (1 source line); A0 capture scripts/mm/a0_audio_ref.py; fixtures tests/vllm/multimodal/fixtures/whisper_audio/ |
A1 audio-processor parity gate PASS 77/77 (tests/vllm/multimodal/test_audio_processor.cpp vs the A0 vLLM-0.25.0/transformers-5.13.1 oracle fixture): log-mel input_features rel-L2 1.96e-7 (stated 2e-4 band; torch.stft-FFT vs our DFT summation order, transformers' own torch/numpy claim is 1e-5 — we sit 2 orders tighter; ids+mm-hash BIT/BYTE-exact); WAV decode byte-identical 0 mismatches; placeholder [0]*1500 byte-identical; mm-hash 2d0c7e4c… byte-identical; RED-first (perturb largest mel weight → rel-L2 2.6e-3, wrong hop 161 → 0.70, skip (x+4)/4 → 9.27). Inertness (shared hasher.cpp/inputs.h): image processor 23/23 (hasher refactor inert — image mm-hash unchanged), video processor 41/41, request 71/71, encoder-cache 32/32, text backbone 85/85; clean CPU -Werror 0 warn; check-device-leakage unchanged (32==baseline); no CUDA kernel ⇒ compute-sanitizer N/A. benchmark_binding=false, speed pending |
audio-track.md §0 (A0/A1) | ACTIVE |
CLAIM-AUDIO-PIPELINE |
ENG-MM-AUDIO-ENCODER |
Whisper-class AUDIO encoder TOWER (audio-track A2), proven faithful vs the transformers-5.13.1 WhisperEncoder in ISOLATION. Consumes the A1 log-mel input_features [80,3000], produces encoder hidden states [1500,768] (the encoder half of audio understanding, toward A3 e2e audio→text on Voxtral-Mini-3B over the LANDED Mistral backbone). Pure-additive C++ forward composed from public vt:: ops: conv frontend as im2col + vt::MatmulBT (Conv1d(80→768,k3,pad1,s1)+GELU-erf → Conv1d(768→768,k3,pad1,s2 halving 3000→1500)+GELU-erf; NO new CUDA kernel — Whisper conv is a full cross-channel conv, not the depthwise vt::CausalConv1d), + fixed sinusoidal embed_positions (dumped golden constant), 12 pre-norm bidirectional encoder blocks (self_attn_layer_norm → q(bias)/k(NO bias)/v(bias) → vt::Attention(causal=false) scale=head_dim⁻⁰·⁵ → out_proj → residual → final_layer_norm → GELU-erf MLP → residual), + final layer_norm. All GEMMs bf16, norm/softmax f32. Delta from the M2a vision tower: NO patch-merger/DeepStack/RoPE (fully bidirectional + fixed additive sinusoid), a conv frontend not a patchify matmul, GELU-erf everywhere (vision used tanh-GELU MLP). NO runner/model/registry/other-model TU edit → every text/image/video/audio-pipeline gate byte-identical by construction. Proven faithful in ISOLATION; the projector + masked-scatter merge into the Mistral decoder + the e2e audio→text gate are A3. The USM-Conformer tower (Gemma-4/Granite family) is a SEPARATE tower delta (A2-follow, Granite-Speech-2b). |
T1 | transformers models/whisper/modeling_whisper.py WhisperEncoder.forward:641-721, WhisperEncoderLayer.forward:400-430, WhisperAttention.forward:298-368 (k_proj no-bias, q pre-scaled, scaling=head_dim⁻⁰·⁵), sinusoids:54 @ 5.13.1; cross-checked vllm/model_executor/models/whisper.py WhisperEncoder:458, WhisperEncoderLayer:353, WhisperMLP:322, conv stride :473-476 @ e24d1b24 |
src/vllm/model_executor/models/whisper_audio.{h,cpp} (+ include/vllm/model_executor/models/whisper_audio.h); CMakeLists.txt (1 source line); dumps scripts/mm/a2_audio_encoder_{ref,weight}_dump.py; committed golden fixtures tests/vllm/multimodal/fixtures/whisper_audio/enc_* |
A2 encoder-tower fidelity gate PASS 203/203 (tests/vllm/multimodal/test_whisper_audio.cpp vs the dumped bf16 transformers-5.13.1 WhisperEncoder reference, GPU under flock on a cutlass-ON build, sibling 27B NOT co-resident): post_conv rel-L2 4.7e-3, post_pos 2.8e-3, block0 6.6e-3, encoder-output 3.0e-2 (bf16-depth envelope ~0.28%/layer over 12 layers, matches M2a); bands post_conv/post_pos<8e-3 / block0<1.5e-2 / final<5e-2 (measured×1.6–2.3). RED-first: wrong conv-stride → post_conv 0.34 (FAIL), missing sinusoid → post_pos 0.86 (FAIL), skipped final-LN → 4.22 (FAIL); honest non-discriminators (GELU-tanh≈erf in-envelope; single conv-weight aggregate-insensitive). cutlass-nvfp4/fp8+FA2+Triton-AOT sm_121a banner CONFIRMED; clean CUDA + CPU -Werror 0 warn; im2col+existing GEMM ⇒ no new kernel, no compute-sanitizer needed; additive ⇒ check-device-leakage unchanged. benchmark_binding=false, speed pending |
audio-track.md §0b (A2) | ACTIVE |
CLAIM-AUDIO-ENCODER |
ENG-MM-AUDIO-E2E |
e2e AUDIO→TEXT on Voxtral-Mini-3B (audio-track A3) — the FIRST audio understanding in the tree. The full C++ pipeline: A1 log-mel input_features [128,3000] (Voxtral config: 128 mel/window 400/hop 160) → the A2 WhisperAudioEncoderForward at Voxtral's Whisper-large-v3 encoder config (d_model 1280/32L/20 heads/head_dim 64/ffn 5120/1500 src-pos) → downsample-concat reshape ([1500,1280]→[375,5120], factor 4) → AudioLanguageAdapter projector (w_in→GELU→w_out, no bias) → masked-scatter merge (Qwen3VLMergeMultimodal, modality-agnostic) into the LANDED Mistral/Llama decoder at the 375 audio-token-24 rows → forked greedy (VoxtralGenerateGreedy, shared dense_attn::AttnBlock, untied lm_head). Weight loader reads Voxtral consolidated.safetensors (mistral naming); the text q/k weights get the Meta-interleaved→HF-NeoX rope PERMUTE vLLM applies on the mistral load path (verified bit-exact permute(wq)==vLLM q_proj), v/o raw. Additive driver+loader gated on audio ⇒ text-only Mistral byte-identical. |
T-e2e | vllm/model_executor/models/voxtral.py embed_multimodal:382-412, AudioLanguageAdapter:660-668, load_weights:502-568, VoxtralEncoderModel:671-839 @ e24d1b24; text q/k permute mirrors vLLM's mistral load path (is_neox_style=True, llama.py:233-244) |
include/vllm/model_executor/models/voxtral.h + src/vllm/model_executor/models/voxtral.cpp; tests/vllm/multimodal/test_voxtral_e2e.cpp; oracle scripts/mm/a3_voxtral_oracle_capture.py + scripts/mm/a3_voxtral_neartie_gate.py; committed fixtures tests/vllm/multimodal/fixtures/voxtral_audio/; CMakeLists.txt (1 source line) + tests/CMakeLists.txt (test) |
A3 e2e audio→text gate PASS 14/14 (test_voxtral_e2e, GPU under flock, cutlass-ON, VLLM_VOXTRAL_SAFETENSORS→consolidated.safetensors; sibling 27B NOT co-resident). Gate form BY MEASUREMENT: vLLM greedy K=5 DETERMINISTIC ⇒ STRICT is the bar — STRICT prefix 33/48 exact vs vLLM greedy (log-mel rel-L2 7.7e-7; decoder proven token-exact: vLLM ref-audio→48/48). Bit-exact infeasible (encoder uses different bf16 GEMM/attn kernels than vLLM's cuBLASLt+FLASH_ATTN → 8.7% encoder rel-L2 = the A2 bf16-depth envelope over 32 layers), so the binding gate is the ratified near-tie-robust one (exactly as M3c/M3d): teacher-force vLLM on OUR sequence — worst gap 0.0 nats, 0 over-band failures; the SOLE greedy branch (pos 33) is a 4-way EXACT bf16 tie at -2.069 nats and every one of our 48 tokens is vLLM's teacher-forced argmax. RED evidence: the mistral q/k rope-permute bug drove text-only 1/22 & e2e 0/48 → after the fix text-only 22/22 & first token exact. INERT: additive only (git diff --stat = 2 modified lines [CMakeLists +1 src, tests/CMakeLists +test] + new files); Mistral text 541/541, A1 77/77, A2 203/203 re-run byte-identical; check-device-leakage unchanged; NO new CUDA kernel (reuses A2 im2col+GEMM + merge scatter) ⇒ no compute-sanitizer needed. Clean CUDA -Werror 0 warn, cutlass-ON banner CONFIRMED. benchmark_binding=false, speed pending. SPEED (2026-07-26, CLAIM-MULTIMODAL-SPEED, multimodal-speed.md): vLLM 0.25.0 GRAPHED denominator captured — Voxtral-Mini-3B audio c1 TTFT 43 ms, TPOT 41 ms/tok (388-tok prompt, 375 audio). OUR-SIDE UNMEASURED — build-blocked (no A3 binary on a reusable dgx tree; dgx disk 100% full ⇒ ENOSPC-risky). Because the 3B decode is cheap (~41 ms), audio is where the eager single-seq driver's per-token host overhead would NOT be hidden — the top follow-on measurement. UPDATE 2026-07-27 (CLAIM-MULTIMODAL-SPEED-DECODE, multimodal-speed.md §8): audio our-side MEASURED + decode lever #2 CLOSED. On-GPU greedy argmax + decode-embed round-trip removal ⇒ Voxtral decode TPOT 61.85 ms (band 61.73–61.94) vs 62.08 ms host = ~0.25 ms/tok (~0.4%) bit-exact win (14/14 held, near-tie seq 48/48). The §-hypothesis is REFINED: our audio decode is ~62 ms/tok eager (not 41), so the host round-trips are a THIN slice; the real 1.52× gap vs vLLM's 40.8 ms graphed is eager per-step launch overhead (lever #3, graphed decode). UPDATE 2026-07-27 (CLAIM-MM-SPEED-GRAPH-W1, multimodal-speed.md §10): lever #3 W1 — the Voxtral decode graph — LANDED. New VoxtralDecodeGraph (voxtral.{h,cpp}, the Voxtral-text sibling of Qwen3MoeDecodeGraph: pure full-attention over the same dense_attn::AttnBlock+vt::PagedAttention stack, no GDN); VoxtralGenerateGreedy's pure-decode loop now runs VoxtralDecodeGraph::Step (captures the exact ForwardLastLogits op sequence; S==B==1 bit-identical rebuild), eager fallback VT_MM_DECODE_EAGER=1. RED line HELD (proven-to-run, VT_DECODE_GRAPH_STATS: captured S=1 + 46 replays): 14/14 (near-tie seq 48/48, strict prefix 33/48); goldens md5 unchanged. A/B (throwaway VT_MM_DECODE_EAGER toggle, 6 reps/mode, rep0 dropped, steady-state): graphed 60.94 ms/tok (60.79–61.07) vs eager 61.71 (61.57–61.88) = −0.77 ms/tok (~1.25%), NON-OVERLAPPING — a real clean win, but it NARROWS the gap 1.52×→1.49× vs vLLM's 40.8 ms, does NOT close it. Honest refinement: the removable launch overhead was only ~1.25% of TPOT, so the ~20 ms/tok residual is per-step COMPUTE/kernel efficiency (vLLM's torch.compile-fused + graphed decode), NOT launch overhead. Structural value: Voxtral's Mistral/Llama stack now HAS a decode-graph class (last mm text stack without one) — prerequisite for batched c2+ (W2). UPDATE 2026-07-27 (CLAIM-MM-SPEED-DECODE-KERN, multimodal-speed.md §11): the ~20 ms/tok residual ATTRIBUTED to ONE kernel + a VALIDATED bf16-near-tie ceiling. nsys (graph-node trace) of the graphed decode: the gap is the decode ATTENTION — the naive scalar PagedAttentionKernel (1410 inst = 30 layers × 47 steps @ 723 µs/call = 21.7 ms/step, ~120× the KV-memory floor), NOT the GEMMs (cuBLAS gemvx, near-BW-floor) nor the norm/rope/silu glue. Voxtral (head_dim 128, GQA 32q/8kv bf16 causal) matches the fa2_decode_qwen3 path (the 1:1 flash_attn_varlen decode vLLM runs, DEFAULT-ON) EXCEPT the driver's single KV block block_size=444 is not ÷16, so decode fell to the scalar fallback. Rounding block_size to ÷16 routes decode through FA2 (flash_fwd_splitkv 1410 @ 18.5 µs = 0.65 ms/step, 39× faster): same-binary A/B (throwaway) TPOT 59.4→38.2 ms/tok (−21.2, ~36%, NON-OVERLAPPING); 38.2 = 0.94× vLLM's 40.8 ms — BEATS parity. BUT it changes the bf16 attention reduction order → flips the committed near-tie golden's exact-tie branch (repro 48→18) → repro==48 FAILS. The FA2 sequence is FULLY VALID (teacher-force vLLM 0.25.0: 0 divergences, worst gap 0.0000 nats, PASS — a different-but-equal greedy branch, not a bug). So the audio gap is a bf16 near-tie / golden-pinning ceiling: the win is real, validated and vLLM-1:1, but shipping it needs the near-tie golden regenerated (which the teacher-force PASS proves valid). The RED line forbids touching the golden, so the shipped byte-exact path keeps the scalar kernel (14/14, golden md5 unchanged, ~1.46–1.49× vLLM). RECORDS-ONLY (no code change; win is a one-line block_size÷16 + golden-regen away). Row stays ACTIVE/speed-pending. UPDATE 2026-07-27 (CLAIM-MM-SPEED-DECODE-KERN-ADOPT, multimodal-speed.md §12): USER-APPROVED — FA2 decode SHIPS as the Voxtral default; the LAST mm decode-speed gap is CLOSED (audio decode BEATS vLLM 0.97×). One-line block_size÷16 in VoxtralGenerateGreedy routes decode through FA2 varlen LaunchDecodeVarlenFA2Bf16 (dispatch fa2_decode_qwen3 needs block_size%16==0, cuda_paged_attn.cu:2621; seq still one block, slot==abs_idx unchanged). FA2-routing PROVEN (nsys --cuda-graph-trace=node): flash_fwd_splitkv 1410 @ 18.5 µs + combine 1410 @ 3.1 µs, ZERO PagedAttentionKernel in decode. Gate converted to the ratified near-tie DISTRIBUTIONAL form: binding correctness = the teacher-force PASS (result==PASS+n_divergent==0+over_band==0+worst_gap<=0.5), KERNEL-INDEPENDENT (scalar AND FA2 both PASS); strict prefix = token-exact vs vLLM greedy up to the first genuine bf16 exact tie — FA2 takes the OTHER side of the pos-18 2-way EXACT tie (24466 vs golden 1584, IDENTICAL logprob −1.9875, gap 0.000) ⇒ strict prefix 18 (was 33 for the scalar branch; both teacher-force-valid), asserted >=18; the old byte-match repro==48 (scalar branch) is downgraded to a determinism anchor (regenerated to the FA2 seq). voxtral_neartie.json md5 3d199c2d…→937b9ad3…; STRICT greedy golden voxtral_golden.json 8ab87b7e… UNCHANGED. Gate PASS 16/16 (strict prefix 18/48; teacher-force result=PASS, divergent=0, worst_gap=0.0, over-band=0; FA2 seq 48/48). Teacher-force (fresh, vLLM 0.25.0 on the FA2 seq): 0 divergent, worst gap 0.0000 nats, PASS. CAPTURE SAFETY (FA2 inside the captured VoxtralDecodeGraph): captured S=1 + 46 replays (all 48 tokens valid); compute-sanitizer memcheck ERROR SUMMARY 0 errors on the graphed-FA2-decode surface (text-only path: prefill + captured FA2 decode, 20 replays, 22/22, exit 0); 3 e2e runs byte-identical ⇒ capture-safe, ships as the DEFAULT graph path (no eager-FA2 fallback). A/B (same-binary, throwaway timer, VT_FA2_DECODE_QWEN3 toggle, 6 reps/mode rep0 dropped, steady-state): scalar 60.50 ms/tok (60.39–60.62) vs FA2 39.50 ms/tok (39.41–39.58) = −21.0 ms/tok (~35%), NON-OVERLAPPING; 39.50 = 0.97× vLLM 0.25.0 graphed 40.8 ms — BEATS parity. Clean CUDA -Werror 0-warn; additive/kernel-config only (kAttention untouched ⇒ text/other-model byte-identical). Audio DECODE now correctness- AND speed-DONE (ratified near-tie + BEATS vLLM); row stays ACTIVE/PARTIAL — the every-axis DONE bar has audio TTFT (32-layer Whisper encoder, UNMEASURED our-side vs vLLM 43 ms) + batched c2+ / audio_url serving ingestion still open (same structural gaps as image/video). UPDATE 2026-07-27 (CLAIM-MM-SPEED-AUDIO-ENC, multimodal-speed.md §13): audio ENCODER TTFT MEASURED our-side + a warp-attention brick LANDED (4.7×), NOT at parity. The Whisper encoder ran the naive vt::Attention (kAttention, O(t²) per-key block-__syncthreads) over the non-causal 1500-frame context — nsys-attributed as the encoder's dominant kernel. Routed the encoder self-attention (head_dim 64, non-causal) to the warp-scoped vt::AttentionDenseFast (the §7 vision-tower fix; kAttention untouched ⇒ text byte-identical). One src file (whisper_audio.cpp), VT_WHISPER_ENC_EAGER=1 fallback. RED line HELD: test_voxtral_e2e 16/16 with the fast kernel default (strict prefix 18/48, teacher-force PASS, seq 48/48); the naive arm ALSO passes 16/16 with the SAME tokens ⇒ warp kernel flips ZERO tokens (bit-exact at token level, like §7's 32/32); goldens md5 UNCHANGED (voxtral_golden.json 8ab87b7e…, voxtral_neartie.json 937b9ad3…). Proof-of-run: nsys shows AttentionWarpKernel 32 inst (= 32 layers), zero naive. A/B (throwaway VT_WHISPER_ENC_TIME, 6 reps rep0 dropped): encoder forward 8870 ms → 1890 ms (4.7×, NON-OVERLAPPING). HONEST verdict — NOT closed: ~1.89 s vs vLLM's 43 ms TTFT (~44×). nsys of the fast arm: AttentionWarpKernel STILL 31.8 ms/layer × 32 = 1.02 s — O(t²) memory-bound on redundant K/V reads (~5.7 GB/layer, no shared-mem tile reuse); the rest is per-call host weight marshalling + conv round-trip. Ranked residual levers (grounded, NOT implemented): (1) flash-TILED non-causal hd-64 encoder attention (vLLM flash_attn_varlen_func; vendored FA2 has non-causal templates but only hd {128,192,256} paged → needs hd-64 + dense layout — LARGE); (2) resident one-time encoder weights + drop conv round-trip (MEDIUM, byte-exact). Row stays ACTIVE/PARTIAL. |
audio-track.md §1 (A3) + multimodal-speed.md §8/§10/§11/§12/§13 | ACTIVE |
CLAIM-AUDIO-E2E + CLAIM-MULTIMODAL-SPEED-DECODE + CLAIM-MM-SPEED-GRAPH-W1 + CLAIM-MM-SPEED-DECODE-KERN + CLAIM-MM-SPEED-DECODE-KERN-ADOPT + CLAIM-MM-SPEED-AUDIO-ENC |
ENG-SGLANG-BEHAVIOR-FLAG |
SGLang-alike runtime behavior scope + the enable/disable control (the "etc." beyond RadixAttention). Per-technique fuse-or-flag: (1) cache-aware LPM scheduling — SGLang reorders the waiting queue by longest matched prefix (schedule_policy.py LPM/DFS + in-batch prefix caching); our scheduler is FCFS/priority only, NO cache-hit ordering ⇒ genuinely DISTINCT ⇒ FLAG --schedule-policy=lpm (SGLang's own default is fcfs, so the default is already covered; lpm is opt-in). (2) overlap/zero-overhead scheduler — SGLang event_loop_overlap == our ENG-ASYNC-SCHED (DONE, default-ON) ⇒ already FUSED, no flag. (3) jump-forward decoding — grammar FSM-forced token elision, our structured-output path masks per-step with no elision ⇒ DISTINCT but opt-in ⇒ FLAG (deferred) --enable-jump-forward. Survey (chunked-prefill/continuous-batching/DFlash/ngram/priority/HiCache/Mamba-radix) = already-covered. Optional --sglang-compat umbrella composes the switches. SGLang pin v0.5.15 f63458b. |
T2 | SGLang cache-aware managers/schedule_policy.py:155,139,176,205,253 (LPM/in-batch), default server_args.py:692; overlap managers/scheduler.py:1563,344, server_args.py:776; jump-forward constrained/outlines_jump_forward.py:182,146,159; eviction strategies server_args.py:739 |
LPM landed (SW1, CLAIM-SGLANG-IMPL): SchedulerPolicy::kLPM (include/vllm/config/scheduler.h, src/vllm/config/scheduler.cpp "lpm"); reorder Scheduler::maybe_reorder_waiting_for_lpm() src/vllm/v1/core/sched/scheduler.cpp (ToQueuePolicy kLPM→FCFS deque; stable descending sort by match, >128-waiting fcfs fallback, ported FROM schedule_policy.py:205,229); side-effect-free match KVCacheManager::num_matched_prefix_tokens() src/vllm/v1/core/kv_cache_manager.cpp (pure find_longest_cache_hit, no stats/LRU touch); RequestQueue::reorder() src/vllm/v1/core/sched/request_queue.cpp; --schedule-policy/--scheduling-policy (fcfs / priority / lpm) + lpm+cache-off→fcfs warn examples/server/main.cpp; overlap == ENG-ASYNC-SCHED; grammar src/vllm/v1/structured_output/manager.cpp:50; SW2 landed (CLAIM-SGLANG-SW2): in-batch prefix-collision de-prioritization inside the kLPM reorder — maybe_reorder_waiting_for_lpm() src/vllm/v1/core/sched/scheduler.cpp:183-243 walks the pre-sort (arrival) queue building an ephemeral seen-set of OUR block-hash APC keys (NOT a second trie), de-prioritizes a later request whose in-batch prefix match ≥ kInBatchDeprioritizeThreshold when its real match ≤ kInBatchCheckThreshold (both 32, scheduler.h), sort key mirrors SGLang's float("inf"); ported FROM schedule_policy.py:253-301,311. Output-neutral; throughput lever NOT-APPLICABLE (our APC caches at allocation kv_cache_manager.cpp:267 → 2nd same-step collider already hits, no redundant prefill vs SGLang's post-forward radix). SW3 jump-forward safe subset landed (CLAIM-SGLANG-SW3): forced-token detection hook StructuredOutputGrammar::forced_token() (include/vllm/v1/structured_output/backend_types.h; native impl src/vllm/v1/structured_output/backend_native.cpp, trie×FSM DFS short-circuiting on a 2nd valid token) + opt-in driver DrainForcedTokens (src/vllm/v1/structured_output/jump_forward.{h,cpp}, env VT_ENABLE_JUMP_FORWARD, default OFF); jumps ONLY the TOKEN-UNIQUE forced run (exactly one grammar-valid token at a non-accepting state ⇒ sampler's only finite-logit token ⇒ PROVABLY byte-identical to per-token decode, no re-tokenization); ported FROM constrained/outlines_jump_forward.py:146-172 + schedule_batch.py@935cda944b^:503-544/:1094-1145. RESIDUAL: the general byte-forced multi-tokenizable span (SGLang's re-tokenize + boundary rollback) deliberately NOT jumped (falls back to normal decode); production scheduler splice needs the jumped-token KV recompute. ABI/API/flag EXPOSURE (CLAIM-SGLANG-ABI-DOCS, 2026-07-28, reconciled to ABI v10): LPM + jump-forward are now first-class DOCUMENTED knobs on all three surfaces (were server-only / env-only). LPM: C++ EngineParams::policy=kLPM, C-ABI vllm_model_params.scheduling_policy="lpm" (the concurrent session's v9 string field; NO duplicate int knob), server --scheduling-policy lpm. Jump-forward: C++ EngineParams::enable_jump_forward (std::optional<bool>), C-ABI vllm_model_params.enable_jump_forward (tri-state int, ABI v10 appended after the v9 fields), server --[enable|disable]-jump-forward; VT_ENABLE_JUMP_FORWARD retained as override (JumpForwardEnabled(optional<bool>), env-wins-when-set). Resolution observable via LoadedEngine::jump_forward_enabled() / Scheduler::policy(); the jump-forward tri-state translation is folded INLINE in vllm_engine_load (src/capi/vllm_c.cpp, single field — no shared helper). ABI e2e tests/capi/test_capi.cpp (2 v10 jump-forward cases; LPM-via-ABI covered by the concurrent session's scheduling_policy string cases). vllm_abi_version()==10. Docs specs/sglang-enablement.md + docs/SGLANG-COMPAT.md. |
tests/vllm/v1/test_scheduler_lpm.cpp 6/6 (47 asserts): SW1 output-neutral + reorder RED-first + hits-under-pressure + cache-off→fcfs (unchanged 3), SW2 de-prioritizes collider RED-first (fcfs/raw-prefix admits collider before solo; lpm+SW2 admits solo before collider) + output-neutral (per-req tokens + total hits 112 identical) + within-step-dedup-subsumes-redundant-prefill (NOT-APPLICABLE proof) + inert-when-prefix-cached (check-threshold gate); inertness test_scheduler 36/36 + test_prefix_cache_stats 12/12 unchanged. SW3 jump-forward safe subset gate tests/vllm/v1/structured_output/test_jump_forward.cpp 5/5 (40 asserts): output-identity WITH vs WITHOUT jump over a forced span + jump-FIRED (4 steps→1) + inert-no-forced-span + default-off untouched + RED (naive longest-match re-tokenization of a boundary-ambiguous span emits DIFFERENT tokens; safe subset refuses and stays byte-identical); inertness test_backend_native 35/35 + test_structured_output 12/12 + test_apply_grammar_bitmask 5/5 + test_response_format_e2e 3/3 unchanged. SW4 deferred; cache-ON throughput A/B owned by BACKEND-GATE-CUDA-SGLANG-PREFIX |
sglang-radixattention.md | ACTIVE |
CLAIM-SGLANG-IMPL |
ENG-PLUGIN-SYSTEM |
Out-of-core PLUGIN system: the discovery + orchestration layer that lets an external translation unit / shared object register a model factory / platform / quant method through the EXISTING registration seams (REGISTER_VLLM_MODEL, RegisterPlatform, the quant registry) WITHOUT editing engine core — directly serves the extensibility-first priority. W0 SPIKE + W1 CPU BRICK LANDED 2026-07-29 (CLAIM-PLUGIN-SYSTEM, NOT pushed): vllm::plugins::LoadGeneralPlugins() (idempotent load-once latch + the VLLM_PLUGINS allowlist parse + per-plugin try/catch failure isolation, all mirroring load_general_plugins) + the general-plugin registration seam RegisterGeneralPlugin / the REGISTER_VLLM_GENERAL_PLUGIN macro (mirror of the vllm.general_plugins entry-point group). Python entry points have NO C++20 analogue, so discovery is the project's static-init/dlopen registration idiom (recorded porting-inventory §9). Proven by an OUT-OF-CORE toy-model plugin TU (compiled only into the test exe) that registers a toy arch via the public vllm::RegisterModel seam: RED-first — the toy arch throws "are not supported for now" before load AND under VLLM_PLUGINS="", and resolves to the plugin factory ONLY after LoadGeneralPlugins runs it. RESIDUALS (named, spec §Work breakdown): real .so dlopen + the C-ABI vllm_plugin_register entry (W2), the engine/CLI --load-plugins wiring (W3), the platform/quant plugin kinds (W4), io_processor/stat_logger/endpoint groups (W5). |
T2 | vllm/plugins/__init__.py:18,33,36-74,77-90 (group, latch, load_plugins_by_group allowlist + failure isolation, load_general_plugins); vllm/envs.py:1104-1108 (VLLM_PLUGINS); vllm/model_executor/models/registry.py:1039-1083 (register_model a plugin calls); vllm/v1/worker/worker_base.py:245-247; vllm/v1/engine/core.py:115-117 (invocation sites); tests/plugins_tests/test_oot_registration_offline.py:14-45 |
include/vllm/plugins/plugins.h; src/vllm/plugins/plugins.cpp:74,80,111; CMakeLists.txt:464 |
tests/vllm/plugins/test_plugin_system.cpp:52; tests/vllm/plugins/toy_model_plugin.cpp:99 (1 case / 29 assertions, RED-first, CPU) |
plugin-system.md | ACTIVE |
CLAIM-PLUGIN-SYSTEM |
Maintenance checkpoint for KV-OFFLOAD (2026-07-31): GCC 12 -Werror
portability is restored for the thread-local temporary suffix with append
operations. Suffix values, atomic publication behavior, and the row's ACTIVE
lifecycle are unchanged.
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
KV-BLOCK-POOL |
Free list, refcounts, LRU eviction, core block lifecycle. Record corrected 2026-07-22 (ANCHOR-BACKFILL -> PARTIAL): the spike gap is closed by prefix-prompt-caching-parity.md, and the row's stated scope UNDERSOLD the port — it also covers group-aware hash mapping, the partial-alias map, the unhashed-front/hashed-tail free ordering that is what actually makes eviction LRU (the queue itself is plain FIFO), reset with the null-block invariant, usage and duplicate-hash eviction, under 13 tests. PARTIAL (not DONE) because four upstream behaviours are genuinely absent, all throw-if-called or inert and none reachable from a ported call site: KV-EVENTS — the emission is now wired at the store/remove/clear sites, guarded default-OFF so this row stays byte-identical), partial-block primitives (upstream's are dead code, so not owed live), connector-driven evict_blocks, and the align path where block_size != hash_block_size |
T0 | vllm/v1/core/block_pool.py:144,163-197,199-224,226-356,542,574-595,597-612,614-635,656-690,700-711; dead partial primitive :358-457; queue vllm/v1/core/kv_cache_utils.py:179-408; tests/v1/core/test_prefix_caching.py:1315,1991 |
src/vllm/v1/core/block_pool.cpp:42,77,206,231,254; deferrals recorded include/vllm/v1/core/block_pool.h:18-46 |
tests/vllm/v1/test_block_pool.cpp:70,115,202,251,280,308,337,384,398 |
prefix-prompt-caching-parity.md | PARTIAL |
- |
KV-MANAGER-ALLOC |
Slot allocation, watermark, admission, release | T0 | vllm/v1/core/kv_cache_manager.py:110,244; tests/v1/core/test_single_type_kv_cache_manager.py:380,413 |
src/vllm/v1/core/kv_cache_manager.cpp:88,124,144 |
tests/vllm/v1/test_kv_cache_manager.cpp:119,168,298,349,380,425 |
planned: specs/kv-cache-manager.md |
ANCHOR-BACKFILL |
- |
KV-DEVICE-RESIDENCY |
Persistent full-attention KV plus GDN convolution/recurrent state must be backend-resident; indexed mixed-prefill state I/O replaces row-wise host round trips. W0 ownership and W1 indexed gather/scatter are implemented and component-gated; W2 direct convolution update, fresh oracle closure, and inherited pool teardown remain | T0 | allocation/views vllm/v1/worker/gpu/attn_utils.py:166-182,327-346; runner ownership vllm/v1/worker/gpu/model_runner.py:478-488; indexed conv/SSM vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py:1309-1375,1503-1532; tests tests/v1/worker/test_gpu_model_runner.py:968,1265; tests/v1/worker/test_mamba_utils.py:342-358 |
W0 owner/allocation include/vllm/v1/worker/gpu/runner.h:173-200,249-266, src/vllm/v1/worker/gpu/runner.cpp:305-330,388-450; W1 API/dispatch include/vt/ops.h:867-877, src/vt/ops.cpp:840-895; CPU/CUDA kernels src/vt/cpu/cpu_ops.cpp:865-909, src/vt/cuda/cuda_gdn.cu:156-232; persistent metadata and mixed-path integration src/vllm/model_executor/models/qwen3_5.cpp:1565-1653,1737-1892; diagnostic fallbacks VT_DEVICE_KV_CACHE=0, VT_GDN_INDEXED_STATE_IO=0 |
W0 gates remain green: c16/48 785.49/769.15 tok/s = 1.021239×, 20/20; access memcheck 234/234 + 315/315. W1 op/mask and mixed-turnover tests tests/vt/test_ops_gdn.cpp:994-1136, tests/vllm/models/test_qwen27_paged_forward.cpp:443-532; focused CPU/access-sanitizer/CUDA/op, indexed/fallback 27B+35B, turnover smoke and current local serial CTest 105/105 pass. Strict local model LSan retains 58,624 B/153 pooled allocations while the indexed op is leak-clean; earlier remote suites exposed the unrelated intermittent C-API timing flake. W1 c16/48 781.799/776.946 tok/s = 1.006246×, 20/20 axes, six memory returns; traces collapse async copies 163,540→7,508 and D2D calls 142,717→1,231. Manifest 34285a91…a5b; profiler ratio is invalidated by unequal perturbation. W0 full leak-check still fails inherited pools at 47.29 MB/36.82 GB while W0 caches are absent |
device-resident-kv-gdn-state.md | ACTIVE |
CLAIM-KV-DEVICE-1 |
KV-HYBRID-COORD |
Full-attention and GDN/Mamba group coordinator. Record corrected 2026-07-22: the row read as breadth-incomplete, but the ENTIRE cross-group intersection is ported verbatim — the iterate-to-fixed-point loop (a single-pass intersection would be WRONG), the full-attention downward-closed fast path with its not-yet-looked-up sentinel, EAGLE extra-block/drop bookkeeping, the is_simple_hybrid shortcut, final full-attention truncation and num_uncached_common_prefix_tokens, plus the two-phase local-then-external touch. Stays PARTIAL for a narrow, assert-guarded residue only: differing per-group block sizes, DCP/PCP > 1, cross-attention encoder branch, find_longest_cache_hit_per_group, and the retention-interval env read |
T0 | vllm/v1/core/kv_cache_coordinator.py:61-374,187-231,514,560-600,630-740,742-779,782-834; tests/v1/core/test_prefix_caching.py:347,836,987 |
src/vllm/v1/core/kv_cache_coordinator.cpp:294,370,389; intersection :424-537; sentinel/EAGLE :281-323; truncation :507-521; guards :344,348-349 |
tests/vllm/v1/test_kv_cache_coordinator.cpp:130,154,169,209,306 |
prefix-prompt-caching-parity.md | PARTIAL |
- |
KV-MAMBA-ALIGN |
Mamba/GDN prefix retention in align mode; required for the matched vLLM/SGLang shared-prefix cache-on gate. mamba_cache_mode has THREE values (all/align/none): APC-on + none resolves to all when the model supports it else align, all on an unsupported model downgrades to align, align ASSERTS chunked prefill, and APC-off forces none; Qwen3.5/3.6 raise on all, so they take align. Record CORRECTED 2026-07-22 — this row was INVENTORIED with no code anchor, and that understated reality: the align ALLOCATOR is already substantially ported (mode read + branched across remove_skipped_blocks, get_num_blocks_to_allocate, allocate_new_blocks, pop_blocks_for_free, with reachable_block_mask and retention_interval). Three specific things are missing, not open-ended breadth: (1) no config path SELECTS align — MambaSpec defaults "none" and no caller overrides it, so production is effectively mode-none; (2) the RUNNER-side recurrent state copy the align path presumes does not exist (zero hits for last_state_block_idx under src/vllm/v1/worker/gpu/); (3) no align-mode test — all four local Mamba cases exercise mode "none" |
T1 | default policy vllm/config/model.py:1842-1847; three-value resolution vllm/model_executor/models/config.py:550-593; Qwen constraint vllm/model_executor/models/qwen3_5.py:297; manager vllm/v1/core/single_type_kv_cache_manager.py:1026,1196-1204,1265-1327,1341,1364; config vllm/config/cache.py:38,134; e2e tests/v1/e2e/general/test_mamba_prefix_cache.py:745,803 (the _mrv2 case is the binding one) |
align allocator ALREADY ported src/vllm/v1/core/single_type_kv_cache_manager.cpp:639,687-700,737,773,803,865; spec field defaulting to none include/vllm/v1/kv_cache_interface.h:306,318 |
mode-none coverage only tests/vllm/v1/test_single_type_kv_cache_manager.cpp:811,849,866,875 |
prefix-prompt-caching-parity.md | SPIKE |
CLAIM-PREFIX-PROMPT-CACHING |
KV-SLIDING-LOCAL-SPECS |
Block row (claim the two leaves below, not this row): sliding-window and chunked-local KV specs | T1 | vllm/v1/kv_cache_interface.py:205-307,480-586; tests/v1/test_kv_cache_spec_registry.py:174-306 |
- | - | sliding-local-yarn-long-context.md | READY |
- |
KV-SLIDING-WINDOW-SPEC |
SlidingWindowSpec sizing, grouping, admission, allocation, eviction, and prefix-cache policy; CPU G1/G2 green, while feature-positive attention/model/oracle/performance gates remain |
T1 | vllm/v1/kv_cache_interface.py:518-586; vllm/v1/core/single_type_kv_cache_manager.py:669-873; tests/v1/core/test_single_type_kv_cache_manager.py:127,259,380,413,489; tests/v1/core/test_prefix_caching.py:2457-3909 |
include/vllm/v1/kv_cache_interface.h:187; src/vllm/v1/kv_cache_spec_registry.cpp:69; src/vllm/v1/core/single_type_kv_cache_manager.cpp:350,377,470,920; src/vllm/v1/core/kv_cache_utils.cpp:21; src/vllm/v1/core/kv_cache_coordinator.cpp:36,119 |
tests/vllm/v1/test_kv_cache_interface.cpp:157,204,258; tests/vllm/v1/test_single_type_kv_cache_manager.cpp:283,331,368,411,453,476; tests/vllm/v1/test_kv_cache_utils.cpp:592,617; tests/vllm/v1/test_kv_cache_coordinator.cpp:163,238,357 |
sliding-local-yarn-long-context.md | GATING |
- |
KV-CHUNKED-LOCAL-SPEC |
ChunkedLocalAttentionSpec sizing, grouping, admission, allocation, fixed-chunk prefix-cache/recycling policy and hybrid-disabled fallback; CPU G1/G2 green, while W4/model/oracle/runtime gates remain |
T1 | vllm/v1/kv_cache_interface.py:480-514; vllm/v1/core/single_type_kv_cache_manager.py:876-1023; vllm/v1/core/kv_cache_utils.py:1403-1496; tests/v1/core/test_single_type_kv_cache_manager.py:54,198,456; tests/v1/test_kv_cache_spec_registry.py:174-315 |
include/vllm/v1/kv_cache_interface.h:219; src/vllm/v1/kv_cache_spec_registry.cpp:71; src/vllm/v1/core/single_type_kv_cache_manager.cpp:535,553,618,933; src/vllm/v1/core/kv_cache_utils.cpp:21; src/vllm/v1/core/kv_cache_coordinator.cpp:47 |
tests/vllm/v1/test_kv_cache_interface.cpp:188,204,258; tests/vllm/v1/test_single_type_kv_cache_manager.cpp:576,643,683,705,730,1072; tests/vllm/v1/test_kv_cache_utils.cpp:629,654,674,686; tests/vllm/v1/test_kv_cache_coordinator.cpp:188,258,380,524 |
sliding-local-yarn-long-context.md | GATING |
- |
KV-FP8 |
FP8 KV cache and scale handling. W0 spike + W1 CPU brick LANDED 2026-07-29 — fp8-e4m3 K/V STORE (Quantize(hp/scale)) + the paged-attention READ dequant (Dequant(fp8)*scale) + the cache_dtype config parse, all CPU-gated RED-first. Storage is 1-byte fp8 (DType::kI8) + the Fp8KVCacheDataType interpretation enum (mirrors vLLM's cache_t=uint8_t+KV_DTYPE), per-tensor k/v scales (kv_cache.py:108-191). Residuals (honest, named): the CUDA fp8 store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + --kv-cache-dtype/--calculate-kv-scales), fp8_e5m2 CPU compute + per-head scales — all W2-W5 in the spec |
T1 | vllm/config/cache.py:19-36,76; vllm/model_executor/layers/quantization/kv_cache.py:42,108-191; store csrc/libtorch_stable/cache_kernels.cu:241-252,314-401; scale convention csrc/quantization/w8a8/fp8/nvidia/quant_utils.cuh:296-308 |
codec include/vt/fp8_kv.h; store src/vt/cpu/cpu_cache.cpp:143; wrapper src/vt/ops.cpp:2255; read dequant src/vt/cpu/cpu_paged_attn.cpp:82; config parse include/vllm/v1/kv_cache_dtype.h:37 |
tests/vt/test_ops_fp8_kv_cache.cpp:1 (8 cases / 511 assertions; RED-first: wrong store direction fails 3/480) |
fp8-kv-cache | ACTIVE |
CLAIM-KV-FP8 |
KV-NVFP4-TURBO |
NVFP4, per-token-head, and TurboQuant KV | T2 | vllm/config/cache.py:14,28-35,272 |
- | - | planned: specs/nvfp4-kv-cache.md |
INVENTORIED |
- |
KV-OFFLOAD |
KV offload tiering: CPU primary tier plus secondary tiers, including the filesystem (disk) tier that is vLLM's KV-persistence-to-disk answer. Record CORRECTED 2026-07-22 (spike) — the prior row text named a class that does not exist and omitted the half the user asked for. There is no LRUOffloadingManager at this pin: LRU and ARC are pluggable CachePolicy objects behind ONE CPUOffloadingManager, and the row's scope ('CPU tiering with LRU and ARC') left out the entire secondary-tier surface. Disk format enumerated: ONE RAW FILE PER BLOCK, no container and no index, <root>/<model>_<sha256 prefix>_r<rank>/<hhh>/<hh>_g<group>/<hash>.bin, written via temp-file + atomic rename under O_DIRECT and self-healing by deleting unreadable files. Two upstream WEAKNESSES recorded as beyond-parity targets: config.json is written and NEVER read (the only identity check is a path digest omitting checkpoint content, weight quantization, rope config and sliding_window), and the disk tier has NO capacity accounting and NO eviction. Secondary tiers can never touch GPU memory — all traffic cascades through the CPU primary tier W1-W3 IMPLEMENTED 2026-07-22. Deterministic block hashes (W1), the CPU primary tier (W2: CachePolicy LRU+ARC with the ref_cnt == -1 tri-state and the ATOMIC evict, CPUOffloadingManager incl. the prepare_store -> nullopt skip path, pinned backing store plus side-queue event-polled device/host transfer), and the DISK tier (W3: one raw file per block, temp-file + atomic rename publish, self-healing unlink, dual-queue read/write pool). BOTH recorded upstream weaknesses are now EXCEEDED, not merely noted: the identity block is a VERIFIED header read on every open that REFUSES on mismatch across 27 fields (upstream's config.json is never read), and the tier carries a byte budget with policy-driven eviction honoured across restarts (upstream has none). O_DIRECT is deliberately NOT ported — a header+payload file breaks its alignment requirement; recorded. W4 IMPLEMENTED 2026-07-23. The TIERING MANAGER (ONE manager over the CPU primary + disk secondary tier: disk→CPU promotion is RETRY this step / HIT the next with the reserved slot marked in-flight, cascade demotion on store, reset drains the secondary FIRST and DELIBERATELY never resets it so a persisted cache survives a prefix-cache reset) and the CONNECTOR/SCHEDULER HALF (OffloadingConnector mirroring KVConnectorBase_V1's scheduler hooks — get_num_new_matched_tokens with the load-bearing NULLOPT third state, Request::block_hashes striding, load-before-compute ordering, build_connector_meta reset — wired OPT-IN and DEFAULT-OFF into the scheduler so a cross-request/restarted-process prefix HIT shortcuts prefill). The semantics are ported, NOT the Python plugin ABI (compile-time wiring replaces the importlib module path; the full 7-method abstract ABI + registration + KVTransferConfig is the W5 generalization behind the same seam). Deviation recorded: W4 ships the SYNCHRONOUS-load shape (async flag always false), the disk→CPU promotion being the async part handled by RETRY/re-ask; the cross-step WAITING_FOR_REMOTE_KVS GPU-load buffer is W5. First measured offload speedup: a restarted-prefix workload through the real scheduler saved 32/48 prefill tokens (2/3 blocks HIT from disk) with the promoted bytes proven byte-identical to the cold store. W5 LANDED 2026-07-23 (the connector seam is now a first-class C++ ABI — abstract KVConnector base + KVConnectorFactory + KVTransferConfig, the disk connector refactored onto it behaviour-identically; see the KV-CONNECTORS row). D1 CORRECTION 2026-07-24 (CLAIM-DOCS-T2-FIXES): the disk connector's WORKER HALF IS NOT IMPLEMENTED and is now REFUSED, not merely absent. OffloadingConnector emits ConnectorLoadJobs that NOTHING consumes, and its bytes live in a host PrimaryByteView that is never copied into a KV page — on any device. Because its scheduler half DOES shortcut prefill for matched blocks, wiring it into an engine would have made the model attend over never-written KV (silently wrong output); BuildKvConnector previously built it for any device with no guard. It is now refused at construction by a per-connector capability predicate (KVConnector::supports_worker_transfer_on / the registered KVConnectorWorkerTransferFn, queried by name BEFORE construction via KVConnectorFactory::WorkerTransferSupportedOn), with an error naming the connector, the device, the consequence and the admissible connectors. The scheduler-side 32/48 e2e is UNAFFECTED (it never reaches a worker). Implementing the worker half remains OPEN work and is NOT claimed. W6 (LMCache study) and W7 (named save/restore) remain open |
T2 | core vllm/v1/kv_offload/base.py:27-47,88-108,177-347,486-588,536-549; CPU tier vllm/v1/kv_offload/cpu/manager.py:36,169-237, policies cpu/policies/base.py:10-33,36-92, lru.py:12, arc.py:12; disk tier vllm/v1/kv_offload/tiering/fs/io.py:32-72,75-101, tiering/fs/manager.py:95-103,131-137, tiering/fs/thread_pool.py:50-57,153-180; naming/identity vllm/v1/kv_offload/file_mapper.py:112-120,128-139; tiering ordering tiering/manager.py:238-329,408-459,498-556,643-681; transfer cpu/gpu_worker.py:240-421,388-394; config docs/features/kv_offloading_usage.md:64-82,95-121; tests tests/v1/kv_offload/tiering/test_fs_tier.py, tests/v1/kv_offload/test_file_mapper.py, tests/v1/kv_offload/cpu/test_manager.py |
W1-W3 LANDED. Core include/vllm/v1/kv_offload/base.h (OffloadKey verified byte-identical to upstream's packing); policies include/vllm/v1/kv_offload/cache_policy.h + src/vllm/v1/kv_offload/cache_policy.cpp; CPU tier include/vllm/v1/kv_offload/cpu_manager.h + src/vllm/v1/kv_offload/cpu_manager.cpp; transfer include/vllm/v1/kv_offload/kv_block_transfer.h + src/vllm/v1/kv_offload/kv_block_transfer.cpp (plus the new non-blocking vt::Backend::QueryEvent seam with its CUDA override in src/vt/cuda/cuda_backend.cu); disk byte path + naming include/vllm/v1/kv_offload/fs_io.h + src/vllm/v1/kv_offload/fs_io.cpp; tier include/vllm/v1/kv_offload/fs_tier.h + src/vllm/v1/kv_offload/fs_tier.cpp; the verified identity header include/vllm/v1/kv_offload/cache_identity.h + src/vllm/v1/kv_offload/cache_identity.cpp; determinism fix src/vllm/v1/core/kv_cache_utils.cpp (init_none_hash seed resolution + none_hash_provenance), caller src/vllm/entrypoints/model_loader.cpp:140-152; W4 tiering manager include/vllm/v1/kv_offload/tiering_manager.h + src/vllm/v1/kv_offload/tiering_manager.cpp; connector/scheduler half include/vllm/v1/kv_offload/kv_connector.h + src/vllm/v1/kv_offload/kv_connector.cpp; scheduler wiring src/vllm/v1/core/sched/scheduler.cpp (set_kv_connector, null = zero change) + include/vllm/v1/core/sched/scheduler.h; BlockPool::evict_blocks src/vllm/v1/core/block_pool.cpp:139-155 (1:1, replaces the throw) |
tests/vllm/v1/test_none_hash_determinism.cpp 7/7 (cross-PROCESS byte-identical hash chains via a /proc/self/exe re-exec, both env escape hatches, and the =random negative control); tests/vllm/v1/test_kv_offload_cpu.cpp 21/21 (atomic evict, pinning, ARC promotion, HIT_PENDING, failed-store rollback, same-batch protection, store_threshold, events, transfer round-trip); tests/vllm/v1/test_kv_offload_fs.cpp 22/22 + 3 SKIP (byte-exact round trip for full attention AND MLA rank-3, truncation/foreign-magic/misfiled refusal with self-heal, a 27-field identity-refusal matrix with a positive control, the byte budget across a restart, and a 6/6 cross-restart hit measurement); the SKIPs are row-tagged to KV-SLIDING-WINDOW-SPEC, KV-FP8/KV-NVFP4-TURBO and KV-MAMBA-ALIGN; W4 tests/vllm/v1/test_kv_offload_tiering.cpp 5/5 (promotion RETRY→HIT byte-identical, CPU-eviction→disk-survival→re-promotion, reset clears CPU but disk survives, a FRESH manager on the same directory promotes = restart, and identity REFUSAL through a promotion — a corrupt disk block is unlinked and treated as absent, never trusted) and tests/vllm/v1/test_kv_offload_connector.cpp 4/4 (null-connector inertness, external match shortcuts prefill by exactly ext, the nullopt third state defers then schedules next step, and the END-TO-END restarted-prefix disk HIT through the real scheduler: hit rate 2/3 blocks, 32/48 prefill tokens saved, promoted bytes byte-identical) |
kv-persistence-lmcache.md | ACTIVE |
CLAIM-KV-PERSISTENCE-LMCACHE |
KV-EXTERNAL-CACHE |
External KV-cache provider ABI plus LMCache interoperability: producer/consumer/both roles, the scheduler/worker metadata split, cache registration, block-hash lookup, asynchronous load/store and completion/free ownership. SPIKED 2026-07-22 (spike) — the ABI is smaller than the row implied and the LMCache half is larger. The minimum viable connector is exactly 7 abstract methods (worker start_load_kv/wait_for_layer_load/save_kv_layer/wait_for_save, scheduler get_num_new_matched_tokens/update_state_after_alloc/build_connector_meta); roughly thirty further hooks all have safe defaults. Three traps recorded: get_num_new_matched_tokens has a THIRD state (None = deschedule and re-ask, not zero), request_finished returning True transfers block-freeing OWNERSHIP to the connector, and non-HMA connectors ASSERT a single KV cache group while our gate models are two-group hybrids. LMCache determination: it is an EXTERNAL PyPI package (lmcache >= 0.3.9 in an opt-in extras file that setup.py/pyproject.toml never reference; not installed on any of this project's boxes). vLLM vendors roughly 2396 lines of lmcache_integration/ glue, but every one of those files imports the external package at module scope — the storage engine, the paged-memory GPU connectors, the config schema, the ZMQ message queue and the CUDA-IPC handoff are all outside the tree, and no upstream test exercises it without importing lmcache. Scoped as an interop STUDY, not a from-scratch client, and gated on two blockers we own: our sha256_cbor hashes are not byte-compatible with vLLM's default, and our NONE_HASH is per-process random. REOPENED 2026-07-23 (client spike) on the user's connect-as-client hypothesis, and the prior "no specified wire protocol" verdict is REFUTED by reading the LMCache package (LMCache/LMCache@8570aad). vLLM connects to a RUNNING LMCache instance over two fully-specified, language-agnostic wires: (1) the lm:// remote-store server — plain TCP + a fixed struct.pack header + raw KV bytes, no ZMQ/msgpack/pickle/CUDA-IPC (lmcache/v1/protocol.py:214-321, server/__main__.py:24-147, lm_connector.py:28-177); and (2) the MP server — ZMQ DEALER↔ROUTER + msgspec.msgpack control + CUDA-IPC data (multiprocess/mq.py:270-353, custom_types.py:120-234), the mode the user recalled as "zmq". BOTH need ZERO lmcache in our process and BOTH sidestep the R1 hash blocker — LMCache keys on its OWN blake3 rolling token hash (token_hasher.py:54-79), never vLLM block hashes. Pickle appears ONLY in the MP one-time IPC-wrapper registration (platform/base/ipc_wrapper.py Serialize); CUDA-IPC ONLY in MP data (portable via RawCudaIPCWrapper cudaIpcGetMemHandle, but co-located). Verdict: a C++ client is FEASIBLE — recommend MODE (1) first (stabler/simpler); the standing risk is LMCache being an unpinned moving target, so it is an interop feature with a version-sync cost, not a mechanical core port |
T2 | ABI vllm/distributed/kv_transfer/kv_connector/v1/base.py:171,293,311,325,347,454,489,510,542,585; roles :124; HMA :85,93; factory + out-of-tree module seam vllm/distributed/kv_transfer/kv_connector/factory.py:28,31,96,102-123,152-238; config vllm/config/kv_transfer.py:22-75,102-106; MRV2 worker hooks vllm/v1/worker/gpu/kv_connector.py:56,61-75,77-95; scheduler call sites vllm/v1/core/sched/scheduler.py:280,736-742,933-937,1118-1119,2340-2371; LMCache vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py:74-115,259,281, lmcache_mp_connector.py:1-50, lmcache_integration/vllm_v1_adapter.py:11-35,175-188,368-376,781, external requirement requirements/kv_connectors.txt:1; tests tests/v1/kv_connector/unit/test_lmcache_integration.py:60-223, test_kv_connector_lifecycle.py:37, test_config.py:51 |
W1 LANDED 2026-07-23 — the LMCache MODE-1 lm:// wire CODEC (pure CPU, INERT: no call site routes to it, the connector is W3): src/vllm/v1/kv_offload/lmcache/remote_protocol.{h,cpp} (186-byte ClientMetaMessage / 36-byte ServerMetaMessage fixed-struct framing + ClientCommand/ServerReturnCode/DTYPE_TO_INT/Location maps), cache_engine_key.{h,cpp} (model@world@worker@chunk_hash_hex@dtype to/from string), token_hasher.{h,cpp} (blake3 rolling chunk hash over vendored third_party/blake3/ 1.5.5), memory_format.{h,cpp} (the KV_2LTD [2,L,T,D] repack); wired in CMakeLists.txt (blake3_vendored static lib). Later-connector seams still NAMED: include/vllm/v1/core/kv_cache_manager.h:31 (ext_comp), include/vllm/v1/core/single_type_kv_cache_manager.h:122, include/vllm/v1/core/sched/output.h:30-31, include/vllm/v1/engine/types.h:26,30. W5 worker-side store/load LANDED 2026-07-24 (the last open arm): src/vllm/v1/worker/gpu/runner.cpp (ConnectorLoadExternalKv writes the external-prefix KV into the allocated GPU blocks BEFORE the forward = load-before-compute; ConnectorStorePromptKv stores each newly-complete prompt block AFTER the forward; both behind a kv_connector_ != nullptr guard so default-off is byte-identical) + include/vllm/v1/worker/gpu/runner.h (set_kv_connector), src/vllm/entrypoints/model_loader.cpp (BuildKvConnector builds the connector from EngineParams::kv_transfer_config via KVConnectorFactory, injects the runner's full-attention KV geometry, wires it to scheduler + runner) + include/vllm/entrypoints/model_loader.h (EngineParams::kv_transfer_config, LoadedEngine::kv_connector()) |
W1 byte/bit-exact gate GREEN (CPU): tests/vllm/v1/kv_offload/lmcache/test_lmcache_codec.cpp (6 cases / 2074 assertions) vs tests/fixtures/lmcache/lmcache_fixtures.json — our wire bytes == the real Python codec's (stdlib struct framing + blake3 PyPI hashes + numpy KV_2LTD); blake3 digest VERIFIED byte-identical on x86-64 AND dgx.casa aarch64. W2 (client, CPU) GREEN — go/no-go PASSED: src/vllm/v1/kv_offload/lmcache/remote_client.{h,cpp} (blocking POSIX-socket PUT/GET/EXIST/HEALTH/LIST + partial-read/write loops + PutKv2ltd/GetKv2ltd KV_2LTD repack + LmcacheClientConfig/VT_LMCACHE_* env); tests/vllm/v1/kv_offload/lmcache/test_lmcache_client.cpp round-trips a REAL lmcache.v1.server (8570aad, run headless from source in a throwaway venv — torch imported before lmcache to dodge a torch circular import, the compiled c_ops ext stubbed as unused by the lm:// CPU store) byte-identical (36/36), and interop is BIDIRECTIONAL with LMCache's OWN Python protocol codec (scripts/lmcache/{lm_server,lm_interop_client}.py+run_live_roundtrip.sh); always-on CI gate = a same-binary C++ mock-server round-trip (45/45, no Python). W3 LANDED 2026-07-23 — the lm:// client wired as a KVConnector over the W5 seam (the FIRST time engine -> connector -> W2 client -> a running lm:// server -> back runs): src/vllm/v1/kv_offload/lmcache/lmcache_connector.{h,cpp} (LMCacheConnector : KVConnector, REGISTER_KV_CONNECTOR("LMCacheConnector", …), selected by KVTransferConfig{kv_connector="LMCacheConnector", kv_connector_extra_config={host,port,hash_algo,chunk_tokens,…}}, default OFF). Scheduler side is real: get_num_new_matched_tokens computes the request's rolling-blake3 chunk hashes, builds the CacheEngineKey per chunk and Exist-probes the REMOTE store for the longest cached prefix (synchronous -> (n, false), mirroring lmcache_connector.py:230-259); update_state_after_alloc records the load (drops blocks upstream, :261-268); worker StoreChunk (PUT KV_2LTD) / LoadChunk (GET+unpack, foreign-block REFUSAL via GetKv2ltd). Gate ACHIEVED = the connector-level round-trip: store -> lookup -> prefill-shortcut through the REAL scheduler -> load byte-identical (32/48 prefill tokens saved), foreign/mismatched-key REFUSAL, default-off inertness (tests/vllm/v1/kv_offload/lmcache/test_lmcache_connector.cpp 5 cases / 50 assertions vs an in-process mock; the store->load round-trip ALSO passes vs a REAL lmcache.v1.server 8570aad, 16 assertions, under VT_LMCACHE_LIVE_*). W4 LANDED 2026-07-23 — REAL peer KEY-AGREEMENT + a peer->us interop LOAD, both PROVEN (the interop-correctness milestone is complete; the row stays ACTIVE only for the DGX full-model output-invariance + throughput arm, spec gates 4/6): the actual lm:// key derivation is NOT the blake3 MP TokenHasher (a different subsystem) but ChunkedTokenDatabase (lmcache/v1/token_database.py:298-449) — chunk_size 256, a rolling prefix-hash chain over the 3-tuple (prefix_int, tuple(tokens), extra_keys=()), keyed by vLLM's OWN hash function (pre_caching_hash_algorithm; the portable interop choice sha256_cbor = cbor2-canonical + SHA-256, vllm/utils/hashing.py:43), folded to uint64 each step (_normalize_hash_to_int token_database.py:34-56), with NONE_HASH = fold8(sha256_cbor(str(PYTHONHASHSEED))) (kv_cache_utils.py:99-114). Mirrored BYTE-EXACT in src/vllm/v1/kv_offload/lmcache/chunked_token_database.{h,cpp} (reusing the project's CborValue+sha256_cbor, already Python-cbor2/hashlib-exact), and wired into the connector as key_mode=kVllmSha256Cbor (hash_algo="vllm"/"sha256_cbor", chunk 256) alongside W3's kept-green blake3 path. Key-agreement gate GREEN: tests/vllm/v1/kv_offload/lmcache/test_lmcache_key_agreement.cpp (4 cases / 85 assertions) asserts our CacheEngineKey strings + chunk boundaries + folded hashes are BYTE-IDENTICAL to the REAL lmcache ChunkedTokenDatabase.process_tokens() (fixtures tests/fixtures/lmcache/key_agreement_fixtures.json dumped by scripts/lmcache/gen_key_agreement_fixtures.py driving the unmodified real driver, with vLLM's pinned sha256_cbor/init_none_hash), incl. the connector's own peer-mode ChunkKey. Sample: tokens 1000..1511 -> meta-llama/Llama-3.1-8B@1@0@33d6862800fff40c@bfloat16. Peer->us interop LOAD gate GREEN (over the wire, real server): scripts/lmcache/{lm_key_interop.py,run_key_interop.sh} has the REAL lmcache ChunkedTokenDatabase derive a key from tokens and PUT KV to a REAL lmcache.v1.server (8570aad, headless); our C++ INDEPENDENTLY re-derives the SAME key and GETs the peer-written 512 B byte-identical (test_lmcache_key_agreement LIVE case under VT_LMCACHE_LIVE_SPEC). ASan+UBSan clean on the connector path. Text-only scope (mm-hash extra_keys deferred); the DGX full-model output-invariance + throughput are the W5 arm below. W5 OUTPUT-INVARIANCE GATE GREEN 2026-07-24 (spec gates 4+6 met — the LAST open arm CLOSED): tests/vllm/models/test_lmcache_output_invariance.cpp on a REAL OPT-125m bf16 loop vs a live lmcache.v1.server (8570aad, headless per the W2 recipe) proves connector-ON generated tokens are BIT-IDENTICAL to connector-OFF cold full prefill (first-divergence index -1) in BOTH modes — (a) store->restart->load within one process AND (b) a genuinely COLD second process that only hits the server (VT_LMCACHE_OI_MODE=loadonly) — with prefill SAVED on the hit = 48 tokens (3×16-token blocks) and chunks_stored>0; driven by scripts/lmcache/run_output_invariance.sh under flock $HOME/gpu.lock, VT_ASYNC_SCHED=0. Throughput reported HONESTLY: on a 125M model wall-clock is noise-dominated (fixed TCP/copy overhead ~ tiny compute saved) so NO binding speedup is claimed — a real speed number is owed by an every-axis grid on a larger model + long shared-prefix corpus (docs/BENCHMARKS.md). No-regression WITNESS: OPT SACRED gate UNCHANGED default-off (test_opt_paged_engine 6/6 prompts, 96/96 tokens, 63/63 assertions) with the connector code present; connector units green (codec 6/6·2074, client 3/3·45, connector 5/5·50, key-agreement 4/4·85, kv_offload_connector 11/11·80); ASan+UBSan clean on the connector path (0 sanitizer hits); CUDA -Werror 0 warnings. Additive + default-off inert (scheduler/worker/seam untouched) |
kv-persistence-lmcache.md; LMCache client wire analysis + W-plan lmcache-cpp-client-connector.md | ACTIVE (W1-W5 landed; the connector-ON full-model OUTPUT-INVARIANCE arm is CLOSED — connector-ON == connector-OFF tokens BIT-IDENTICAL on a real OPT-125m loop vs a live lmcache.v1.server, both after an in-process restart and from a cold second process, spec gates 4/6 met; a BINDING every-axis LMCache throughput grid on a LARGER model stays PENDING, mirroring the Llama 'correctness DONE, speed PENDING' disposition — a 125M model's wall time is noise-dominated) |
CLAIM-LMCACHE-CPP-CLIENT (W1 codec + W2 client + W3 connector + W4 key-agreement + W5 output-invariance); parent seam CLAIM-KV-PERSISTENCE-LMCACHE |
KV-CONNECTORS |
Remaining connector breadth and prefill/decode disaggregation over the KV-EXTERNAL-CACHE base seam. SPIKED 2026-07-22 (spike): the registry is enumerated at SIXTEEN connectors and the breadth is DISPOSITIONED, not merely listed. Registered: ExampleConnector (the former SharedStorageConnector — RENAMED at this pin, so the old name in the record is stale), ExampleHiddenStatesConnector, LMCacheConnectorV1, LMCacheMPConnector, NixlConnector/NixlPullConnector/NixlPushConnector, MultiConnector, MoRIIOConnector, OffloadingConnector, DecodeBenchConnector, MooncakeConnector, MooncakeStoreConnector, FlexKVConnectorV1, SimpleCPUOffloadConnector, HF3FSKVConnector. P2pNcclConnector NO LONGER EXISTS — no file, no registry entry, no reference. NIXL / Mooncake / MoRI-IO / HF3FS / FlexKV and P/D disaggregation are NOT SCHEDULED: each needs an external RDMA or store dependency absent from our boxes and ungateable on GB10. What this row DOES own after the spike is the generalization of the first ported connector into a reusable C++ seam — abstract base, compile-time registration replacing vLLM's dynamic Python module-path import (recorded deviation), KVTransferConfig, the fail-by-default load-failure policy, deferred block-free ownership, and SupportsHMA multi-group finish. W5 LANDED 2026-07-23: the abstract KVConnector base (the full scheduler + worker method set mirroring KVConnectorBase_V1 — the scheduler methods get_num_new_matched_tokens/update_state_after_alloc/build_connector_meta/request_finished are load-bearing for correctness, the worker hooks register_kv_caches/start_load_kv/wait_for_layer_load/save_kv_layer/wait_for_save/get_finished are defaulted no-ops for our synchronous runner, documented in the header), a compile-time KVConnectorFactory + REGISTER_KV_CONNECTOR macro (the C++ analogue of the Python importlib module path, mirroring REGISTER_VLLM_MODEL), the KVTransferConfig selection surface (default kv_connector empty == no connector == zero behaviour change) with kv_role validation and the fail-default load policy, and SupportsHMA multi-group finish + deferred-free ownership on the base. The W4 disk connector now IMPLEMENTS this abstract base behaviour-identically (the restart-hit e2e reproduces byte-for-byte, and a config-selected owning connector shortcuts prefill by the exact same 32/48). This closes the seam so LMCache W3 is 'implement the abstract KVConnector with the landed W2 lm:// client' — no further seam change. NIXL / Mooncake / MoRI-IO / HF3FS / FlexKV and P/D disaggregation remain NOT SCHEDULED |
T2 | registry vllm/distributed/kv_transfer/kv_connector/factory.py:152-238; renamed disk reference impl vllm/distributed/kv_transfer/kv_connector/v1/example_connector.py:104,109,203,405-442; MultiConnector HMA gate factory.py:145; SimpleCPUOffloadConnector vllm/v1/simple_kv_offload/worker.py:198,204,269; load-failure policy vllm/config/kv_transfer.py:70-74 with vllm/v1/core/sched/scheduler.py:1529-1535; tests tests/v1/kv_connector/unit/test_multi_connector.py:818, tests/v1/kv_connector/unit/test_kv_load_failure_recovery.py, tests/v1/kv_connector/unit/test_remote_prefill_lifecycle.py, tests/v1/kv_connector/unit/test_nixl_connector.py |
W5 abstract ABI + factory + config LANDED. The abstract base + factory + registration include/vllm/v1/kv_offload/kv_connector.h + src/vllm/v1/kv_offload/kv_connector.cpp (the KVConnector base with the full method set, KVConnectorFactory/KVConnectorRegistrar/REGISTER_KV_CONNECTOR, the OffloadingConnector now deriving from it plus its CreateFromConfig owning builder registered as "OffloadingConnector"); the selection surface include/vllm/config/kv_transfer.h + src/vllm/config/kv_transfer.cpp (KVTransferConfig, KVRole, KVLoadFailurePolicy, Validate/predicates/string round-trips); scheduler holds the renamed KVConnector* (non-owning, null = zero change) include/vllm/v1/core/sched/scheduler.h + src/vllm/v1/core/sched/scheduler.cpp; wired in CMakeLists.txt. Still the throw stub for evict_blocks at src/vllm/v1/core/block_pool.cpp:139-155 (owned jointly with KV-BLOCK-POOL, replaced in W4) |
tests/vllm/v1/test_kv_offload_connector.cpp 11/11 (80 assertions): the pre-existing null-connector inertness + external-match shortcut + nullopt-third-state + restart-hit e2e, PLUS W5 — KVTransferConfig validation/predicates (kv_role required when kv_connector set, producer/consumer predicates, policy-default fail, string round-trips), KVConnectorFactory selection (absent config == nullptr, unknown name + duplicate registration throw, disk connector registered + selectable + reports HMA, missing root_dir refused), an interface-completeness oracle overriding EVERY base method (mirrors test_multi_connector_overrides_all_base_methods) with the worker-hook defaults exercised, and a behaviour-identical e2e where the config-selected OWNING connector shortcuts prefill by the exact same 32/48 as the borrowing path. 2026-07-24 (CLAIM-DOCS-T2-FIXES) — WORKER-HALF CAPABILITY on the seam + a USER-FACING selector. The base gained supports_worker_transfer_on(vt::DeviceType) (default FALSE: the base's worker hooks are no-ops, so a connector that does not override it has no worker half); the registry gained a parallel static KVConnectorWorkerTransferFn (REGISTER_KV_CONNECTOR_WITH_WORKER) so an unsafe (connector, device) pair is refused BY NAME before anything is constructed; EnsureWorkerTransferSupported is the single refusal point the engine calls. LMCacheConnector registers the predicate (true on every device — its worker half is vt::Backend::Copy-based and device-agnostic); OffloadingConnector does not, so it is refused everywhere. Selection is now reachable from the server: --kv-transfer-config '<json>' (vllm::ParseKVTransferConfigJson, vLLM's own flag name and JSON shape) threaded into EngineParams::kv_transfer_config; absent == no connector == byte-identical. New cases in tests/vllm/v1/test_kv_offload_connector.cpp (18/18, 144 assertions): the base default, a per-device partial-implementation stand-in, the disk refusal + message content on cpu/cuda/metal, the LMCache admission, empty-name inertness, unknown-name reporting, and the JSON parse/round-trip/malformed matrix |
kv-persistence-lmcache.md | ACTIVE |
CLAIM-KV-PERSISTENCE-LMCACHE; worker-half guard + CLI selector by CLAIM-DOCS-T2-FIXES |
KV-EVENTS |
Block create/evict event publication (BlockStored / BlockRemoved / AllBlocksCleared, ZMQ publisher, enable_kv_cache_events default off). W1 DONE 2026-07-27 (CLAIM-ROADMAP-D4-KV-EVENTS) — event GENERATION + PAYLOAD ported and gated. The event data types (kv_events.py:25-121) + the msgpack payload encoder (byte-exact vs msgspec.msgpack.Encoder(), verified against msgspec 0.21.1 on the 1:1 upstream structs, BOTH the default int-truncated hash form and the raw-bytes form) + the emission at the BlockPool store/remove/clear sites are landed, guarded by enable_kv_cache_events (default OFF ⇒ default path byte-identical) + a NullEventPublisher/CollectingEventPublisher seam faithful to --kv-events-config. DEFERRED (honest residual): the LIVE ZMQ transport (PUB/ROUTER sockets, replay buffer, publisher thread, DP port offset) is stubbed behind the seam (EventPublisherFactory throws loudly on "zmq"); the engine/scheduler wiring of the batch envelope + the report_mode=="full" reuse path (Request has no kv_cache_report_mode) are not wired. Optional int-truncated block hashes (VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES, upstream DEFAULT True) narrow the digest to its low 64 bits — ported. SPIKE context (2026-07-22): the scaffolding pre-existed (placeholder KVCacheEvent, always-empty take_events, marked-out emission no-ops), so this was a bounded fill-in — KVCacheEvent is an empty placeholder struct, take_events() returns an always-empty queue, and the three emission points inside cache_full_blocks / _remove_cached_block_hashes / reset_prefix_cache are marked-out no-ops, so this is a bounded fill-in rather than a fresh port. Optional int-truncated block hashes (VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES) narrow the digest to its low 64 bits |
T2 | vllm/config/kv_events.py:11-52; event classes vllm/distributed/kv_events.py:48-108,278,505; emission vllm/v1/core/block_pool.py:340-356; vllm/v1/core/kv_cache_manager.py:121,149; scheduler wiring vllm/v1/core/sched/scheduler.py:154,1791,1802-1805; hash truncation vllm/v1/core/kv_cache_utils.py:79-82; tests tests/v1/core/test_prefix_caching.py:2040,2170,2228,2282,2368,2405 |
types + encoder + publisher include/vllm/distributed/kv_events.h + src/vllm/distributed/kv_events.cpp; ExternalBlockHash/maybe_convert_block_hash include/vllm/v1/core/kv_cache_utils.h + src/vllm/v1/core/kv_cache_utils.cpp; emission src/vllm/v1/core/block_pool.cpp (cache_full_blocks store, _emit_block_removed_events/_maybe_evict_cached_block remove, reset_prefix_cache clear, emit_cached_block_events, _build_block_stored_event); KVCacheEvent alias include/vllm/v1/core/block_pool.h |
tests/vllm/v1/test_kv_events.cpp (6 cases / 62 assertions: byte-exact msgpack ×2 + store/reuse/evict/reset SEQUENCE + RED-first); default-off + APC unchanged tests/vllm/v1/test_block_pool.cpp 132/132, tests/vllm/v1/test_prefix_cache_stats.cpp 36/36 |
kv-events.md; prefix-prompt-caching-parity.md | ACTIVE |
CLAIM-ROADMAP-D4-KV-EVENTS |
KV-MLA-SPEC |
Latent MLA KV specification | T2 | vllm/v1/kv_cache_interface.py:363 |
- | - | planned: specs/mla-kv-spec.md |
INVENTORIED |
- |
KV-CROSS-ENCODER-SPECS |
CrossAttentionSpec and EncoderOnlyAttentionSpec KV interface specs (ATTN-ENCODER-CROSS covers backends only); carried from porting-inventory §2 (T2) at the v1 fold |
T2 | vllm/v1/kv_cache_interface.py:710,717 |
- | - | planned: specs/encoder-cross-kv-specs.md |
INVENTORIED |
- |
KV-SIZING |
GPU memory utilization and block-count overrides | T0 | vllm/config/cache.py:68,87,168; tests/v1/core/test_kv_cache_utils.py:2224,2303 |
fixed inputs src/vllm/entrypoints/model_loader.cpp:117,129; watermark src/vllm/v1/core/kv_cache_manager.cpp:118 |
watermark only tests/vllm/v1/test_kv_cache_manager.cpp:298 |
planned: specs/kv-sizing.md |
PARTIAL |
- |
KV-WARMUP-PROFILE |
Dummy runs, warmup, and startup memory profiling that derive the KV budget (KV-SIZING covers the sizing knobs only); carried from porting-inventory §3 (T0 there) at the v1 fold |
T0 | vllm/v1/worker/gpu/model_runner.py:504,647; vllm/v1/worker/gpu_worker.py:430 |
- | - | planned: specs/warmup-memory-profiling.md |
INVENTORIED |
- |
ENG-EXPERT-STREAM |
Expert streaming from disk: bank-only routed-MoE weights paged into fixed contiguous Marlin slots after logical-expert→slot remap (low-concurrency capacity mode; surpass-track — inference-time disk expert paging is ABSENT in pinned vLLM) | T2 | absent in-pin: vllm/model_executor/offloader/uva.py:21 (CPU-blanket UVA only), vllm/model_executor/offloader/prefetch.py:557-560 (cpu-only); design reference antirez/ds4 (ds4_metal.m, ds4_cuda.cu, ds4_ssd.c); local dense-stride constraint src/vt/cuda/marlin/libtorch_stable/moe/marlin_moe_wna16/marlin_template.h:543-550 |
- | - | expert-streaming.md | READY |
- |
ENG-WEIGHT-OFFLOAD |
Inference-time CPU weight offload mirror floor: UVA cpu_offload_gb per-parameter offload with pinned+zero-copy views and opt-in name-segment targeting (cpu_offload_params), plus layer-group PrefetchOffloader; v1-supported at the pin |
T2 | vllm/config/offload.py:23,34-44,47-76; vllm/model_executor/offloader/uva.py:64,80-108; vllm/model_executor/offloader/base.py:126-162; vllm/v1/worker/gpu_model_runner.py:445,913; tests tests/basic_correctness/test_cpu_offload.py:11, tests/quantization/test_cpu_offload.py:18-64 |
- | - | planned: specs/weight-offload-uva.md |
INVENTORIED |
- |
KV-SGLANG-RADIX-CACHE |
SGLang RadixAttention behavior-parity scope (fuse-or-flag). VERDICT: already FUSED into our APC. SGLang's radix TREE of KV prefixes (token/page-granular trie, longest-prefix match with mid-node split, LRU-over-tree eviction, lock_ref protection, extra_key tenant isolation) is functionally equivalent to our block-hash APC (per-block chain hash, block pool, block-LRU, ref-count, extra_keys). The ONLY behavioral delta is sharing granularity (token/page vs block_size), bounded to one block at the divergence point, already parametrized by our block_size, and output-neutral (APC-ON==APC-OFF token-exact, KV-PREFIX-CACHE W3). A second token-granular trie would be a redundant incompatible abstraction against MIRROR-vLLM ⇒ NO distinct path. The flag --enable-radix-attention is an ALIAS for the APC toggle (enable_prefix_caching); implementation = CLI alias + a C-ABI enable_prefix_caching tri-state field (currently absent from vllm_model_params). SGLang pin v0.5.15 f63458b. |
T2 | SGLang mem_cache/radix_cache.py:280 (RadixCache), :217 (TreeNode), :355/:648 (match_prefix/helper), :563 (evict LRU heap), :592/:607 (lock_ref), :136/:191 (page-aligned match); mem_cache/registry.py:83-94 (ChunkCache fallback); server_args.py:755 (--disable-radix-cache), :739 (eviction policy) |
our APC (== the fused equivalent): src/vllm/v1/core/kv_cache_utils.cpp:259,291; src/vllm/v1/core/kv_cache_coordinator.cpp:260; src/vllm/v1/core/kv_cache_manager.cpp:124; toggle examples/server/main.cpp:185, include/vllm/entrypoints/model_loader.h:76; alias landed (RW1, CLAIM-SGLANG-IMPL): --enable-radix-attention/--disable-radix-attention server aliases for the same enable_prefix_caching tri-state examples/server/main.cpp; C-ABI int32_t vllm_model_params.enable_prefix_caching (0/1/2) ABI v6→v7 include/vllm.h, mapped in src/capi/vllm_c.cpp |
KV-PREFIX-CACHE cache-ON proof reused (tests/parity/test_qwen3_apc_e2e.cpp, hits 0.807, token-exact); ABI round-trip tests/capi/test_capi.cpp "enable_prefix_caching tri-state" 9/9 (default 0, valid 0/1/2, out-of-range→INVALID_ARGUMENT); alias adds server-help + C-ABI contract only, NO engine change (RadixAttention is fused into APC) |
sglang-radixattention.md | ACTIVE |
CLAIM-SGLANG-IMPL |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
PAR-TP |
Tensor parallelism | T2 | vllm/distributed/parallel_state.py:1695-1783; vllm/config/parallel.py:117 |
- | - | tensor-parallelism.md | READY |
- |
PAR-PP |
Pipeline parallelism | T2 | vllm/config/parallel.py:120 |
- | - | planned: specs/pipeline-parallel.md |
INVENTORIED |
- |
PAR-EP-EPLB |
Expert parallelism and EPLB | T2 | vllm/config/parallel.py:162,481-487; vllm/v1/worker/gpu/eplb_utils.py:1 |
- | - | planned: specs/expert-parallel.md |
INVENTORIED |
- |
PAR-DP |
Data parallelism | T2 | vllm/config/parallel.py:126-155 |
- | - | planned: specs/data-parallel.md |
INVENTORIED |
- |
PAR-SEQUENCE-MOE |
MoE sequence parallelism without requiring data parallelism, including TP/EP collectives, token padding and CUDA-graph sizing; v0.25.0 reports a 1.9–5.0% end-to-end throughput gain | T2 | vllm/config/parallel.py:642; vllm/forward_context.py:62-108; vllm/distributed/parallel_state.py:1224-1266; vllm/v1/worker/gpu_model_runner.py:3430,3879; tests/kernels/moe/test_moe_layer.py:1359-1509 @ 702f481 |
- | - | planned: specs/sequence-parallel-moe.md |
INVENTORIED |
- |
PAR-MULTINODE |
Multi-node Ray and multiprocessing executor behavior | T3 | vllm/v1/executor/abstract.py:37; vllm/config/parallel.py:245 |
- | - | planned: specs/multi-node.md |
INVENTORIED |
- |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
SAMPLE-CORE |
Ordered temperature, top-k/p, min-p, penalties, seed, stop, length, output-kind pipeline | T0 | vllm/v1/sample/sampler.py:20,72,243; vllm/sampling_params.py:264,500; vllm/v1/worker/gpu_input_batch.py:889-963; tests/v1/sample/test_sampling_params_e2e.py:17,25,40,176 |
src/vllm/sampling_params.cpp:25,35,167; src/vllm/v1/sample/sampler.cpp:152,215; src/vllm/v1/worker/gpu/input_batch.cpp:255,344,451; src/vllm/v1/core/sched/utils.cpp:12 |
tests/vllm/test_sampling_params.cpp:13,64,231,265; tests/vllm/v1/sample/test_sampler.cpp:46,78,118,142,165,253; tests/vllm/v1/worker/test_input_batch.cpp (C7 wiring: min_p/min_tokens/logprobs-count reach SamplingMetadata + condense/swap); tests/vllm/v1/test_input_processor.cpp (all_stop_token_ids) |
sampling-controls-c7.md (SAMPLE-CORE) |
ACTIVE |
CLAIM-ROADMAP-C7 |
SAMPLE-PHILOX |
Torch-Philox bit-exact stochastic parity | T1 | vllm/v1/sample/ops/topk_topp_sampler.py:70; vllm/v1/sample/sampler.py:243 |
- | - | planned: specs/philox-rng-parity.md |
INVENTORIED |
- |
SAMPLE-LOGPROBS |
Token logprobs payload end to end | T1 | vllm/logprobs.py:12,157,175; vllm/v1/engine/logprobs.py:29,69,348; vllm/v1/outputs.py:28,38; vllm/v1/core/sched/scheduler.py:1815-1836; vllm/entrypoints/openai/completion/serving.py:652; vllm/entrypoints/openai/chat_completion/serving.py:1114,1141; tests/v1/sample/test_logprobs.py:303 |
include/vllm/logprobs.h (Logprob/LogprobsOnePosition/SampleLogprobs + AppendLogprobsForNextPosition); include/vllm/v1/engine/logprobs.h+src/vllm/v1/engine/logprobs.cpp (LogprobsProcessor); src/vllm/v1/outputs.cpp:24 (LogprobsTensors::slice_request); src/vllm/v1/worker/gpu/runner.cpp:1272 (ModelRunnerOutput.logprobs); src/vllm/v1/core/sched/scheduler.cpp:674 (slice -> EngineCoreOutput.new_logprobs); src/vllm/v1/engine/output_processor.cpp (LogprobsProcessor integration -> CompletionOutput.logprobs); src/vllm/entrypoints/openai/serving_utils.cpp (BuildCompletionLogProbs/BuildChatLogprobs); src/vllm/entrypoints/openai/protocol.cpp (CompletionLogProbs/ChatCompletionLogProbs to_json); src/vllm/entrypoints/openai/serving_{completion,chat}.cpp |
tests/vllm/entrypoints/openai/test_logprobs.cpp:67,103 (serialization vs vLLM oracle, RED-first N vs N+1; LogprobsProcessor accumulation + inertness :137,167); tests/vllm/entrypoints/openai/test_serving.cpp:411,668 (e2e through the CPU engine: logprobs=K + chat top_logprobs, inert-when-off :467). Closing record parity-ledger.md#L741 2026-07-27 W5 row |
sampling-controls-c7.md (W5) | DONE |
ecda3ce1 |
SAMPLE-PROMPT-LOGPROBS |
Prompt logprobs (payload plumbing + serialization done; runner prompt-position logits SOURCE pending) | T1 | vllm/v1/engine/logprobs.py:121; vllm/v1/worker/gpu_model_runner.py:3842 (_get_prompt_logprobs_dict); vllm/sampling_params.py:303 |
payload path landed: src/vllm/v1/engine/logprobs.cpp:75,100 (UpdatePromptLogprobs/pop_prompt_logprobs); include/vllm/v1/engine/types.h:129,161 (ModelRunnerOutput.prompt_logprobs_dict, EngineCoreOutput.new_prompt_logprobs_tensors); src/vllm/v1/core/sched/scheduler.cpp:688 (prompt_logprobs_dict slice); src/vllm/v1/engine/output_processor.cpp:224 (RequestOutput.prompt_logprobs). RESIDUAL: the runner does not yet compute prompt-position logits (lm_head over prompt tokens) — the tensor SOURCE; a runner/prefill addition adjacent to C5 |
tests/vllm/entrypoints/openai/test_logprobs.cpp:137,167 (LogprobsProcessor accumulation/inertness cover the shared consume path; the prompt-only source is inert until the runner lands) |
planned: specs/prompt-logprobs.md |
PARTIAL |
- |
SAMPLE-LOGPROB-TOKEN-IDS |
logprob_token_ids generative scoring and logprobs_mode variants beyond raw/processed (SAMPLE-LOGPROBS covers the payload only); carried from porting-inventory §6 (T1) at the v1 fold |
T1 | vllm/sampling_params.py:278,727; vllm/config/model.py:82,221 |
- | - | planned: specs/logprob-token-ids.md |
INVENTORIED |
- |
SAMPLE-LOGIT-FILTERS |
Logit bias, allowed-token IDs, bad words | T1 | vllm/sampling_params.py:318,321,337,341,388-413,659-698; vllm/v1/sample/sampler.py:396; vllm/v1/worker/gpu_input_batch.py:446-471; vllm/entrypoints/openai/completion/protocol.py:369-371; tests/v1/sample/test_sampler.py:367,413; tests/v1/sample/test_sampling_params_e2e.py:106,147 |
include/vllm/sampling_params.h+src/vllm/sampling_params.cpp (fields+validation); src/vllm/entrypoints/openai/protocol.cpp (ParseLogitFilters/ApplyLogitFilters clamp); src/vllm/v1/worker/gpu/input_batch.cpp:255,344 (per-slot wiring+condense/swap); src/vllm/v1/engine/input_processor.cpp (bad_words tokenization); src/vllm/v1/sample/sampler.cpp:239; src/vllm/v1/sample/logits_processor/builtin.cpp:41; src/vllm/v1/sample/ops/bad_words.cpp:13,55 |
tests/vllm/v1/sample/test_logits_processors.cpp:121,163,200; tests/vllm/test_sampling_params.cpp (bad_words/allowed_token_ids validation); tests/vllm/entrypoints/openai/test_protocol.cpp (logit_bias clamp+parse); tests/vllm/v1/worker/test_input_batch.cpp (wiring, RED-first); tests/vllm/v1/test_input_processor.cpp (bad_words tokenization) |
sampling-controls-c7.md (SAMPLE-LOGIT-FILTERS) |
ACTIVE |
CLAIM-ROADMAP-C7 |
SERVE-COMPLETION-LONGTAIL |
Best-of, echo, suffix, user request fields | T1 | vllm/entrypoints/openai/completion/protocol.py:56,67,70; tests/entrypoints/openai/completion/test_token_in_token_out.py:56 |
echo parse only include/vllm/entrypoints/openai/protocol.h:196; src/vllm/entrypoints/openai/protocol.cpp:204,295 |
acceptance-only tests/vllm/entrypoints/openai/test_conformance.cpp:589 |
planned: specs/completions-longtail-fields.md |
PARTIAL |
- |
SAMPLE-BEAM |
Beam search: an OUTER loop over the engine (NOT a core-sampler param). Each step runs ONE decode per active beam (logprobs=2*beam_width, max_tokens=1, the beam temperature), expands each beam to those next tokens (cum_logprob += logprob), keeps the top-beam_width by the length-penalty score get_beam_search_score = cum_logprob / seq_len**length_penalty (seq_len INCLUDES the prompt, −1 when the last token is EOS), retires EOS-terminated beams into completed, and after max_tokens (or once all beams complete) returns the top-beam_width completed beams as multiple outputs (reuses the SAMPLE-N multi-output aggregation seam). The scoring + top-k-beam selection + EOS + length-penalty are DETERMINISTIC ⇒ token-EXACT vs vLLM, gated model-free on a hand-computed toy tree. std::stable_sort DESCENDING reproduces vLLM's sorted(reverse=True) tie behaviour. OpenAI-endpoint use_beam_search is WIRED on both /v1/completions and /v1/chat/completions over BOTH engine seams, REAL vLLM-0.26 surface: the SYNC LLMEngine (BeamSearch, offline.py) AND the PRODUCTION AsyncLLM HTTP server (BeamSearchAsync, online.py) — the server (examples/server/main.cpp) holds an AsyncLLM, so a beam request there now RUNS instead of raising "requires the synchronous engine". BeamSearchAsync drives the AsyncLLM per-beam single-token generate (pre-tokenized overload added to AsyncLLM) and calls the SAME model-free BeamSearchStep/get_beam_search_score — the algorithm is shared verbatim via a template driver body, only the engine object differs (mirrors online.py mirroring offline.py). GATE: BeamSearchAsync returns beams token-IDENTICAL to sync BeamSearch over the same synthetic CPU model (tokens/order/scores/text), for beam_width 1/2/3. CONCURRENCY FINDING: per-step beam decodes are issued SEQUENTIALLY (one isolated request each), byte-identical to the sync driver; online.py's asyncio.gather per-beam CONCURRENT stepping is a NAMED RESIDUAL (AsyncLLM supports concurrent requests — a future throughput optimization, correctness-first here). OTHER RESIDUALS: streaming beam (rejected like upstream), C-ABI beam params, grammar-constrained beam search (structured-output bitmask branch), encoder-decoder/LoRA beams |
T1 | vllm/entrypoints/generate/beam_search/utils.py:18,102,112,137,156; vllm/entrypoints/generate/beam_search/offline.py:58,118,160,193,291-327; vllm/entrypoints/generate/beam_search/online.py:28-220 (the OpenAI-serving beam generator); vllm/entrypoints/openai/completion/protocol.py:260/chat_completion/protocol.py:589 (to_beam_search_params); vllm/entrypoints/openai/completion/serving.py:173-205/chat_completion/serving.py:319-343 (use_beam_search routing); vllm/sampling_params.py:1114 (BeamSearchParams) |
include/vllm/entrypoints/beam_search.h + src/vllm/entrypoints/beam_search.cpp (model-free core + shared template BeamSearchDrive + BeamSearch(LLMEngine&, …) sync driver + BeamSearchAsync(AsyncLLM&, …) production driver); include/vllm/v1/engine/async_llm.h+src/vllm/v1/engine/async_llm.cpp (pre-tokenized add_request/generate overloads the async beam driver steps on); include/vllm/entrypoints/openai/protocol.h+src/vllm/entrypoints/openai/protocol.cpp (use_beam_search/length_penalty fields + to_beam_search_params, both requests); src/vllm/entrypoints/openai/serving_completion.cpp + serving_chat.cpp (use_beam_search routes to BeamSearchAsync when async-backed, else BeamSearch + set_beam_search_tokenizer); examples/server/main.cpp (wires set_beam_search_tokenizer on the production handlers so beam runs on the HTTP server); CMakeLists.txt |
tests/vllm/entrypoints/test_beam_search.cpp (model-free token-EXACT tree) + tests/vllm/v1/test_llm_engine.cpp (e2e beam over the CPU engine; BeamSearchAsync == sync BeamSearch token-identical for bw 1/2/3) + tests/vllm/entrypoints/openai/test_serving.cpp (endpoint use_beam_search choices IDENTICAL to the direct driver, completion + chat, over BOTH the sync AND the production AsyncLLM engine; to_beam_search_params round-trip; streaming-beam + tokenizer-less async beam rejected) |
sampling-controls-c7.md (SAMPLE-BEAM) |
ACTIVE |
CLAIM-C7-BEAM-ASYNC |
SAMPLE-REASONING |
Reasoning parsers (<think> reasoning/content split, streamed as reasoning deltas + non-stream reasoning_content) and reasoning-gated grammar integration |
T1 | vllm/reasoning/abs_reasoning_parsers.py:26,213; vllm/reasoning/__init__.py:22 (registry, 28 names); vllm/reasoning/basic_parsers.py:18 (BaseThinking); deepseek_r1_reasoning_parser.py:10; deepseek_v3_reasoning_parser.py:20,83; identity_reasoning_parser.py:17 |
SEAM LANDED (record backfill 2026-07-28 — the seam shipped under da933828/eb9d1291/5fffe7e6 but this row was never advanced): base+registry src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp:19 + include/vllm/entrypoints/openai/reasoning_parsers/abstract.h:53; BaseThinkingReasoningParser src/vllm/entrypoints/openai/reasoning_parsers/basic.cpp:27; parsers deepseek_r1.cpp, mistral.cpp, minimax_m2.cpp, step3.cpp, olmo3.cpp, think_auto.cpp (auto-detect default); template detection + --reasoning-parser resolve src/vllm/entrypoints/openai/reasoning_parsers/detect.cpp:59; C ABI v5 src/capi/vllm_c.cpp (reasoning_parser); serving src/vllm/entrypoints/openai/serving_chat.cpp (reasoning-before-tools routing, reasoning SSE delta). W1 2026-07-28 (CLAIM-SAMPLE-REASONING): + src/vllm/entrypoints/openai/reasoning_parsers/identity.cpp:8 (passthrough delegate) + src/vllm/entrypoints/openai/reasoning_parsers/deepseek_v3.cpp:9 (thinking-gated: deepseek_v3→Identity / holo2→R1) → 9 registered names |
tests/vllm/entrypoints/openai/reasoning_parsers/test_deepseek_v3.cpp:40 (ports tests/reasoning/test_deepseekv3_reasoning_parser.py: thinking-gated selection + identity passthrough + no-think edge, RED-first) + tests/vllm/entrypoints/openai/reasoning_parsers/test_detect.cpp:102 (name-count 7→9) + existing test_{base_thinking,deepseek_r1,mistral,minimax_m2,step3,olmo3,detect,think_auto}.cpp + reasoning_test_utils.h (ports tests/reasoning/utils.py) |
specs/reasoning-parsers.md | ACTIVE |
CLAIM-SAMPLE-REASONING |
SAMPLE-THINKING-BUDGET |
Thinking budget state and logit combination | T1 | vllm/v1/sample/sampler.py:381-386 |
- | - | planned: specs/thinking-budget.md |
INVENTORIED |
- |
SAMPLE-REPETITION |
Repetition detection and penalty state | T1 | vllm/v1/sample/sampler.py:437 |
- | - | planned: specs/repetition-detection.md |
INVENTORIED |
- |
SAMPLE-CUSTOM-PROCESSORS |
Custom logits-processor plugin point — a host-registered per-request callback the sampler invokes each decode step (generated token-ids + a mutable logits view) BEFORE sampling, at vLLM's non-argmax-invariant stage (after allowed_token_ids/bad_words/min_tokens/logit_bias, before penalties). Exposed through the C-ABI (vllm_logits_processor, ABI v8); default (no processor) byte-identical. Mirrors vLLM's SamplingParams.logits_processors structure/ordering; also satisfies SGLang's custom_logit_processor. Residual: single per-request C callback (not a batched plugin graph); no Python-side registration; on the async scheduler the generated-token view is fed by the scheduler (may lag) — the strict token-ids contract is gated at the sampler level |
T2 | vllm/v1/sample/logits_processor/__init__.py:49-97; vllm/v1/sample/logits_processor/interface.py:60; vllm/v1/sample/sampler.py:399; sglang python/sglang/srt/sampling/custom_logit_processor.py:24 |
ABI include/vllm.h:186 (typedef) + :239 (field, v8); include/vllm/logits_processor_callback.h:40; src/capi/vllm_c.cpp:207; include/vllm/sampling_params.h:214; include/vllm/v1/sample/metadata.h:101; src/vllm/v1/sample/logits_processor/builtin.cpp:75 (apply_logits_processors) + include/vllm/v1/sample/logits_processor/builtin.h:59; wired src/vllm/v1/sample/sampler.cpp:308; per-slot src/vllm/v1/worker/gpu/input_batch.cpp:284 + emit :468 |
tests/vllm/v1/sample/test_sampler.cpp:310,353,378 (forces-token EXACT + per-request + inert; RED-first); tests/vllm/v1/sample/test_logits_processors.cpp:236,253,264 (mutate/no-op/null-skip); tests/capi/test_capi.cpp:409 (ABI v8 e2e forces token, fires per step) |
sampling-controls-c7.md (SAMPLE-CUSTOM-PROCESSORS) |
ACTIVE |
CLAIM-C7-CUSTOM-LOGITS |
SAMPLE-ROUTED-EXPERTS |
Routed-experts return (enable_return_routed_experts per-token expert-routing output); carried from porting-inventory §6 (T2) at the v1 fold |
T2 | vllm/v1/outputs.py:281; vllm/sampling_params.py:328 |
- | - | planned: specs/routed-experts-return.md |
INVENTORIED |
- |
SAMPLE-N |
Parallel sampling: a request with n>1 fans out into n child sequences sharing the prompt tokens (and its prefill KV via the block-hash APC), each with its own decode state + RNG offset (seeded children get seed+index), aggregated back into ONE RequestOutput carrying n CompletionOutputs (OpenAI: n indexed choices). n==1 (default) never constructs a ParentRequest — byte-identical single-sequence path. Mirrors vLLM's ParentRequest/child-request machinery. Greedy n>1 is rejected exactly as upstream (_verify_greedy_sampling); the determinism gate uses top_k=1 (a legal n>1 config that collapses to the argmax ⇒ every child token-identical to the single greedy result). RESIDUALS: best_of/beam (SAMPLE-BEAM), async-streaming per-child collation via RequestOutputCollector, and the C-ABI n field (needs an ABI bump + multi-output return) |
T1 | vllm/v1/engine/parallel_sampling.py:13,52,83,100; vllm/v1/engine/llm_engine.py:270-293; vllm/v1/engine/output_processor.py:217,323-331,720; vllm/sampling_params.py:213,625; vllm/entrypoints/offline_utils.py:561 |
include/vllm/v1/engine/parallel_sampling.h+src/vllm/v1/engine/parallel_sampling.cpp (ParentRequest: get_child_info/get_outputs); src/vllm/v1/engine/llm_engine.cpp (FanOutParallelSampling, the n>1 fan-out, n==1 untouched); src/vllm/v1/engine/output_processor.cpp (parent_req aggregation in make_request_output + parent_requests_ cleanup); include/vllm/sampling_params.h:128 (n); src/vllm/entrypoints/openai/protocol.cpp:398,433 (sp.n) |
tests/vllm/v1/test_llm_engine.cpp (n>1 fans out into n token-identical deterministic outputs; RED-first 1→n; n=1 inertness) + tests/vllm/entrypoints/openai/test_serving.cpp (n>1 returns n indexed deterministic choices) |
sampling-controls-c7.md (SAMPLE-N) |
ACTIVE |
CLAIM-C7-N-SAMPLING |
SAMPLE-BEST-OF |
OpenAI best_of endpoint control: generate best_of sequences (via the SAMPLE-N ParentRequest fan-out — sp.n = best_of) and RETURN the n highest-cumulative-logprob ones, re-indexed 0..n-1. best_of >= n; best_of == n (default) is byte-identical (no fan-out, no re-rank). best_of < n rejected; best_of > 1 under greedy rejected via PostInit (upstream greedy restriction). Ranking needs the per-child cumulative logprob, which our engine accumulates only when logprobs are computed, so best_of>n forces sp.logprobs=0 (the sampled-token logprob; NO user-visible payload). HONEST FINDING: vLLM 0.26 has DROPPED best_of from the live path — NOT on CompletionRequest/ChatCompletionRequest, no SamplingParams.best_of; the only survivor is a vestigial, NEVER-consumed field on BatchChatCompletionRequest (chat_completion/protocol.py:1048). We therefore implement the CLASSIC OpenAI-spec / vLLM-V0 best_of contract, gated on OUR deterministic fan-out (there is no 0.26 best_of oracle). RESIDUALS: streaming/async best_of ride the SAMPLE-N engine coverage; C-ABI best_of; grammar-beam N/A |
T1 | vllm/entrypoints/openai/chat_completion/protocol.py:1048 (vestigial best_of); classic OpenAI Completions best_of contract (best_of >= n, return top-n by per-token logprob) |
include/vllm/entrypoints/openai/protocol.h (best_of field, both requests); src/vllm/entrypoints/openai/protocol.cpp (ApplyBestOf — fan-out + forced ranking logprob + best_of<n reject); include/vllm/entrypoints/openai/serving_utils.h+.cpp (SelectBestOf — top-n by cum logprob, stable, re-index, inert); src/vllm/entrypoints/openai/serving_completion.cpp + serving_chat.cpp (guarded trim) |
tests/vllm/entrypoints/openai/test_serving.cpp (SelectBestOf unit rank/tie/inert; best_of→sp.n + forced logprob RED-first; best_of<n reject; e2e best_of=4,n=2 returns exactly n ranked choices; unset/==n inertness) |
sampling-controls-c7.md (SAMPLE-BEST-OF) |
ACTIVE |
CLAIM-C7-BESTOF-BEAM-API |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
TOOLS-STRUCTURED-CORE |
JSON schema/object, regex, choice, grammar, response format | T0 | vllm/v1/structured_output/request.py:77; vllm/v1/structured_output/__init__.py:36,204; vllm/v1/structured_output/backend_xgrammar.py:78; tests/entrypoints/llm/test_struct_output_generate.py:214 |
src/vllm/v1/structured_output/request.cpp:12,43; src/vllm/v1/structured_output/manager.cpp:14,67; src/vllm/v1/structured_output/backend_native.cpp:1334; production wiring src/vllm/entrypoints/model_loader.cpp:283 (manager into Scheduler/EngineCore/AsyncLLM); C ABI v2 src/capi/vllm_c.cpp:143 + include/vllm.h:127 |
tests/vllm/v1/structured_output/test_structured_output.cpp:114,352; tests/vllm/v1/structured_output/test_response_format_e2e.cpp:205; tests/capi/test_capi.cpp:669 (production-wired blocking/streaming constraint + exactly-one rejection); native backend tests |
planned: specs/structured-outputs.md |
PARTIAL |
- |
TOOLS-XGRAMMAR |
xgrammar structured-output backend (vLLM's DEFAULT auto) as a 2nd registerable backend behind the shared seam. W1 landed (CLAIM-TOOLS-XGRAMMAR, 2026-07-29): XgrammarStructuredOutputBackend composes the native matcher (xgrammar's algorithm — pushdown FSM + token-byte trie — already ours portably; §9 decision: mirror portably, do NOT vendor the C++ lib) and adds the xgrammar-FAITHFUL front-end where the two diverge: JSON-schema→EBNF preserving property DECLARATION order (nlohmann::ordered_json) + the any_whitespace ws rule + the basic_* set emitted VERBATIM — closing the whitespace/key-order/exotic-schema parity gap. disable_any_whitespace mirrored (= not any_whitespace). GRAMMAR/REGEX/CHOICE/STRUCTURAL_TAG delegate to the native compile paths. Backend selection mirrored: ResolveStructuredOutputBackend (auto→xgrammar, sampling_params.py:1031) + MakeStructuredOutputBackendFactory. RESIDUALS (W2+): optional object properties, strict-compact separators, the has_xgrammar_unsupported_json_features guard + validate_xgrammar_grammar feeding the auto fallback, model_loader.cpp production wiring, GPU oracle parity (DGX offline). |
T1 | vllm/v1/structured_output/backend_xgrammar.py:36,78,128 (json_schema_converter.cc @ mlc-ai/xgrammar a32ac89); selection sampling_params.py:932-949,1024-1061, __init__.py:133-165, config/structured_outputs.py:13 |
include/vllm/v1/structured_output/backend_xgrammar.h + src/vllm/v1/structured_output/backend_xgrammar.cpp (XgrammarStructuredOutputBackend, ResolveStructuredOutputBackend, MakeXgrammarBackendFactory, MakeStructuredOutputBackendFactory); include/vllm/v1/structured_output/xgrammar_json_schema.h + src/vllm/v1/structured_output/xgrammar_json_schema.cpp (XgrammarJsonSchemaToEbnf, XgrammarJsonObjectEbnf) |
tests/vllm/v1/structured_output/test_backend_xgrammar.cpp 6/6 (39 asserts): exact-valid-next-tokens; declaration key order vs native sort (RED-first); disable_any_whitespace; json_object; converter EBNF; auto→xgrammar selection |
xgrammar-backend.md | ACTIVE |
CLAIM-TOOLS-XGRAMMAR |
TOOLS-STRUCTURAL-TAG |
Full structural-tag surface | T1 | vllm/v1/structured_output/backend_xgrammar.py:108; vllm/tool_parsers/structural_tag_registry.py:238; tests/entrypoints/llm/test_struct_output_generate.py:990 |
src/vllm/v1/structured_output/json_schema_to_gbnf.cpp:415; src/vllm/v1/structured_output/backend_native.cpp:1366; src/vllm/entrypoints/openai/serving_chat.cpp:98 |
tests/vllm/v1/structured_output/test_backend_native.cpp:910,939,952,987; tests/vllm/entrypoints/openai/tool_parsers/test_tool_choice_grammar.cpp:180 |
planned: specs/structural-tag.md |
PARTIAL |
- |
TOOLS-GUIDANCE-OUTLINES |
Guidance, outlines, and LM-format-enforcer backends | T2 | vllm/v1/structured_output/__init__.py:140-159 |
- | - | planned: specs/guidance-outlines-backends.md |
INVENTORIED |
- |
TOOLS-CALLING-CORE |
Auto, required, named tool choice; streaming deltas; Hermes and Qwen3 parsers | T0 | vllm/entrypoints/openai/chat_completion/serving.py:428,688,872; vllm/tool_parsers/hermes_tool_parser.py:34; vllm/tool_parsers/qwen3_engine_tool_parser.py:7 |
src/vllm/entrypoints/openai/serving_chat.cpp:98,224,239; src/vllm/entrypoints/openai/tool_parsers/hermes.cpp:124,180; src/vllm/entrypoints/openai/tool_parsers/qwen3.cpp:13 |
tests/vllm/entrypoints/openai/tool_parsers/test_tool_parsers.cpp:27,121,159; tests/vllm/entrypoints/openai/tool_parsers/test_tool_choice_grammar.cpp:248,260,280; tests/vllm/entrypoints/openai/test_serving.cpp:625,741,903; SELECTION SURFACE 2026-07-24 (CLAIM-DOCS-T2-FIXES) — the bundled OpenAI server no longer hardcodes "hermes"/"": --tool-call-parser and --reasoning-parser (vLLM's own flag names) select any registered dialect, auto runs the same chat-template detection the C ABI uses, none disables, and an unknown name aborts startup listing the registry. The name lists are ENUMERATED from the factories (tool_parser_names() 40 names / reasoning_parser_names() 7), not hand-written at the flag; the whole flag behaviour lives in ResolveToolParserName / ResolveReasoningParserName so it is unit-tested without a server. DEFAULTS REPRODUCE THE OLD HARDCODE EXACTLY (hermes / disabled), so an invocation naming neither flag is unchanged. Anchors: examples/server/main.cpp (flags), src/vllm/entrypoints/openai/tool_parsers/{abstract,detect}.cpp, src/vllm/entrypoints/openai/reasoning_parsers/{abstract,detect}.cpp; tests tests/vllm/entrypoints/openai/tool_parsers/test_detect.cpp, tests/vllm/entrypoints/openai/reasoning_parsers/test_detect.cpp |
planned: specs/tool-calling.md |
PARTIAL |
- |
TOOLS-STREAMING-PARSER |
Unified streaming parser engine for reasoning and tool calls, including token-ID scanning, coalesced deltas, replay adapters and parser-specific configurations | T1 | vllm/parser/engine/streaming_parser_engine.py:89; vllm/parser/engine/token_id_scanner.py:29; vllm/parser/engine/incremental_lexer.py:80; vllm/parser/engine/events.py:11,22; vllm/parser/qwen3.py:88; vllm/parser/kimi_k2.py:52; vllm/parser/engine/parser_engine.py:79 (assembly, residual); tests/parser/engine/test_engine.py, test_token_id_scanner.py, test_qwen3.py @ 555967922 |
CORE engine landed 2026-07-27 (CLAIM-ROADMAP-C8-PARSER): include/vllm/parser/engine/{events,parser_engine_config,incremental_lexer,token_id_scanner,streaming_parser_engine,configs,registry}.h + src/vllm/parser/engine/{incremental_lexer,token_id_scanner,streaming_parser_engine,configs,registry}.cpp (scanner + prefix-buffering lexer + transition state machine + JSON-arg brace hold-back + drop-info + qwen3/seed_oss/kimi_k2 configs + unified name->config registry). ASSEMBLY landed 2026-07-27 (CLAIM-ROADMAP-C8-ASSEMBLY): include/vllm/parser/engine/{parser_engine,py_json}.h + src/vllm/parser/engine/parser_engine.cpp (ParserEngine: SemanticEvent -> streaming DeltaMessage + one-shot ExtractedToolCallInformation, held-back streaming-arg prefix, tool_index++, finish() flush, qwen3 <parameter=> arg-converter) + include/vllm/parser/{kimi_k2,parser_manager}.h + src/vllm/parser/{kimi_k2,parser_manager}.cpp (kimi native-header id/name overrides + name->parser dispatch) + assembly fields on parser_engine_config.h/configs.cpp. SERVING-SSE dispatch swap landed 2026-07-27 (CLAIM-ROADMAP-C8-SERVING): src/vllm/entrypoints/openai/serving_chat.cpp (ShapeChatDeltaEngine/ShapeChatMessageEngine/MakeParserEngine + 3 name-selected drive-site branches + ChatSseStream engine member) + include/vllm/entrypoints/openai/serving_chat.h; faithful include_reasoning request field on protocol.{h,cpp}. CONFIG FAMILIES landed 2026-07-27 (CLAIM-ROADMAP-C8-CONFIGS): 5 more engine-backed families ported as additive ParserEngineConfig builders in src/vllm/parser/engine/configs.cpp (minimax_m2_config, glm47_moe_config, deepseek_v4_config, deepseek_v32_config, nemotron_v3_config + _minimax_m2/_glm47/_dsml std::regex arg-converters) + registry.cpp/parser_manager.cpp dispatch + include/vllm/parser/glm47_moe.{h,cpp} (name-.strip() over the existing hooks). CONFIG FAMILIES C8-2 landed 2026-07-27 (CLAIM-ROADMAP-C8-CONFIGS-2): the last 2 deferred families gemma4 + inkling PORTED — added 4 additive assembly-core virtual seams (default-inert for the other 8 families) preprocess_feed (parser_engine.py:210), virtual events_to_delta (:706), virtual single_pass_parse (:645), args_wrapper_keys from _extract_args_value (:1064) + virtual reset/extract_reasoning; src/vllm/parser/engine/configs.cpp (gemma4_config + _gemma4_arg_converter key:value scanner; inkling_config + _inkling_arg_converter JSON-span carver) + include/vllm/parser/{gemma4,inkling}.{h,cpp} (gemma4 _preprocess_feed channel-injection + _events_to_delta thought\n-strip + extract_reasoning; inkling args_wrapper_keys unwrap + _single_pass_parse trailing flush) + registry.cpp/parser_manager.cpp dispatch. JSON-SCHEMA ARG-TYPE COERCION landed 2026-07-28 (CLAIM-C8-ARG-COERCION): _fix_arg_types / _streamable_string_keys / find_tool_properties (parser_engine.py:227,269,365,348) ported over the ALREADY-ported extract_types_from_schema / coerce_to_schema_type helpers (tool_parsers/utils.cpp) — ParserTool now carries the function parameters JSON-schema (threaded from serving_chat.cpp ToParserRequest), so a request whose tools declare typed params (int/number/bool/string/array/null) has its assembled tool_calls[].function.arguments coerced to the declared types in BOTH streaming (parse_delta) and one-shot (extract_tool_calls/parse); no-schema/absent-tools = identity (byte-identical). Additive to parser_engine.{h,cpp} (recursive _coerce_dict/_coerce_value + find_tool_properties) + serving_chat.cpp schema threading; no other TU changed |
tests/vllm/parser/engine/test_streaming_parser_engine.cpp (586/586, 8 scenarios) + test_parser_engine_assembly.cpp (..._goldens.inc): 5038/5038 field-for-field over 30 scenarios (streaming DeltaMessage + one-shot extract_tool_calls + non-streaming parse()) vs vLLM 0.26 assembly — scenarios 27-30 (CLAIM-C8-ARG-COERCION) add qwen3 typed-schema coercion (whole+char, days5/activetrue/temp3.14/tags[1,2,3] coerced, unit stays string), qwen3 schema-mismatch (uncoercible "abc" left as-is + nullable "null"->null), kimi_k2 JSON-native "5"->int in extract (converter-less: streaming stays raw, extract coerces — divergence gated); RED-first proven (38 asserts, first boundary qwen3_typed_schema_wholedelta extract tc[0] arguments: identity {"days": "5", …} vs coerced {"days": 5, "unit": "celsius", "active": true, "temp": 3.14, "tags": [1, 2, 3]}); scenarios 10-19 (CLAIM-ROADMAP-C8-CONFIGS) add minimax_m2 / glm47_moe / deepseek_v4 / deepseek_v32 / nemotron_v3, scenarios 20-26 (CLAIM-ROADMAP-C8-CONFIGS-2 2026-07-27) add gemma4 (explicit + elided channel, whole+char) + inkling (think/tool/trailing-text + non-object-args fallback, whole+char), each whole-delta AND char-by-char; RED-first proven for all seams (32 asserts _safe_arg_prefix; 2 asserts glm47 name-.strip(); 13 asserts gemma4 _events_to_delta at gemma4_channel_tool_wholedelta delta[0] reasoning; 5 asserts gemma4 _preprocess_feed at gemma4_elided_channel_wholedelta delta[0] content; 4 asserts inkling args_wrapper_keys at inkling_nonobject_args_wholedelta extract tc[0] arguments; 2 asserts inkling _single_pass_parse at inkling_think_tool_text_wholedelta parse content); serving tests/vllm/entrypoints/openai/test_serving_chat_stream.cpp (..._goldens.inc): 210/210 chunk-for-chunk SSE parity over the 9 scenarios vs vLLM 0.26 chat_completion_stream_generator (role frame + per-delta reasoning/content/tool-call deltas + terminal tool_calls flip + name-selected dispatch), RED-first proven (6 CHECKs, first boundary chunk[1] reasoning-vs-raw-content); goldens byte-reproduced by tools/parity/dump_{streaming_parser_engine,parser_engine_assembly,serving_chat_stream}.py |
specs/streaming-parser-engine.md, specs/parser-assembly-c8.md | ACTIVE |
CLAIM-ROADMAP-C8-PARSER, CLAIM-ROADMAP-C8-ASSEMBLY, CLAIM-ROADMAP-C8-SERVING, CLAIM-ROADMAP-C8-CONFIGS, CLAIM-ROADMAP-C8-CONFIGS-2, CLAIM-C8-ARG-COERCION |
TOOLS-PARSER-BREADTH |
Qwen-Coder XML, Mistral, pythonic, and remaining parsers | T1 | vllm/tool_parsers/__init__.py:16-206 |
- | - | planned: specs/tool-parser-breadth.md |
INVENTORIED |
- |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
SPEC-MTP |
Qwen3.6 MTP heads, k=1 first. M-mtp-0 CLOSED 2026-07-24: the standalone draft head is oracle-parity-proven on BOTH checkpoints (27B dense + 35B MoE, k=1, vLLM 0.25.0 executable @ pin e24d1b24) - argmax exact on 26/26 unambiguous rows each; the one remaining row per checkpoint is an EXACT oracle top1==top2 tie where vLLM's own argmax and topk disagree and our pick is a tied maximum; logits within the whole-model bound (atol 0.05 + rtol 0.05), 0/216 out-of-tol on both; shared lm_head isolated is bit-exact on the 35B NVFP4 head. I2 scheduler-half LANDED (2026-07-24): host-side spec plumbing + the FROZEN spec-metadata ABI (spec §2.7) - SpeculativeConfig, DraftTokenIds, Request::spec_token_ids/NumTokensWithSpec, populated scheduled_spec_decode_tokens, Scheduler::update_draft_token_ids, take_draft_token_ids seam, EngineCore::post_step, InputBatch::num_accepted_tokens/update_req_spec_token_ids; DEFAULT-OFF and INERT (no SpeculativeConfig => num_lookahead_tokens == 0). I3 verify-half LANDED (2026-07-24): greedy rejection sampler + per-request logits expansion (see SPEC-REJECTION, now ACTIVE). I4 GDN-half LANDED (2026-07-24): the GDN speculative slot path + bit-exact state rollback, the piece BOTH GDN-hybrid gate checkpoints need (see SPEC-GDN-SEGMENTS, now ACTIVE). I5a GDN LAYER ROUTING + runner spec-metadata upload LANDED (2026-07-24, CLAIM-SPEC-MTP-I5A): GdnBlockPaged now routes a pure-spec batch through vt::GdnSpecDecode/vt::CausalConv1dSpecUpdate and the runner uploads I4's six spec device tensors — first sub-increment of the scoped M-mtp-1 (I5a GDN wiring → I5b prepare_prefill → I5c MTP paged propose → I5d config+runner-loop+the 27B token gate, spec §5). DEFAULT-OFF INERT, bit-exact vs the I4 ops, no e2e loop yet. I5b prepare_prefill_inputs LANDED (2026-07-24, CLAIM-SPEC-MTP-I5B, recorded under SPEC-REJECTION): the drafter prefill input-prep host routine (shift-splice + query_len -= num_rejected + last-token index / metadata) — second scoped M-mtp-1 sub-increment, DEFAULT-OFF INERT, unit-gated RED-first, additive. I5d CONFIG + RUNNER LOOP LANDED, PARTIAL (2026-07-25, CLAIM-SPEC-MTP-I5D): --speculative-config JSON parse -> EngineParams::speculative_config; LoadedEngine resolution (ResolveSpecConfig/ResolveMtp, widened KV MakeQwen3_5KVCacheSpec(num_spec>0), BuildMtpDraft, forced sync scheduling, MakeScheduler(spec), EngineCore(check_for_draft=true)); the full runner verify/propose loop (draft splice, hidden-tap capture, GDN builder spec-overload feed, k+1 GDN state-slot remap + widened conv cache + draft-KV alloc, MtpProposePrefill post-sampling, take_draft_token_ids, acceptance telemetry). CUDA -Werror 0 warnings, cutlass-ON banner. SPEC-OFF BYTE-IDENTICAL (all gated on spec_on()): SACRED 27B 235/235, 35B 315/315, Coder 138/138 + unit test_runner 257 / test_mtp_speculator 169 / test_gdn_metadata_builder 483 / test_ops_gdn 3630 ALL PASS. The three-way 27B token gate is NOT yet passing (tests/parity/test_qwen27_spec_decode.cpp RUNS the loop + MEASURES the blocker): the spec-ON engine throws on the FIRST prefill step at gdn_state_gather: working/cache row shapes must match (src/vt/ops.cpp:1773) — I4's spec conv rollback needs the conv row widened to (K-1)+num_spec but the non-spec GDN conv ops assume (K-1). Closing needs widened-cache-aware non-spec GDN conv ops + the MIXED GdnBlockPaged split/merge. Row LEFT GATING at I5e. I5e LANDED 2026-07-25 (CLAIM-SPEC-MTP-I5E) — SPEC-MTP LEAVES GATING. Made the non-spec GDN conv ops widened-cache-aware (mirror vLLM state_len=KERNEL_WIDTH-1 + physical stride_conv_state_tok; leading (K-1) sub-window; byte-identical at num_spec==0, contiguous fast path kept) AND RCA'd the resulting 0-acceptance dead-drafter to the async input-combine overwriting the verify batch's draft position with the committed token (forced off under spec, nullopt-guarded). THREE-WAY 27B GATE PASSES (single-request greedy): our-ON == vLLM --speculative-config mtp greedy == our-OFF token-for-token; acceptance 16/16 drafts accepted, ~16 target steps saved. Spec-OFF SACRED byte-identical (27B 235/235, 35B 315/315, Coder 138/138), test_ops_gdn 3678, compute-sanitizer 0 on the spec step. NOT DONE: MIXED GdnBlockPaged split/merge (concurrency) + throughput A/B are I6. I6 LANDED 2026-07-25 (CLAIM-SPEC-MTP-I6), benchmark_binding=true — the §5 c1 THROUGHPUT GATE, first spec-decode speed number: OURS spec-ON (examples/vllm-bench + an additive --speculative-config flag, production config) vs pinned vLLM 0.25.0 spec-ON (graphed vllm serve --speculative-config mtp + vllm bench serve, enforce_eager=False/FULL_AND_PIECEWISE/inductor; MTP confirmed Resolved architecture: Qwen3_5MTP), SAME {"method":"mtp","num_speculative_tokens":1}, 27B ~/bench/q36-27b-nvfp4-vllm, c1, greedy, 8 real prompts x 256 out, prose + code, idle box one-engine-at-a-time under one flock, 3 reps (cold TTFT discarded), token-identity re-confirmed FIRST (test_qwen27_spec_decode PASS 16/16). RESULT — ours AT/ABOVE vLLM on EVERY measured axis (prose / code): TPOT 66.2/62.95 vs 69.1/65.3 ms (ours ~1.04x faster), output tput 15.10/15.72 vs 14.43/15.13 tok/s (+4.6%/+3.9%), ITL 121.6/121.1 vs 123.2 ms, TTFT(warm) 131/131 vs 151.5/181 ms, acceptance ours 0.85/0.92 vs vLLM 0.838 overall (within noise, live drafter both), peak RSS 28.4 GB ON / 24.8 GB OFF (both inside the 119 GiB pool). Spec helps both (ours 1.52x/1.59x, vLLM 1.51x/1.60x TPOT); ours already ~4% faster spec-OFF. STAYS ACTIVE: the c>1 mixed spec+non-spec GdnBlockPaged split/merge is still refused (needs a row IndexSelect/IndexCopy vt op) + owes a c>1 A/B, and no user-facing supported --speculative-config on the OpenAI server yet (bench flag example-only/additive). Raw logs dgx ~/work/mtp-bench-i6/{results,vresults}. I7 LANDED 2026-07-25 (CLAIM-SPEC-MTP-I7, benchmark_binding=true) — the MIXED spec+non-spec GDN batch (concurrency), the server/CLI --speculative-config, and the c>1 A/B — implementation COMPLETE + at vLLM parity; STAYS ACTIVE for one honest reason (below), NOT a lag. New row op vt::IndexSelect/vt::IndexCopy (CUDA==CPU bit-exact at GDN widths, RED-first); GdnBlockPagedMixedSpec split/merge (mirror qwen_gdn_linear_attn.py:1329-1576) proven MODEL-INDEPENDENTLY bit-exact (mixed == pure spec + pure prefill, 27B/35B, test_qwen3_5_gdn_spec_routing, RED-first by a broken merge); compute-sanitizer 0 on the mixed step + op; server (I5d) + CLI (ABI v6) --speculative-config. c>1 A/B (both spec-ON, same config): ours ON-PAR-OR-ABOVE vLLM at c2/c4/c8 (output tput within ~+/-2%, ours +1.6%/+2.5% c2, +0.9%/+1.7% c4, +0.9%/-1.1% c8 within noise, prose/code; both ~1.5x spec speedup — does NOT go neutral; acceptance 0.84-0.92 vs vLLM 0.835). Why STAYS ACTIVE (honest, not a lag): the DONE criterion's strict token-exact at c>1 clause is a proven MODEL impossibility — the 27B greedy is bf16-batch-nondeterministic (spec-OFF max_seqs 4-vs-1 differs 2/3 short prompts, NO spec involved), affecting vLLM identically, so exact c>1 token identity cannot be met by any correct implementation; c>1 correctness is instead established by the model-independent bit-exact split/merge proof + acceptance parity (near-tie-distributional-gate), with token-exact strict at c1 (I6). No missing work, no lever — the DONE final call is deferred to the user given this criterion ambiguity. SACRED spec-OFF byte-identical 27B 235/235, 35B 315/315, Coder 138/138; CUDA -Werror 0 warnings. Raw logs dgx ~/work/mixed-batch/{cN_results,cN_vresults}. I8 — SPEC-MTP → DONE 2026-07-26 (CLAIM-SPEC-MTP-DONE, records-only, ZERO code): the user RATIFIED the deferred c>1 criterion — at concurrency > 1 the DONE bar is the near-tie-distributional form (ours ∈ vLLM's batch-nondeterministic set) + the SPEED delta, NOT strict token-exact (a proven bf16-batch-nondeterminism MODEL impossibility that affects vLLM identically). Both I6-owed DONE items are therefore CLOSED: (1) the MIXED spec+non-spec GdnBlockPaged split/merge (I7, model-independently bit-exact + compute-sanitizer 0) with the c2-c8 A/B on-par-or-above vLLM, and (2) the server + CLI + C-ABI(v6) --speculative-config flag (I5d/I7, examples/server/main.cpp+examples/cli/main.cpp+src/capi/vllm_c.cpp). MTP k=1 spec-decode is COMPLETE and gated: 27B three-way token-exact at c1 (I5e), c1 above vLLM on every axis (I6), c2-c8 on-par-or-above (I7), spec-OFF byte-identical SACRED (27B 235/235, 35B 315/315, Coder 138/138). This transition is byte-identical BY CONSTRUCTION (git diff --stat = records only; ZERO src//include//examples/ touched, so the I5d/I6/I7 GPU gates stand on this exact code). Tracked follow-ons: the 35B Qwen3_5MoeMTP full e2e token gate (M-mtp-2) is now CLOSED — DONE 2026-07-26 (CLAIM-SPEC-MTP-M-MTP-2): three-way token-exact 16/16 vs the live vLLM 0.25.0 oracle (spec-ON AND spec-OFF), acceptance 16/16 both sides, c1 spec-ON 1.19x TPOT / +16.3% output-tput vs spec-OFF (0.908) — MODEL-SPEC-qwen3-5-mtp-qwen3-5-moe-mtp GATING→DONE, so MTP is DONE on BOTH gate models. Remaining spec-decode follow-on: SPEC-DFLASH (oracle-BLOCKED, vllm#40898) |
T1 | vllm/v1/worker/gpu/spec_decode/mtp/speculator.py:12; vllm/model_executor/models/qwen3_5_mtp.py:63,129-165,192-301; I5d vllm/engine/arg_utils.py (--speculative-config); vllm/v1/worker/gpu/model_runner.py:1455-1489 |
include/vllm/config/speculative.h; include/vllm/v1/core/sched/scheduler.h; src/vllm/v1/core/sched/scheduler.cpp; include/vllm/v1/worker/gpu/input_batch.h; include/vllm/model_executor/models/qwen3_5_mtp.h:23,58; src/vllm/model_executor/models/qwen3_5_mtp.cpp:271; src/vllm/model_executor/models/qwen3_5.cpp:3336,3359; I5d src/vllm/config/speculative.cpp; src/vllm/entrypoints/model_loader.cpp (ResolveSpecConfig/MakeKVCacheMaybeSpec/ctor wiring); src/vllm/v1/worker/gpu/runner.cpp (splice/tap/GDN spec feed/propose_drafts/take_draft_token_ids/spec-slot remap/draft-KV alloc); examples/server/main.cpp |
tests/vllm/v1/test_scheduler.cpp:1135,1238,1272,1316; tests/vllm/v1/worker/test_input_batch.cpp; tests/vllm/v1/spec_decode/test_mtp_speculator.cpp:201,225,263,299,331 (7/7 cases, 141 assertions); oracle runner tests/parity/test_op_parity.cpp:1373 + focused case :1914 (20/20 assertions, both checkpoints, VLLM_MTP_REQUIRE_CHECKPOINTS=1); goldens tests/parity/goldens/qwen3_5_mtp_head_{27b,35b}/; dump tools/parity/dump_qwen3_5_mtp.py:144; I5d tests/parity/test_qwen27_spec_decode.cpp (three-way gate, RUNS + measures the RCA blocker); I6 examples/bench/{main.cpp,bench_core.h} (additive --speculative-config bench flag + acceptance telemetry); I7 tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp (mixed == pure spec + prefill bit-exact), tests/parity/test_qwen27_spec_decode_concurrent.cpp, tests/vt/test_ops_gdn.cpp (IndexSelect/IndexCopy); DONE closure ledger |
mtp-spec-decode.md | DONE |
72f9fb1 |
SPEC-MTP-GGUF |
MTP speculative decoding from a GGUF TARGET. Today FromModelDir refuses mtp+GGUF outright (src/vllm/entrypoints/model_loader.cpp:717-723) on the original spike's assumption that GGUF exports carry no mtp.* (mtp-spec-decode.md:979-980, "until we re-export GGUFs with the head"). That is stale: llama.cpp's Qwen3.5 converter DOES emit the head, under layer-indexed nextn naming, and our own HfConfigFromGguf ALREADY reads nextn_predict_layers (it just discards the value into the trunk layer count). Gap is a TensorResolver over GgufFile mapping mtp.* onto blk.{L+i}.nextn.* with dequant-to-bf16, one config field, and narrowing the rejection to dflash. ngram+GGUF already works and is untouched. Qwen3.5/3.6 only (the widened spec KV path serves no other arch). NO ABI change |
T2 | llama.cpp (the producer contract; vLLM has no GGUF MTP path) conversion/qwen.py:535-604 _Qwen35MtpMixin (the authoritative mtp.*->nextn remapper + add_nextn_predict_layers); gguf-py/gguf/constants.py:129,910-917,1494-1501; gguf-py/gguf/tensor_mapping.py NEXTN_* |
G1-G3 LANDED 2026-07-28. HfConfigFromGguf republishes the head depth src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:598 (c.raw["mtp_num_hidden_layers"] = nextn, previously read then discarded); the head loader LoadQwen3_5MTPFromGguf src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:971 (+ decl include/vllm/model_executor/models/qwen3_5_gguf_weights.h:143) reusing the TRUNK helpers OwnNormMinus1/OwnMatmulWeight/OwnBf16/LoadAttnGguf/LoadMoeGguf so the head inherits the GGUF (w+1) norm storage, quantization/residency routing and torch [N,K] shapes; NumMtpLayers/UsesDedicatedEmbeddings exported out of the anon namespace include/vllm/model_executor/models/qwen3_5_mtp.h; rejection narrowed to dflash + a head-less-GGUF check src/vllm/entrypoints/model_loader.cpp and the head attached in the GGUF branch; G4 GREEN + CPU-SPEC-DIVERGENCE FIXED 2026-07-28: root cause src/vllm/model_executor/models/qwen3_5.cpp:3616 sized the GDN state gather/scatter row by (Kw-1) while the speculative persistent row is (Kw-1)+num_spec, so GatherRows/ScatterRows mis-strode the slot AND every channel past the first, corrupting post-prefill recurrent state. Fix = CopyStateRowsStrided (same TU) used by GatherStateF32/ScatterStateF32 when cache.shape[2] != work.shape[2]; the contiguous helpers are kept when the widths agree, so every non-spec path is byte-identical by construction. CPU-only in effect (the fp16/bf16 arm routes through the GdnStateGather/Scatter ops, so CUDA was never exposed; no GPU result affected) |
tests/vllm/models/test_qwen3_5_gguf_mtp.cpp:37,58 2 cases / 18 assertions against a REAL llama.cpp-converted Qwen3.5-2B (env-gated VLLM_MTP_GGUF_MODEL, so CI stays asset-free): depth reaches config.raw; fc is [H,2H] verbatim; 3 norms [H]; head block is full-attention. RED-first BEHAVIOURAL (reverting only the G1 line fails both cases 2/2). Trunk inertness: test_gguf 103, test_gguf_qwen36_loader 99, test_gguf_keep_quant 5958, test_gguf_dequant 215, test_capi 33/232 all unchanged; tests/parity/test_qwen35_gguf_spec_decode.cpp:74,139 - spec-ON == spec-OFF token-exact with 13 proposed/11 accepted, plus an ngram regression guard (widens the cache, never runs the spec conv update) that was token-exact throughout and pinned the widening as innocent. Regression sweep all unchanged: ops_gdn 1825, gdn_metadata_builder 483, gdn_prefill_conv 28, gdn_spec_routing 12, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, gguf_dequant 215, llm_engine 196, input_batch 163, runner 257, capi 232 GPU CLOSE-OUT + DEVICE-DELTA ATTRIBUTION 2026-07-28 (G5-G7), ledger parity-ledger.md#L800. The GPU end-to-end gate re-run on a from-scratch RELEASE-TARGET build (-DVLLM_CPP_CUDA_ARCHITECTURES=121a, build dir DELETED first; arch VERIFIED by build-cuda/CMakeFiles/vllm.dir/flags.make --generate-code=arch=compute_121a,code=[compute_121a,sm_121a] and by cuobjdump -lelf 20 cubins ALL sm_121a zero sm_75, NOT by CMakeCache.txt, whose CMAKE_CUDA_ARCHITECTURES:STRING=75 is the enable_language(CUDA) compiler-probe default shadowed by the normal variable at CMakeLists.txt:186 - the prior wrong-arch conclusion was that decoy): dgx.casa GB10 under flock $HOME/gpu.lock, 35B A3B NVFP4 GGUF, 2/2 cases, 10/10 assertions, exit 0, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted, 90.2 GiB peak RSS, 8m01s; re-run on the EXACT committed source 3/3 cases, 10/10 assertions, exit 0, 7m25s, the new probe case SKIPping and adding zero assertions. The CPU-vs-GPU token delta is a MEASURED near-tie, not a defect (it was never this row's bar - spec-ON == spec-OFF WITHIN a device is): NEW double-gated spec-OFF-only probe tests/parity/test_qwen35_gguf_spec_decode.cpp:217 (asset + VLLM_MTP_GGUF_PROBE=1, 20 alternatives per position, 484/484 assertions per arm, GPU then CUDA_VISIBLE_DEVICES= in one flock series) shows both arms picking 11751 at position 0 and forking at position 1 on a BIT-IDENTICAL prefix: GPU rank1 13 -0.773180 over rank2 11 -0.847055 (margin 0.0739 nats), CPU rank1 11 -0.765499 over rank2 13 -0.830374 (margin 0.0649 nats). Each device's pick is the other's rank 2, both ~7x inside the ratified 0.5-nat band, and the cross-device disagreement on the SAME token (0.057 and 0.082 nats) EXCEEDS the margin being decided, so rounding settles it; the 24 texts look unrelated only because positions 2+ cascade off that one coin flip. Margin sweep over all 24 positions: GGUF GPU and GGUF CPU carry ZERO exact ties, minimum margins 0.0482 and 0.0649 nats, and both arms reproduced their sequence across every run. Gate 4 MET on the safetensors sibling of the same quantization run (FromModelDir takes it unchanged): acceptance 12 proposed / 11 accepted vs the GGUF's 13 / 11. That arm, however, FAILS spec-ON == spec-OFF at concurrency 1 and does not reproduce its own spec-OFF sequence run to run, and the probe attributes both to THREE EXACT ties (positions 7, 10, 16, bit-identical logprobs) produced by its 1/16-grid quantized-GEMM logits - which EXONERATES the GGUF arm and opens a recorded, not-root-caused SPEC-MTP item on the safetensors NVFP4 path, not on this row. Gate 3 is NOT APPLICABLE twice over: no F16/F32 head-carrying export exists, and the only same-weights sibling is not token-stable against itself. EVIDENCE RE-ANCHORED 2026-07-29 to a PRODUCTION-CONFIGURED build, because every GPU number above came from a build configured WITHOUT -DVLLM_CPP_CUTLASS_DIR and WITHOUT -DVLLM_CPP_TRITON=ON (the defect CLAIM-27B-GATE-RCA proved, which runs the emulation fp4 GEMM + hand GDN kernels). Re-run from a clean git archive tree of main 3f34534d, build proven correct three ways (configure log has ZERO CUTLASS not found and prints CUTLASS found ... sm120a NVFP4 cutlass GEMM + FlashAttention-2 ... ENABLED for arch(es) [121a] + the vendored sm_121a Triton-AOT lines with MANIFEST hashes OK; cuobjdump -lelf 40 cubins ALL sm_121a, zero sm_75; SACRED test_qwen27_paged_engine 235/235 exit 0, and the build precondition proven to FIRE by recompiling only that TU without the two defines against the same libvllm.a, which throws and exits 1 with 0 assertions). The row PASSES UNCHANGED: tests/parity/test_qwen35_gguf_spec_decode 3/3 cases, 10/10 assertions, exit 0, spec-ON token-identical to spec-OFF, 13 proposed / 11 accepted (identical to the recorded number), 90.26 GiB, 7m13.59s; loader gate 19 assertions on the Qwen3.5-2B and 18 on the 35B A3B, unchanged. ONE recorded finding is RETRACTED by the re-measurement: the CPU-vs-GPU token delta was a BUILD artifact, not a device near-tie cascade. On the production build both devices emit the SAME 24 tokens; the probe shows GPU rank1 11 -0.763897 over rank2 13 -0.824083 where the defective build had rank1 13 -0.773180 over rank2 11 -0.847055, while the CPU arm is bit-identical to the earlier measurement (CUTLASS and Triton are CUDA-only). Zero exact ties in either arm, min margins 0.060186 GPU / 0.064875 CPU, 484/484 assertions per arm. Evidence: docs/BENCHMARKS.md top section, parity-ledger.md |
specs/gguf-mtp-spec-decode.md | DONE |
edf91449 |
SPEC-DFLASH-GGUF |
DFlash speculative decoding from GGUF, two axes: (A) GGUF DRAFT + safetensors target, (B) GGUF target too. llama.cpp master carries a full dflash GGUF contract (arch string dflash, tensors fc/enc.output_norm/output_norm/blk.N.*, KVs dflash.target_layers + dflash.target_hidden_size); the arch is ABSENT from checkouts older than ~2026-07, so a stale tree reads as "no contract exists". The GGUF tensor set omits token_embd/output because the draft SHARES the target's embed+lm_head, which is exactly what LoadDflashDraft already does. Blockers are in the loader, not the model: MakeDflashDraftConfig reads draft_dir/config.json (a GGUF has none), ResolveDflashDraftDir probes for config.json so it cannot see a .gguf, and LoadDflashDraft is typed on std::vector<SafetensorsFile> for the shared bf16 head (the axis-B blocker). Axis A independently shippable. NO ABI change |
T2 | llama.cpp origin/master @ 2026-07-28 (tag era b10158): gguf-py/gguf/constants.py:547,1151,4350; gguf-py/gguf/tensor_mapping.py:1297-1305 (ENC_OUTPUT_NORM<-model.hidden_norm, FC<-model.fc); conversion/qwen.py:351 (mask token via the standard tokenizer KV); convert_hf_to_gguf.py --target-model-dir |
GD1-GD7 LANDED 2026-07-28 (BOTH AXES COMPLETE and PROVEN end to end on GB10): MakeDflashGgufConfig + LoadQwen3DFlashFromGguf src/vllm/model_executor/models/qwen3_dflash_gguf.cpp:88,227 (+ header), IsDflashGgufDraft + the .gguf branch in ResolveDflashDraftDir/LoadDflashDraft src/vllm/entrypoints/model_loader.cpp:121,222. Goes through the TensorResolver seam (unlike SPEC-MTP-GGUF) because dflash norms are RAW, so the existing LoadQwen3DFlash qkv/gate_up concatenation is reused unchanged. GD4 defect FIXED (model_loader.cpp:238-249): the GGUF branch left config.vocab_size 0 - correct for MakeDflashGgufConfig (the DFLASH arch has no vocab KV and no token_embd) but fatal for the forward, which sizes the shared embedding view as {config.vocab_size, H}, so the first propose threw cuda embedding: empty table (vocab 0). Now back-filled from the target's embed_tokens rows (the condition is on the VALUE, not the draft source, so it generalizes to a GGUF target). Load-level green had hidden it; only GENERATING found it. GD5-GD7 = axis B: SharedHeadSource src/vllm/entrypoints/model_loader.cpp re-expresses the shared bf16 embed_tokens+lm_head seam as a SOURCE and re-types LoadDflashDraft's second parameter - THAT TYPE was the whole axis-B blocker - with the GGUF arm LoadGgufSharedEmbedAndHeadBf16 src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:773 reusing the trunk loader's tied-embedding rule and sidecar-aware dequant instead of restating them; the shared-head load moved into ONE common tail so all four (draft format x target container) combinations run identical code; the dflash half of the GGUF-branch rejection model_loader.cpp is deleted (the mtp half untouched) and the draft load is wired into the GGUF branch |
tests/vllm/models/test_qwen3_dflash_gguf.cpp:36,84 2 cases / 47 assertions against the REAL published Qwen3.6-27B DFlash draft (env-gated VLLM_DFLASH_GGUF_MODEL, CI asset-free): the +1 target-layer offset undone against the KV read back from the same file, block_size/mask_token present, vocab_size left 0, layer_types cover every block, fc [H, H*num_taps] with nk SET, qkv/gate_up row-concat shapes, embed/lm_head left EMPTY for the target. RED-first BEHAVIOURAL (dropping the -1 fails the offset checks). GD4 e2e gate tests/parity/test_qwen27_dflash_spec_decode.cpp:343 (second case, draft source env-driven via VLLM_DFLASH_DRAFT/_B; asset-gated, CI-inert): on dgx GB10 sm_121a against the Qwen3.6-27B NVFP4 safetensors target, the Q4_K_M GGUF draft and the bf16 z-lab safetensors draft produce token-for-token IDENTICAL DFlash-ON continuations with IDENTICAL accepted/proposed (20/80 on a 24-token prompt, 42/96 on a 48-token prompt), spec-OFF self-reproducible 3/3 and 0 exact ties (min margin 0.197/0.400 nats). Regression: gguf_mtp 19, qwen35_gguf_spec_decode 10, gguf 103, gguf_qwen36_loader 99, gguf_keep_quant 5958, ops_gdn 1825, llm_engine 196, capi 232, runner 257 all unchanged. GD5 unit gate tests/vllm/test_gguf_qwen36_loader.cpp 3 new synthetic-GGUF cases (6 cases / 286 assertions total, CPU and the dgx CUDA build): the untied head really comes from output.weight and not the embedding (distinct fill values), the tied fallback aliases it onto token_embd, the nk flags separate the gather table from the MatmulBT weight, a file with no token_embd is refused. 3-mutant battery, 3 caught (nk flipped, head forced to the embedding, tied forced false). GD7 e2e gate tests/parity/test_qwen27_dflash_spec_decode.cpp third case (targets env-driven via VLLM_DFLASH_TARGET_B; asset-gated, CI-inert): on dgx GB10 sm_121a the Qwen3.6-27B NVFP4 GGUF target + Q4_K_M GGUF draft loads, takes the shared head from the GGUF, generates, and its DFlash-ON continuation is token-for-token IDENTICAL to that same target's spec-OFF (24/24, the STRICT form) with acceptance ALIVE at 14/160; 1 case / 15 assertions, exit 0. The spike's highest risk is EMPTY on this asset, proven not assumed: the 27B NVFP4 GGUF stores token_embd/output as ggml BF16, byte-identical to the safetensors sibling (2,542,796,800 bytes each, ZERO differing), so B1's shared-head read is verbatim, not a dequant. Acceptance IS lower than the safetensors-target arm and is NOT chargeable to the head: the two containers diverge at index 4 with NO speculation, because QUANT-GGUF-NVFP4 is dequant-only so the GGUF target computes in bf16 while the safetensors target runs the true W4A4 kernels. RE-MEASURED 2026-07-29 on a PRODUCTION-CONFIGURED build (CLAIM-GGUF-SPEC-REVERIFY), because every GD4/GD7 GPU number above came from a build configured WITHOUT -DVLLM_CPP_CUTLASS_DIR and WITHOUT -DVLLM_CPP_TRITON=ON. Build proven correct three ways (see the SPEC-MTP-GGUF row; SACRED 27B 235/235, cuobjdump 40 cubins all sm_121a). AXIS B HOLDS EXACTLY: test_qwen27_dflash_spec_decode -tc="dflash axis-B*" 15/15 assertions, exit 0, GGUF-target DFlash-ON token-identical to that target's own spec-OFF 24/24, acceptance 14/160 unchanged, cross-target spec-OFF divergence still at index 4, 81.01 GiB peak RSS, 6m53.08s. AXIS A WAS RED ON THE 48-TOKEN PROMPT (reproducibly, 3 of 3 runs) AND IS NOW CLOSED. The RED was real: cross-format TOKEN identity held on both prompts, but the exact accept-count half of bar (a) failed (arm_a.proposed == arm_b.proposed / arm_a.accepted == arm_b.accepted) because the Q4_K_M draft measured 46/112 against the bf16 z-lab draft's 47/96 (one extra 16-wide propose block, one fewer acceptance, zero token difference), 15/17, exit 1; the 24-token prompt stayed green at 17/17 with both drafts at 15/144. GD9 2026-07-29 root-caused it IN WEIGHT SPACE as ordinary Q4_K_M cost, category (a), not a defect in our GGUF draft path - and the bar's own premise ("Same weights, two containers") was false for the asset it was pointed at. The publishing repo also carries an UNQUANTIZED BF16 GGUF (3,471,497,440 B) beside Q8_0/Q6_K/Q5_K/Q4_K_M, which the spec had recorded as nonexistent; that retired the NOT APPLICABLE on gate 2. CPU gate tests/vllm/models/test_qwen3_dflash_gguf.cpp third case (asset-gated VLLM_DFLASH_GGUF_BF16_MODEL + VLLM_DFLASH_ST_DIR): LoadQwen3DFlashFromGguf(BF16) is BYTE-IDENTICAL to LoadQwen3DFlash(z-lab shards) on all 58 tensors, 302/302 assertions, exit 0, and FUNCTIONALLY RED against the Q4_K_M file (21/302 red, exactly the 21 quantized matmul tensors), so not a vacuous pass. Supporting: our DequantGgufRowToBf16 is bit-equal to gguf-py's gguf.quants.dequantize on the real fc.weight (Q4_K), blk.0.attn_q.weight (Q4_K) and blk.2.ffn_down.weight (Q6_K), zero differing bf16 values; the ladder's mean relative weight error is monotone and uniform with NO outlier tensor (BF16 0, Q8_0 5.6e-3, Q6_K 1.85e-2, Q5_K 3.85e-2, Q4_K_M 7.6e-2); the only numeric config delta is rms_norm_eps at 2.5e-9 relative. Also landed: an off-by-default VT_SPEC_TRACE=1 per-block propose/accept trace in GPUModelRunner::sample_tokens_with_rejection (src/vllm/v1/worker/gpu/runner.cpp). GD10 2026-07-29 CONFIRMED IT END TO END ON GB10 and closed gates 3 and 5. Build proven production-configured three ways (configure log 0 CUTLASS not found; cuobjdump -lelf 40 cubins ALL sm_121a zero sm_75 on both binaries; SACRED test_qwen27_paged_engine 235/235, exit 0, 31.34s, 23.67 GiB). The BF16 GGUF draft reads EXACTLY 47/96, the safetensors draft's own number, at 48 tokens on the discriminating prompt - reproduced 2 of 2 - plus 27/64 = 27/64 at 24 tokens and 15/144 = 15/144 on the second prompt, tokens IDENTICAL throughout, 17/17 exit 0 each time; the Q4_K_M arm reads 46/112 on the SAME binary in the SAME flock series. Restoring only the draft's numeric precision restores the count, so quantization is the whole cause and nothing structural survives. Bar (a) is consequently SPLIT rather than relaxed (tests/parity/test_qwen27_dflash_spec_decode.cpp): tokens stay EXACT unconditionally; accept counts are EXACT on a cross-FORMAT arm and BANDED (abs(d_accepted) <= 2, abs(d_proposed) <= k*2) on a cross-QUANTIZATION one, with the arm chosen by IsQuantizedGgufDraft reading the draft file's ggml types (GgmlTraits().block_elems > 1) rather than by a flag. The band is derived, not picked: measured d_accepted is 0, 0, -1, so the bound is that maximum plus one quantum; and d_proposed = -k * d_accepted EXACTLY once the token streams match (confirmed at -1 / +16), so the proposed bound follows. Mutation-proved non-vacuous: rebuilt at band 0 the Q4_K_M arm is 15/17 exit 1 while the BF16 arm stays 17/17 exit 0 on the exact branch. AXIS B BROADENED from ONE prompt to THREE, strict form green on all: "The capital of France is" IDENTICAL 14/160 (15/15), "Write a Python function that reverses a string:" IDENTICAL 24/64 (15/15), "Photosynthesis is the process by which" IDENTICAL 15/128 (9/9), all exit 0, ~6m30-6m52 and ~81 GiB peak RSS each. The second prompt REFINES the recorded acceptance claim: the safetensors-target arm is ALSO 24/64 there with the two containers' DFlash-ON streams IDENTICAL, so the GGUF target's lower acceptance is prompt-dependent (their spec-OFF streams diverge at index 4 on the first prompt, index 16 on the second) and not a standing penalty; the cause remains QUANT-GGUF-NVFP4 being dequant-only, with the shared head excluded by a byte comparison. Gates 1-5 and 7 MET; gate 6 (speed) PENDING BY DESIGN and not owed - a DFlash-ON throughput A/B between the two target containers is not a fair comparison until a native NVFP4 GGUF GEMM exists. Evidence: docs/BENCHMARKS.md top section, parity-ledger.md |
specs/gguf-dflash-draft.md | DONE |
c62f2fa3 |
SPEC-REJECTION |
Rejection sampler. I3 verify half LANDED (2026-07-24): per-request logits EXPANSION to 1 + k_i rows (StepInputs::cu_num_logits / num_draft_tokens_per_req / expanded logits_indices) plus the GREEDY rejection sampler — accept a draft iff it equals the target argmax at its own position, emit the target argmax on the FIRST mismatch and stop, emit the bonus argmax when all k_i accept, num_sampled = accepted + 1, num_rejected = k_i - accepted (feeds I2's num_computed_tokens rollback and InputBatch::num_accepted_tokens). One additive vt op (kGreedyRejectionSample) with a CPU reference and a CUDA two-phase mirror of upstream's row-argmax + one-thread-per-request accept walk. DEFAULT-OFF and INERT: with no SpeculativeConfig no drafts are ever scheduled, cu_num_logits is arange(num_reqs+1), logits_indices is the pre-change array and the runner never enters the rejection branch. STOCHASTIC/Gumbel, block verification, apply_sampling_params over the expanded batch, and the spec grammar bitmask stay DEFERRED (M-mtp-3). I5b DRAFTER PREFILL INPUT-PREP LANDED (2026-07-24, CLAIM-SPEC-MTP-I5B): the draft-token input splice this row's I3 note deferred to I5 — vllm::v1::prepare_prefill_inputs + its SpecPrefillInputs output struct shift each request's input_ids left one within its query span, splice the just-sampled next token (num_sampled>0 ? last_sampled[idx_mapping[r]] : next_prefill_tokens[...]) into the freed slot, query_len -= num_rejected, and emit last-token index / query_start_loc / seq_lens + CG padding (mirror speculator.py:469-588, k=1 early-exit :236-238). A HOST routine in a NEW spec_decode-tree TU (no new CUDA kernel; mirrors the DEVICE-NEUTRAL prepare_inputs/combine_sampled_and_draft_tokens family — the DGX runner leaf ports the loop to the Triton kernel at I5d), unit-gated test_prepare_prefill_inputs 7 cases / 27 assertions RED-first, DEFAULT-OFF INERT (nothing calls it until I5d), additive by construction. Row stays ACTIVE — the e2e greedy token gate (M-mtp-1) is owed before DONE |
T1 | vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:43,101-160; rejection_sampler_utils.py:524,564-585,628,828-841,846-849,863-1125; vllm/v1/worker/gpu/model_runner.py:866-898,1065-1077; vllm/v1/worker/gpu/input_batch.py:303-397,408-453; I5b vllm/v1/worker/gpu/spec_decode/autoregressive/speculator.py:469-588,236-238 |
include/vllm/v1/spec_decode/rejection_sampler.h; src/vllm/v1/spec_decode/rejection_sampler.cpp; include/vt/ops.h (kGreedyRejectionSample, vt::GreedyRejectionSample); src/vt/cpu/cpu_sample.cpp (CPU reference); src/vt/cuda/cuda_sample.cu (RejectionRowArgmaxKernel + GreedyRejectAcceptKernel); src/vt/ops.cpp; include/vllm/v1/worker/gpu/prepare_inputs.h + src/vllm/v1/worker/gpu/prepare_inputs.cpp (the expansion); include/vllm/v1/worker/gpu/runner.h + src/vllm/v1/worker/gpu/runner.cpp (step_num_logits, sample_tokens_with_rejection); I5b include/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.h + src/vllm/v1/worker/gpu/spec_decode/autoregressive/prepare_prefill_inputs.cpp |
tests/vllm/v1/spec_decode/test_rejection_sampler.cpp; tests/vllm/v1/worker/test_prepare_inputs.cpp (expansion + no-draft byte-identity); tests/vt/test_cuda_ops.cpp (CUDA==CPU bit-exact at vocab 248320); I5b tests/vllm/v1/spec_decode/test_prepare_prefill_inputs.cpp (7 cases / 27 assertions, RED-first) |
mtp-spec-decode.md §2.4,§5 | ACTIVE |
CLAIM-SPEC-REJECTION-I3, CLAIM-SPEC-MTP-I5B |
SPEC-GDN-SEGMENTS |
GDN speculative metadata and slot-snapshot rollback. I4 LANDED (2026-07-24): the spec/non-spec metadata split with decode→prefill reclassification (the #34845 case), the T>1/IS_SPEC GDN recurrence with per-timestep state snapshots, the conv sliding window advancing by the ACCEPTED count, and the k+1 state-slot allocation. DEFAULT-OFF and INERT (num_spec==0 ⇒ num_spec_decodes==0, no shipped kernel branched — both spec kernels are NEW op ids). ROLLBACK PROVEN bit-exact: for every rejection point j the surviving SSM state and conv window are memcmp-identical to running only the accepted prefix through the shipped vt::GdnDecode/CausalConv1dUpdate, at the real 27B (Hv=48) and 35B (Hv=32) GDN dims on CPU and CUDA. MEASURED state cost: one f32 SSM slot = Hv·Dv·Dk·4B ⇒ 144 MiB/req (27B, 48 layers) / 60 MiB/req (35B, 30 layers) per extra slot; k=1 doubles the GDN SSM state. I5a GDN LAYER ROUTING WIRED (2026-07-24, CLAIM-SPEC-MTP-I5A): GdnBlockPaged's num_spec_decodes>0 branch now routes a PURE-spec batch through vt::CausalConv1dSpecUpdate + vt::GdnSpecDecode (mirror qwen_gdn_linear_attn.py:1344-1357,1455-1475), and the runner per-step upload (StepDevInputs/BuildStepDevInputs + the two decode-graph Refresh copies) now carries I4's six spec device tensors, gated by the extended ValidateGdnAttentionMetadata spec contract. DEFAULT-OFF INERT (num_spec_decodes==0 ⇒ stub uploads + the identical non-spec branch). BIT-EXACT vs the I4 ops applied as a token-sequential decode chain, at the real 27B/35B GDN dims, via GdnBlockPagedForTest (tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp, CPU bit-exact + CUDA on-device); RED-first by a reverted stub (spec recurrence zeroed ⇒ 4/8 fail, maxΔ 1.3-1.6). MIXED spec+non-spec batch refused loudly — lands with I5d's runner loop. Row advances to ACTIVE: the M-mtp-1 e2e greedy token gate (verify/propose runner wiring) is owed before DONE, and SPEC-MTP STAYS GATING |
T1 | vllm/v1/attention/backends/gdn_attn.py:189-326,413-462; fla/ops/fused_sigmoid_gating.py:66-72,103-116,156-166; mamba/ops/causal_conv1d.py:818-1067,1181-1184; qwen_gdn_linear_attn.py:1329-1576; mamba_utils.py:213-234; mamba/abstract.py:55-59 |
include/vllm/v1/attention/backends/gdn_attn.h; src/vllm/v1/attention/backends/gdn_attn.cpp; include/vt/ops.h (kGdnSpecDecode, kCausalConv1dSpecUpdate); src/vt/ops.cpp; src/vt/cpu/cpu_ops.cpp; src/vt/cuda/cuda_gdn.cu; src/vllm/model_executor/models/qwen3_5_common.{h,cpp} (MakeQwen3_5KVCacheSpec); I5a: src/vllm/model_executor/models/qwen3_5.cpp (GdnBlockPaged spec branch, StepDevInputs/BuildStepDevInputs, ValidateGdnAttentionMetadata), src/vllm/model_executor/models/qwen3_5_internal.h (GdnBlockPagedForTest) |
tests/vllm/v1/attention/test_gdn_metadata_builder.cpp (20 cases / 483 assertions incl. the full upstream GDN_BUILD_TEST_CASES + default-off byte-identity); tests/vt/test_ops_gdn.cpp (reject-at-every-j rollback, CPU + CUDA, real dims); tests/vllm/models/test_model_registry.cpp (k+1 slot / widened-conv sizing + num_spec==0 identity); I5a tests/vllm/models/test_qwen3_5_gdn_spec_routing.cpp (spec-routing bit-exact, RED-first) |
mtp-spec-decode.md §3,§5 | ACTIVE |
CLAIM-SPEC-GDN-I4, CLAIM-SPEC-MTP-I5A |
SPEC-DFLASH |
Block-diffusion drafter. READINESS RE-ASSESSED 2026-07-25 (CLAIM-SPEC-DFLASH-READINESS, design-only, DONE) against the LANDED MTP machinery (SPEC-MTP I1..I7). Verdict GREEN, dispatch-ready, NO hardware/oracle/download blocker (spec §0). Refreshed reuse-vs-new map: DFlash gets FREE from landed MTP — the frozen spec-metadata ABI, the greedy rejection sampler (k-general, I3 tested k∈{1,3}), the GDN spec slot path + rollback + mixed spec/non-spec batch (GdnBlockPagedMixedSpec/IndexSelect/IndexCopy, general num_spec), the widened-cache-aware conv ops (I5e), the draft-KV layer pattern (fa_draft), the I5d/I7 runner verify/propose loop, and num_lookahead_tokens=k+1 ALREADY coded (speculative.h:91-108 use_dflash()); EXTENDS the single I5d-pre hidden_tap seam to multi-tap [T,H×taps]; builds NEW the qwen3_dflash drafter, the project's FIRST non-causal in-block attention primitive, context-KV precompute, prepare_dflash_inputs, and the uniform-1+k FULL CG. k>1 verdict: the landed rejection + GDN machinery is MECHANICALLY k-general (no k==1 hardwiring) — DFlash's k=15 blocks need NO mechanism extension, only exercise/validation at scale (D4) + the k+1-slot memory measurement (~2.3 GiB/req 27B GDN state at block-16, the #1 risk, §5). Checkpoint-fit: both z-lab drafts EXIST on HF (27B 1.73 GB / 35B 368 MB bf16, DFlashDraftModel) and FIT the 119 GiB pool trivially (drafts NOT yet on dgx — D0 downloads ≤1.73 GB); the active dgx oracle vllm-oracle-v0.25.0-stage CONSTRUCTS DFlash (registry DFlashDraftModel→qwen3_dflash, speculator dir present) — soft D0 risk = confirm it SERVES DFlash+NVFP4 on sm_121 (non-causal backend; community AEON-7/vllm-dflash container proves the combination runs on GB10). W-plan D0-D6 in the spec. D0+D1 LANDED 2026-07-26 (CLAIM-DFLASH-D0D1) on the ADVANCED pin 555967922/vLLM 0.26.0.dev0 — SPEC-DFLASH → ACTIVE. D0 UNBLOCKED (vllm#40898 resolved under VLLM_USE_V2_MODEL_RUNNER=1): the mixed-attn z-lab 27B draft CONSTRUCTS + the drafter is ALIVE (acceptance 2.21/8.80/4.75/4.57 > 1, num_spec=16, flashinfer-native fp8-KV, goldens committed); gate FORM measured STRICT MODE-MATCHED (vLLM-ON run-deterministic K>=3 but != vLLM-OFF — the k=16 block verify diverges at bf16 near-ties, so NOT the MTP three-way identity). D1 DF-AUX-TAPS DONE: Qwen3_5AuxTaps + ModelForwardInput::aux_tap route to Qwen3_5{,Dense}Model::ForwardDeviceMultiTap capturing (hidden+res) at target_layer_ids into [T,H×taps] (eagle3 _maybe_add_hidden_state, aux key L+1); config-gated byte-identical off. Unit gate 598 assertions (independent truncated-model reference, RED-first reversed-concat 384 fail); CUDA 697/697 + compute-sanitizer 0; INERTNESS PROVEN — 27B MTP e2e 9/9 + 27B text SACRED 235/235 byte-identical on the new oracle. D2 DF-DRAFT-MODEL CODE LANDED + CPU-GATED 2026-07-26 (CLAIM-DFLASH-D2, kernel row KERNEL-ATTN-DFLASH-BLOCK): the qwen3_dflash draft model (plain 5-layer Qwen3-dense reusing dense_attn_block.h ops), the project's FIRST non-causal / bidirectional attention primitive vt::DFlashBlockAttention (a SEPARATE op — causal kAttention/kPagedAttention byte-identical), the fc aux-combine, mask-embed, per-layer SWA/full resolution, and the z-lab loader. CPU gate GREEN (op 12/12 incl. RED non-causal; model forward 95/95 incl. RED full-layer-causal-flip + block isolation + fc RED); existing causal test_ops_attention 9/9 + test_qwen3_forward 1028 UNCHANGED. D2 GPU PROMOTION GREEN on dgx (CLAIM-DFLASH-D2): CUDA -Werror clean, CUDA==CPU 198412/198412 + compute-sanitizer 0, draft-forward parity vs the REAL vLLM draft (fc rel-L2 0.46%, hidden ≤1.3%, 11 STRICT + 5 near-tie ids), 27B SACRED 235/235 + MTP 9/9 byte-identical — D2 DONE. D3 DF-DRAFT-KV-PREP DONE 2026-07-26 (CLAIM-DFLASH-D3): PrecomputeContextKV + PrepareDflashInputs + ForwardBlockLogitsWithContext (reuse the UNCHANGED D2 kernel via [context;block]); GPU numeric-parity test_qwen3_dflash_kvprep_parity 61/61 (prepare INTEGER bit-exact vs vLLM's Triton kernel, context-KV K/V rel-L2 0.31%/0.26%, 13 STRICT + 3 near-tie = 16/16), CPU 114/114 RED-proven, inertness 235/235 + 9/9 + D2 37/37 byte-identical. D4 DF-ENGINE-INTEGRATION propose brick + dflash config-select CODE LANDED + CPU-GATED 2026-07-26 (CLAIM-DFLASH-D4D5): DflashProposeBlock/SampleDflashBlockDrafts (the non-autoregressive whole-block propose composing D3 ForwardBlockLogitsWithContext + greedy per-mask argmax, anchor not sampled, dflash/speculator.py:300-413) + ParseSpeculativeConfigJson/ResolveDflash accept method:"dflash". CPU gate test_dflash_propose 5/19 GREEN (RED-first anchor-read fails 4/5; brick composes forward+sampler; empty-ctx degenerates to D2; config lookahead k+1). Additive + config-gated ⇒ MTP + non-spec byte-identical BY CONSTRUCTION (git diff --stat = new speculator TU + config accept-list + CMake + test, NO runner/model/loader/scheduler edit). D5 DF-ENGINE-INTEGRATION runner-loop LANDED + e2e RUNS on dgx 2026-07-26 (CLAIM-DFLASH-D5): full verify/propose loop wired — loader loads the SEPARATE z-lab draft (LoadDflashDraft, host bf16 + target-SHARED bf16 embed/lm_head) via a --speculative-config model key + ResolveSpecConfig dflash branch + runner.set_dflash_draft; the verify forward captures the D1 multi-tap (aux_tap→ForwardDeviceMultiTap) instead of the MTP single tap; propose_drafts_dflash ACCUMULATES the per-request combined-feature context (CombineAuxFeatures(aux_tap)) across steps and honors the num_rejected rollback by appending only the (T_req−num_rejected) accepted-prefix features, then runs DflashProposeBlock (k=16 GDN-spec exercised first time). e2e (test_qwen27_dflash_spec_decode, 4 prompts×32 tok, our-DFlash-ON vs the committed vLLM-DFlash-ON golden): 2/4 STRICT token-exact (fibonacci, three-laws) + acceptance ~ vLLM on ALL 4 (accepted 19/39/29/25 vs golden 17/39/30/25, deltas +2/0/−1/0 — the MANDATORY dead-drafter-trap condition MET). The 2 divergences (France tok11 2972↔11751, 1723 tok12 567↔488) are SINGLE bf16 near-tie flips (1723 RE-CONVERGES after one token = proven near-tie; France cascades from one flip) — the ratified near-tie ROOT the D0 gate-form anticipated, rooted in the D3-documented inline bf16 context-KV recompute envelope (~0.3-1.3% rel-L2), NOT a wiring bug (proven by the 2 exact prompts + near-exact acceptance + a non-trivial shared prefix). Inertness GREEN on this build: SACRED test_qwen27_paged_engine 235/235 + MTP test_qwen27_spec_decode 9/9 byte-identical; CUDA -Werror clean; NO new CUDA kernel (host orchestration reusing D1/D2/D3-sanitized ops). NOT a clean strict-4/4 pass; STRICT 4/4 token-identity + the speed A/B = D6 (the persistent paged draft-KV bit-matching vLLM's fused context-KV projections + the uniform-1+k FULL CG). Row STAYS ACTIVE (correctness at the ratified near-tie envelope; D6 remains) D6 2026-07-27 (CLAIM-DFLASH-D6) — c1 SPEED A/B DONE + STRICT-irreducibility RCA + CG feasibility (records-only, NO source code): (1) c1 speed A/B (examples/vllm-bench at 361189a7, 8 prose+code prompts×256 tok greedy c1, 2 reps): our DFlash-ON = 2.50x TPOT (40.4 vs 101.2 ms) / 2.48x output-tput (24.4 vs 9.86 tok/s) over our OFF, acceptance 0.22 (3.56/16), rep-stable <1.5%; benchmark_binding=true. vs vLLM-DFlash-ON graphed (same workload): vLLM-DFlash-ON graphed = 28.5 tok/s / 35.1 ms TPOT / acceptance_len 4.30 (same 8 prompts, VLLM_USE_V2_MODEL_RUNNER=1, mm-off, gpu_util 0.30), so OURS IS ~14% BELOW vLLM-DFlash-ON on output throughput (24.4 vs 28.5 tok/s) - both ~on-par at spec-OFF (9.86 vs 9.83 tok/s), but vLLM extracts a larger DFlash speedup (2.90x vs our 2.47x) because its draft step is fully device-resident + CUDA-graphed (ours host-orchestrates 13 downloads/step) + slightly higher acceptance (~4.3 vs ~3.6 draft tokens/step). The DONE speed bar (ours >= vLLM) is NOT met; closing it = the device-resident draft rewrite + FULL CG (D6 part 2). (2) STRICT-4/4 proven bf16-IRREDUCIBLE — the draft KV cache is bf16 not fp8 (torch_utils.py:398 auto→model dtype; the D0 "fp8-KV" was the backend name, not the KV storage dtype), the D3 golden already compares pre-storage bf16 (residual K 0.31%/V 0.26% = sub-ULP kernel noise), and a fused multi-layer KV GEMM is per-element invariant to our per-layer GEMMs ⇒ bit-exact needs vLLM's exact kernels ⇒ the ratified near-tie gate is the FINAL correctness form (no fused-KV code landed). (3) FULL CG BLOCKED on a device-resident draft-path rewrite (the D5 path does 13 device→host downloads/step + host [context;block] interleaving) — the remaining throughput-parity increment (the perf form of persistent-paged-KV + the graph). Inertness by construction (the gated binary is the D5 binary; SACRED 235/235 + MTP 9/9 stand). Evidence tool scripts/spec/vllm_dflash_timing.py. **D7 2026-07-27 (CLAIM-DFLASH-D7) — within-step draft forward made DEVICE-RESIDENT (source-owning): PrecomputeContextKVDevice keeps per-layer K/V on device; ForwardBlockLogitsWithContext builds [context;block] with vt::IndexCopy/IndexSelect (removes ~30 D→H Downloads/step). BIT-IDENTICAL (identity bf16↔f32 round-trips replaced) — e2e test_qwen27_dflash_spec_decode 27/27 SAME tokens (2/4 STRICT + 2/4 near-tie, acceptance 19/39/29/25), SACRED 235/235 + MTP 9/9, CUDA -Werror clean, compute-sanitizer 0 (198412). But the direct old-vs-new A/B = +2.0% output-tput (IN-NOISE) ⇒ D6's "downloads = the ~14% gap" REFUTED by measurement; ours 19.68 tok/s STILL ~33% BELOW vLLM-DFlash-ON 29.2 tok/s (reconstructed 8-prompt set, more prose-heavy); OFF parity our 9.97 ≥ vLLM 9.66. Residual re-attributed: acceptance (ours 2.49 vs vLLM ~3.13 accepted draft-tok/step, bf16-irreducible) + per-step context-KV RECOMPUTE (O(context²), needs the cross-step persistent paged draft-KV store) + eager-vs-graphed. SPEED BAR NOT met; SPEC-DFLASH stays ACTIVE; next = persistent paged draft-KV store → then FULL CG. D9 2026-07-27 (CLAIM-DFLASH-D9) — PERSISTENT PAGED DRAFT-KV LANDED (bit-identical, +22.7% throughput, 0.69×→0.917×); D8 acceptance-ceiling REFUTED; residual = FULL CG ONLY: qwen3_dflash.cpp AppendContextKVHost (project ONLY newly-accepted rows → per-layer bf16 K/V, append to PrecomputedContextKV) + ForwardBlockLogitsWithPrecomputedKV (upload the persistent store, NO re-projection) share the core ForwardWithCtxKVDev with the old recompute; runner.cpp::propose_drafts_dflash swaps the O(context²) per-step recompute (dflash_ctx_feats_) for an append-only per-request dflash_kv_store_ (rollback=don't-append). NO new CUDA kernel; config-gated. BIT-IDENTICAL: CPU test_dflash_propose two new D9 cases = exact float equality vs full recompute; GPU e2e test_qwen27_dflash_spec_decode 27/27 SAME tokens (acceptance 19/39/29/25, same divergences France@11/17×23@12); SACRED 235/235 + MTP 9/9 byte-identical; CUDA -Werror clean. A/B (c1, 8 prose+code×256 tok input-len 512, 2 reps <0.1%, benchmark_binding=true): ours-ON 25.75 tok/s (was D8 20.99, +22.7%) / 38.40 ms TPOT / acc 3.68/step vs vLLM-ON graphed 28.09 / 35.60 / acc 3.31 = 0.917× (~8% below, was 0.69×). Part 1 same-trajectory: on the 2 token-identical-trajectory prompts ours per-step acceptance == vLLM's EXACTLY (fibonacci 7.80/7.80, three-laws 3.571/3.571, ratio 1.00) AND on the A/B ours acceptance (3.68) is HIGHER than vLLM's (3.31) ⇒ D8's 0.80–0.85× "bf16 acceptance ceiling" is a trajectory-divergence CONFOUND, REFUTED. Residual (~8%) = eager-vs-graphed ONLY (ours ON/OFF 2.60× vs vLLM 2.91×, OFF at parity, recompute eliminated, acceptance higher) — NOT an irreducible ceiling; the FULL uniform-(1+k) CG (device paged-KV store + paged attn, new-CUDA multi-file) is the SOLE un-landed increment. SPEC-DFLASH stays ACTIVE (speed not yet ≥ vLLM; residual isolated to FULL CG). D12 2026-07-27 (CLAIM-DFLASH-D12) — A-wire + Part B LANDED + GPU-gated; Part C (capture) remaining; 0.917×: A-wire makes the D11 Part-A device store the PRODUCTION path (runner.{h,cpp} dflash_kv_store_→shared_ptr<DflashDeviceKVStore>, MakeDeviceKVStore/AppendContextKVDevice/ForwardBlockLogitsWithDeviceKV; GPU-gated e2e test_qwen27_dflash_spec_decode 27/27 all-exact acceptance 19/39/29/25 + SACRED 235/235 + MTP 9/9 byte-identical, -Werror clean). Part B adds vt::DFlashPagedBlockAttention (OpId::kDFlashPagedBlockAttention), the capture-safe paged kernel with EVERY metadata input a persistent DEVICE tensor and NO function-local host cu_seqlens upload (fixes the cuda_ops.cu:1277-1280 capture-UAF class), gated CPU==CUDA + cross-check vs materialized DFlashBlockAttention test_ops_dflash_paged_block_attn 795648/795648 + compute-sanitizer 0. Speed 0.917× (A-wire eager + Part B not yet wired into the forward); benchmark_binding=false. Part C (static-shape capture + device mask-scatter + BeginCapture/replay + the ≥vLLM c1 A/B) is the SOLE remaining piece; if ours-ON-graphed ≥ vLLM-ON → SPEC-DFLASH DONE. Stays ACTIVE. D13 2026-07-27 (CLAIM-DFLASH-D13) — Part C LANDED + GPU-GATED; capture-correctness PROVEN; c1 throughput NEAR-PARITY (ours 0.978x, ~2% below vLLM); gap CLOSED 0.917x→0.978x; STAYS ACTIVE (≥vLLM bar not yet met): single-file additive change (qwen3_dflash.cpp +368/-58). (C.1) DflashDeviceKVStore → fixed-capacity PAGED cache (per-layer pool [max_pages,16,Hkv,Dh] + identity block_table + seq_lens; append = vt::IndexCopy scatter at slot==abs-pos, bit-identical to the D9/D11 store). (C.2) ForwardPagedBody runs the (1+k) block through the D12 vt::DFlashPagedBlockAttention reading the paged store (no [context;block] materialization, no function-local host uploads); runner P==1 propose routes through it, P>1 bit-identical materialized fallback. (C.3) per-request CUDA GRAPH over the paged draft step (warm-in-step repopulates the shared pool free-list right before BeginCapture — the fix for a cudaMalloc-in-capture Get miss from the intervening 27B target forward — then BeginCapture → ForwardPagedBody → EndCaptureGraph, replay with growing context entering only via in-place seq_lens). Capture-correctness (MANDATORY): test_qwen27_dflash_spec_decode 27/27 with the graph (VT_DFLASH_GRAPH=1) BIT-IDENTICAL to eager (=0) — same divergence tokens (France@11 got[…2972…], 17×23@12 got[…567…]), same acceptance 19/39/29/25 as D5/D7/D9/D12; graph ENGAGED (5 captures C=2048/5/4/15/6, 32+ replays); the token-diff is the capture-safety proof ([[cudagraph-capture-bakes-stack-addresses]]). c1 A/B (one flock series, cold rep discarded, 8 prompts×256 tok): our OFF 10.24 / our ON eager-paged 28.65 (28.69,28.61) / our ON GRAPHED 28.70 (28.70,28.70), TPOT 34.40 / vLLM-ON graphed steady-state 29.35 (tight 3-rep 29.33/29.37/29.33, TPOT 34.07, acc_len 4.44); D9's 28.09 was a colder cross-session outlier — NEAR-PARITY: ours 0.978× (~2% below) on the rigorous same-session band (across sessions ours 28.70 falls inside vLLM's observed 28.09–29.37 range). ON/OFF 2.80× (vLLM ~2.98×), our OFF ≥ vLLM OFF. Per the acceptance rule ("below on any axis = an open gap; near-parity is NOT met"), the ≥vLLM bar is NOT met; STAYS ACTIVE. Residual (data-grounded): NOT acceptance (ours realized ~3.68 accepted draft-tok/step > vLLM's 3.44) and NOT launch/graph (both graphed, CG neutral) — per-step COMPUTE (~2% slower target-step); next lever = nsys both draft steps (--cuda-graph-trace=node), no premature ceiling. ATTRIBUTION (supersedes D9): the CUDA graph is perf-NEUTRAL (+0.3%); the ACTUAL lever was the paged context read (C.1/C.2) removing the D9/D12 per-layer [context;block] IndexCopy materialization of the whole growing context (25.75 D9 → 28.65 eager-paged, +11%) — the roadmap's "the full CG closes the gap" premise is corrected by measurement. Inertness VERIFIED on the capture binary: SACRED 235/235 + MTP 9/9 byte-identical, CUDA -Werror clean, no new kernel (D12 paged kernel already memcheck-0 795648), check-device-leakage not increased (paged path REMOVES the materialized-buffer allocs + host uploads). benchmark_binding=true. Correctness-complete (ratified near-tie); throughput NEAR-PARITY (0.978×, ~2% residual) ⇒ STAYS ACTIVE (the capture-correctness gate is MET; the ≥vLLM speed bar is the sole remaining item, a ~2% per-step-compute residual for an nsys). Anchors: src/vllm/model_executor/models/qwen3_dflash.cpp (DflashDeviceKVStore paged store, ForwardPagedBody, the per-request graph in ForwardBlockLogitsWithDeviceKV). D14 2026-07-27 (CLAIM-DFLASH-D14) — SPEED GATE MET → SPEC-DFLASH DONE: an nsys (--cuda-graph-trace=node) of the graphed spec-on step attributed the D13 ~2% residual to the from-scratch DFlashPagedBlockAttentionKernel draft attention (242.9 ms = 1.8% of GPU time, median cutlass_80_wmma for the draft bf16 GEMMs, so the GEMMs were NOT the gap). Ported it to a WARP-scoped online-softmax variant DFlashPagedBlockAttentionWarpKernel (mirrors the shipped AttentionWarpKernel: one warp per (block-query,head), __shfl_xor butterfly reduction, register accumulator, NO __syncthreads storm; SAME paged/block combined-index read + causal/SWA mask + GQA; default ON, VT_DFLASH_ATTN_BLOCK=1 keeps the bit-identical D12/D13 block kernel for A/B). Draft attn 242.9 → 77.9 ms (3.1x); our-ON c1 28.60 → 29.32 tok/s (+2.5%). FINAL same-session 3-rep A/B (8 prompts×256 tok, cold leg discarded): our-ON graphed 29.42/29.27/29.32 (med 29.32) vs vLLM-ON graphed 29.240/29.247/29.233 (med 29.240) — our WORST rep (29.27) > vLLM's BEST (29.247), NON-OVERLAPPING bands, 1.003× ⇒ the ≥vLLM speed gate is MET. Correctness UNCHANGED (output is exact by spec-decode construction — the target verify is untouched, only which draft proposals are accepted can shift): e2e test_qwen27_dflash_spec_decode 27/27 with graph==eager BIT-IDENTICAL, acceptance 19/39/29/25 unchanged (draft accepted 1629 identical warp-vs-block across the whole A/B set), 2/4 STRICT (France@11, 17×23@12 unchanged); CUDA==CPU test_ops_dflash_paged_block_attn 795648/795648 (warp within the f32 1e-4 / bf16 3e-2 envelope) + compute-sanitizer 0. Inertness SACRED 235/235 + MTP 9/9 byte-identical; CUDA -Werror clean; check-device-leakage not increased. benchmark_binding=true. Block-diffusion drafting is now correctness-complete (ratified near-tie) AND at/above vLLM throughput — this was the roadmap's FINAL open speed item. Anchors: src/vt/cuda/cuda_ops.cu (DFlashPagedBlockAttentionWarpKernel + UseDflashAttnBlockKernel; the D12 block kernel retained as the VT_DFLASH_ATTN_BLOCK=1 reference). |
T1 | vllm/v1/worker/gpu/spec_decode/dflash/speculator.py; vllm/model_executor/models/qwen3_dflash.py; vllm/model_executor/models/interfaces.py:1382 (aux value); eagle3_utils.py:41-56 (+1 shift) |
include/vllm/model_executor/models/qwen3_5.h (Qwen3_5AuxTaps, ForwardDeviceMultiTap); qwen3_5_dense.h; model_registry.h (aux_tap); src/vllm/model_executor/models/qwen3_5.cpp (MaybeCaptureAuxTap/ValidateAuxTapLayerIds/ForwardDeviceMultiTap); qwen3_5_moe.cpp+qwen3_5_dense.cpp (routing); D2/D3 include/vllm/model_executor/models/qwen3_dflash.h + src/vllm/model_executor/models/qwen3_dflash{,_weights}.cpp; D4 include/vllm/v1/worker/gpu/spec_decode/dflash/speculator.h + src/vllm/v1/worker/gpu/spec_decode/dflash/speculator.cpp (DflashProposeBlock/SampleDflashBlockDrafts); D5 src/vllm/entrypoints/model_loader.cpp (LoadDflashDraft/DflashDraft) + include/vllm/entrypoints/model_loader.h; D5 src/vllm/v1/worker/gpu/runner.cpp (set_dflash_draft/propose_drafts_dflash/aux-tap capture) + include/vllm/v1/worker/gpu/runner.h; src/vllm/config/speculative.cpp + include/vllm/config/speculative.h (ResolveDflash + dflash/model parse); D14 warp kernel cuda_ops.cu |
tests/vllm/models/test_qwen27_paged_forward.cpp (multi-tap 598); tests/vt/test_ops_dflash_block_attn.cpp; tests/vllm/models/test_qwen3_dflash_forward.cpp; tests/vllm/v1/spec_decode/test_dflash_kvprep.cpp; tests/parity/test_qwen3_dflash_{draft,kvprep}_parity.cpp; D4 tests/vllm/v1/spec_decode/test_dflash_propose.cpp (5/19, RED-first); D5 tests/parity/test_qwen27_dflash_spec_decode.cpp (e2e 27/27, 2/4 strict + acceptance~vLLM); scripts/spec/d{0,2,3}_dflash_*.py; tests/parity/goldens/dflash_27b{,_draft,_kvprep}/; D6 scripts/spec/vllm_dflash_timing.py (vLLM-DFlash c1 timing); D7 device-resident src/vllm/model_executor/models/qwen3_dflash.cpp (PrecomputeContextKVDevice + ForwardBlockLogitsWithContext via vt::IndexCopy/IndexSelect); D9 persistent paged draft-KV qwen3_dflash.{h,cpp} (AppendContextKVHost/ForwardBlockLogitsWithPrecomputedKV/ForwardWithCtxKVDev/PrecomputedContextKV) + runner.{h,cpp} (dflash_kv_store_/propose_drafts_dflash) + tests/vllm/v1/spec_decode/test_dflash_propose.cpp (2 D9 bit-identity cases); D12 A-wire runner.{h,cpp} (device store as production path) + D12 Part B include/vt/ops.h/src/vt/ops.cpp/src/vt/cpu/cpu_ops.cpp/src/vt/cuda/cuda_ops.cu (kDFlashPagedBlockAttention) + tests/vt/test_ops_dflash_paged_block_attn.cpp (CPU==CUDA + cross-check, 795648/795648 + sanitizer-0); D13 src/vllm/model_executor/models/qwen3_dflash.cpp (fixed-capacity paged DflashDeviceKVStore + ForwardPagedBody + the per-request draft-step CUDA graph in ForwardBlockLogitsWithDeviceKV); D14 test_ops_dflash_paged_block_attn + ledger |
dflash-spec-decode.md | DONE |
489a7544 |
SPEC-DSPARK |
DSpark semi-autoregressive block drafter for DeepSeek-V4 and Qwen3, including native and Speculators checkpoint layouts, anchor-vs-bonus-token semantics, reduced/heterogeneous vocabulary mapping, sequential Markov sampling, noncausal draft attention, rejection/metrics and full-CUDA-graph compatibility; user-promoted scope at the v0.25.0 audit | T1 | vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:3-99; vllm/model_executor/models/qwen3_dspark.py:3-172; vllm/models/deepseek_v4/nvidia/dspark.py:56-457; tests/v1/e2e/spec_decode/test_spec_decode.py:1415-1469 @ 702f481 |
- | - | planned: specs/dspark-spec-decode.md |
INVENTORIED |
- |
SPEC-TLI |
Tokenizer-agnostic speculative decoding across heterogeneous draft/target vocabularies: shared-token mapping, target↔draft ID translation, constrained draft logits and greedy-only validation | T1 | vllm/config/speculative.py:145-149,1173-1203; vllm/v1/spec_decode/vocab_mapping.py:68-160; vllm/v1/spec_decode/draft_model.py:34-58; vllm/v1/spec_decode/llm_base_proposer.py:432-495,688-691,831-837; tests/v1/spec_decode/test_vocab_mapping.py:1-50 @ 702f481 |
- | - | planned: specs/tli-spec-decode.md |
INVENTORIED |
- |
SPEC-NGRAM |
Draft-FREE n-gram proposer. DONE 2026-07-27 (CLAIM-ROADMAP-D3): 1:1 port of ngram_proposer.py (KMP-LPS suffix-ngram matcher + batch propose) wired as a third --speculative-config method reusing the LANDED MTP/DFlash verify/reject/take_draft_token_ids loop (no draft model / hidden tap / draft KV; GDN spec verify reused via MakeQwen3_5KVCacheSpec(num_spec>0)). 27B gate 5/5 STRICT our-ngram-ON == vLLM-ngram-ON + 180/180 drafts accepted; unit 19/19; spec-OFF byte-identical; host-side, no new kernel, -Werror clean |
T2 | vllm/v1/spec_decode/ngram_proposer.py:184-276,128-180; vllm/config/speculative.py:734-762,1224-1234; tests/v1/spec_decode/test_ngram.py @ 555967922 |
src/vllm/v1/spec_decode/ngram_proposer.{h,cpp}; include/vllm/config/speculative.h (ResolveNgram/use_ngram); src/vllm/config/speculative.cpp; src/vllm/entrypoints/model_loader.cpp (ResolveSpecConfig); src/vllm/v1/worker/gpu/runner.cpp (propose_drafts_ngram) |
tests/vllm/v1/spec_decode/test_ngram_proposer.cpp (19/19); tests/parity/test_qwen27_ngram_spec_decode.cpp (5/5 STRICT, 180/180 accepted, dgx); golden tests/parity/goldens/ngram_27b/ngram_27b_spec_on.json + scripts/spec/ngram_27b_golden.py; ledger parity-ledger.md 2026-07-27 |
specs/spec-decode-breadth-d3.md | ACTIVE |
CLAIM-ROADMAP-D3 |
SPEC-EAGLE3 |
EAGLE3 proposer and draft model. SCOPED — reachable-blocked 2026-07-27 (CLAIM-ROADMAP-D3): port designed (reuse DFlash D5 separate-draft loader + D1 aux multi-tap + verify/reject loop; config-select method="eagle3"). BLOCKER (W0 RUN-verify, no fabrication): no ungated oracle-runnable EAGLE3 draft ARCH/checkpoint for a Qwen3.6 gate model at pin 555967922 — registry.py:572-648 has Qwen3_5MTP but no Eagle3Qwen3_5*; dgx cache has zero eagle checkpoints; z-lab published DFlash not EAGLE3; our spec path is Qwen3.6-only. Mirrors the Command-R HF-gate honest-blocked pattern |
T2 | vllm/v1/spec_decode/eagle.py; vllm/model_executor/models/registry.py:612-648; vllm/config/speculative.py:869-885 @ 555967922 |
- (port scoped, not implemented) | - (blocked: no checkpoint) | specs/spec-decode-breadth-d3.md | BLOCKED |
- |
Note: grammar-bitmask application under speculative decode (the multi-row
bitmask, one row per draft token) is deferred — see
porting-inventory.md §6 — and is in scope for neither
the TOOLS-STRUCTURED-CORE row nor the SPEC-* rows above until a spike
claims it.
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
SPEC-DRAFT-MODEL |
Generic model-agnostic SEPARATE draft-model proposer (method="draft_model"): a full smaller standalone LM runs K autoregressive greedy steps to propose K draft tokens; the target verifies in one forward and the longest-accepted-prefix is emitted. Distinct from MTP/EAGLE/DFlash — pass_hidden_states_to_model=False, shares NEITHER embeddings NOR lm_head with the target. W0 spike + W1 CPU brick LANDED 2026-07-29 (CLAIM-SPEC-DRAFT-MEDUSA, NOT pushed): the greedy k-step autoregressive propose (DraftModelProposeGreedy/Batch) over a DraftLogitsFn next-token oracle, reusing the LANDED SPEC-REJECTION verify/accept UNCHANGED (only the proposer is net-new, mirror of the SPEC-NGRAM shape). Unit-gated RED-first: propose->verify->accept equivalence (accepted tokens == the target's own greedy run, every draft/target (dis)agreement pattern) + full-acceptance on a matching draft (num_sampled==k+1) + the RED witness that full acceptance DEPENDS on the autoregressive feed-back (5/6 fail with feed-back dropped). ParseSpeculativeConfigJson accepts "draft_model" (requires model + num_speculative_tokens). Additive + default-inert (no runner construction; engine byte-identical with no SpeculativeConfig). Clean CPU -Werror. RESIDUAL (W3, DGX-offline): the real draft-model forward behind the oracle (paged KV + CUDA-graph) + e2e greedy our-ON==vLLM-ON token-exact gate + throughput speed gate. |
T2 | vllm/v1/spec_decode/draft_model.py:19 (pass_hidden_states :29, no shared embed/lm_head :108-115); propose vllm/v1/spec_decode/llm_base_proposer.py:502-767 (_greedy_sample :428-438, set_inputs_first_pass :838-851, K-1 feed-back :682-761); config vllm/config/speculative.py:684,692-701,1195; runner vllm/v1/worker/gpu_model_runner.py:604-609; e2e tests/v1/e2e/spec_decode/test_spec_decode.py:500-561 @ 555967922 |
include/vllm/v1/spec_decode/draft_model_proposer.h; src/vllm/v1/spec_decode/draft_model_proposer.cpp; src/vllm/config/speculative.cpp (draft_model accept); reuses src/vllm/v1/spec_decode/rejection_sampler.{h,cpp} |
tests/vllm/v1/spec_decode/test_draft_model_proposer.cpp (6/6, 41 assertions, CPU; RED-first 5/6 fail with feed-back dropped); ledger parity-ledger.md 2026-07-29 |
specs/draft-model-medusa-spec.md | ACTIVE |
CLAIM-SPEC-DRAFT-MEDUSA |
SPEC-MEDUSA |
Medusa multi-head speculator (method="medusa"): the target carries N extra Medusa LM heads, each predicting ONE future position from the SAME target hidden state in a single (non-autoregressive) pass; draft_tokens = stack([argmax(head_logits)]) -> [batch, num_heads], num_speculative_tokens == num_heads. Verify/accept is the SAME SPEC-REJECTION loop (linear, not tree, at this pin). W0 spike ONLY (CLAIM-SPEC-DRAFT-MEDUSA, 2026-07-29): proposer scoped in specs/draft-model-medusa-spec.md; deferred to W2 because its multi-head target-tap propose needs the target model's Medusa heads (a model change) a pure host brick cannot meaningfully stand up. No code yet. |
T2 | vllm/v1/spec_decode/medusa.py:18 (propose :40-58: model(hidden)->per-head compute_logits->stacked argmax); config vllm/config/speculative.py:822,888-889; runner vllm/v1/worker/gpu_model_runner.py:642-645 @ 555967922 |
- (spike only, not implemented) | - (W2) | specs/draft-model-medusa-spec.md | SPIKE |
CLAIM-SPEC-DRAFT-MEDUSA |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
SERVE-OAI-BASIC |
Chat/completions endpoints with SSE transport | T0 | vllm/entrypoints/openai/completion/api_router.py:34; vllm/entrypoints/openai/chat_completion/api_router.py:40; tests/entrypoints/openai/completion/test_completion.py:50,259 |
src/vllm/entrypoints/openai/api_server.cpp:60,108,183; src/vllm/entrypoints/openai/serving_completion.cpp:22,116; src/vllm/entrypoints/openai/serving_chat.cpp:231,380 |
tests/vllm/entrypoints/openai/test_api_server.cpp:351,381,403,449,483,617; tests/vllm/entrypoints/openai/test_conformance.cpp:469,502,614,641 |
planned: specs/chat-completions-endpoints.md |
ANCHOR-BACKFILL |
- |
SERVE-DISCOVERY-HEALTH |
Models, health, and version endpoints | T0 | vllm/entrypoints/openai/models/api_router.py:20; vllm/entrypoints/serve/instrumentator/health.py:22; vllm/entrypoints/serve/instrumentator/basic.py:53; tests/entrypoints/openai/models/test_models.py:48 |
src/vllm/entrypoints/openai/api_server.cpp:149,158,167,215; src/vllm/entrypoints/openai/serving_models.cpp:22,50 |
tests/vllm/entrypoints/openai/test_api_server.cpp:434; tests/vllm/entrypoints/openai/test_conformance.cpp:953,976,990 |
planned: specs/models-health-version.md |
PARTIAL |
- |
SERVE-METRICS |
Prometheus /metrics with vLLM names. LANDED + CPU-GATED 2026-07-27 (CLAIM-ROADMAP-C8, NOT pushed): self-contained Prometheus registry (PromRegistry, text-format-0.0.4 exposition: counter _total, histogram _bucket{le}/_sum/_count, Info {labels} 1.0) + the ALWAYS-ON vLLM metric catalog (PrometheusStatLogger) registered 1:1 (names/help/type/buckets, {model_name,engine} labels) + record(SchedulerStats,IterationStats) + GET /metrics opt-in route. Gated by the vLLM scrape spec EXPECTED_METRICS_V1 (substring presence, RED-first). LIVE PER-STEP WIRING LANDED 2026-07-27 (CLAIM-ROADMAP-C8-METRICS-WIRE, NOT pushed): the /metrics endpoint now serves LIVE values, not the primed schema. EngineCoreOutputs carries scheduler_stats (filled by new Scheduler::make_stats(), scheduler.py:2399-2436 — running/waiting/kv-usage + the per-step prefix-cache delta stashed by schedule()) + a stamped timestamp; OutputProcessor::process_outputs builds IterationStats (token counts, TTFT/ITL samples, finished-request breakdowns via RequestState timing — stats.py:377-475); the sync LLMEngine::step() folds both into the attached logger's Record() guarded by outputs>0 (llm_engine.py:308-329). Additive + opt-in: null logger ⇒ no IterationStats, process_outputs byte-identical no-stats path, greedy token stream untouched. /metrics PRODUCTION WIRING NAMED-BLOCKED 2026-07-28 (CLAIM-C8-SERVE-PROD-WIRING): the shipped server drives AsyncLLM (main.cpp: loaded->async_engine()), whose output handler records NO stats to any logger (async stats deferred) and neither LoadedEngine nor AsyncLLM constructs/exposes one — so /metrics is deliberately NOT wired in ConfigureUtilityEndpoints (attaching a never-Recorded logger would serve a permanently-zero exposition). Missing accessor: LoadedEngine::stat_logger() + a Record() call site in AsyncLLM::RunOutputHandler. RESIDUAL: that AsyncLLM stat-logger wiring; config-gated families (spec-decode/kv-connector/mm/LoRA); per-request queue/prefill/inference timing + preemption counter (EngineCoreEvents deferred → SERVE-RESPONSE-METRICS) |
T0/T1 | vllm/entrypoints/serve/instrumentator/metrics.py:52-82; vllm/v1/metrics/loggers.py:480-1060,1100-1257,1284-1305; vllm/v1/metrics/stats.py:186-259,377-475; vllm/v1/core/sched/scheduler.py:2399-2436; vllm/v1/engine/llm_engine.py:308-329; scrape spec tests/entrypoints/serve/instrumentator/test_metrics.py:182-228 |
registry include/vllm/v1/metrics/prometheus.h, src/vllm/v1/metrics/prometheus.cpp:13,189; catalog+record include/vllm/v1/metrics/loggers.h, src/vllm/v1/metrics/loggers.cpp:10,60,208; stats structs + MonotonicSeconds include/vllm/v1/metrics/stats.h:56,160,175,194; make_stats include/vllm/v1/core/sched/scheduler.h, src/vllm/v1/core/sched/scheduler.cpp (+prefix-delta stash in schedule()); scheduler_stats/timestamp on EngineCoreOutputs include/vllm/v1/engine/types.h, stamped src/vllm/v1/engine/core.cpp; IterationStats build src/vllm/v1/engine/output_processor.cpp (+RequestState timing include/vllm/v1/engine/output_processor.h); step-site Record src/vllm/v1/engine/llm_engine.cpp:99, setter include/vllm/v1/engine/llm_engine.h; endpoint src/vllm/entrypoints/openai/api_server.cpp:251 (handle_metrics), route :488 |
tests/vllm/v1/test_prometheus_metrics.cpp 4/4 (81 assertions: EXPECTED_METRICS_V1 substring gate RED-first, label schema, TYPE lines, bucket schedules, record() folding); live-wiring behavioural gate tests/vllm/v1/test_llm_engine.cpp case 6 (44 assertions, RED-first: 14 flip 0→correct when Record disabled) — running/waiting gauges track the batch, prompt/generation counters == exact token counts, request_success counts finished reqs, TTFT/ITL/e2e/TPOT/iteration histograms observe the right sample counts; endpoint tests/vllm/entrypoints/openai/test_api_server.cpp:921 |
prometheus-metrics.md | ACTIVE |
CLAIM-ROADMAP-C8 |
SERVE-RESPONSE-METRICS |
Per-request timing surface: the QUEUED/SCHEDULED/PREEMPTED EngineCoreEvents the scheduler emits + the per-request queue/prefill/inference timing intervals + preemption counter they feed. EngineCoreEvents + timing LANDED + CPU-GATED 2026-07-27 (CLAIM-ROADMAP-C8-RESPONSE-METRICS, NOT pushed): EngineCoreEventType{QUEUED,SCHEDULED,PREEMPTED} + EngineCoreEvent{type,timestamp} recorded on Request at the add_request / batch-admission / KV-preempt sites (1:1 with vLLM, gated on log_stats_, default no-stats path byte-identical), drained onto EngineCoreOutput.events via take_events(); OutputProcessor folds them (update_from_events) into RequestState.queued_ts/scheduled_ts → FinishedRequestStats.queued_time(=scheduled−queued)/prefill_time(=first_token−scheduled)/inference_time(=last_token−scheduled) + IterationStats.num_preempted_reqs, feeding the vllm:request_{queue,prefill,inference}_time_seconds histograms + vllm:num_preemptions_total (already in the catalog, left at 0 by the live-metrics wiring for lack of events). Additive; scheduling/compute/token stream unchanged. RESIDUAL: the streaming/non-streaming chat/completion RESPONSE-BODY timing surface (protocol/serving) + CLI validation; AsyncLLM production metrics. |
T1 | vllm/v1/engine/__init__.py:150-176 (EngineCoreEvent(Type)); vllm/v1/core/sched/scheduler.py:2135,1003,1221,461,1839 (record/take_events sites); vllm/v1/metrics/stats.py:428-476 (update_from_events / update_from_finished_request); response-body: vllm/entrypoints/openai/engine/protocol.py:118; vllm/entrypoints/openai/{completion,chat_completion}/serving.py:461-481,765-784 @ 555967922 |
events include/vllm/v1/engine/event.h, Request.events+record_event/take_events include/vllm/v1/request.h; EngineCoreOutput.events include/vllm/v1/engine/types.h; emission src/vllm/v1/core/sched/scheduler.cpp (add_request/preempt_request/schedule/update_from_output) + log_stats_ include/vllm/v1/core/sched/scheduler.h; consumption src/vllm/v1/engine/output_processor.cpp (process_outputs) + RequestState.queued_ts/scheduled_ts include/vllm/v1/engine/output_processor.h; logger already consumes src/vllm/v1/metrics/loggers.cpp:225,254-257 |
tests/vllm/v1/test_scheduler.cpp "records QUEUED/SCHEDULED/PREEMPTED engine-core events" (15 assertions, RED-first, real KV-exhaustion preemption); tests/vllm/v1/test_llm_engine.cpp "per-request queue/prefill/inference timing populates" (26 assertions, RED-first: 5 flip 0→positive; asserts inference=prefill+decode, prefill≤inference≤e2e) |
per-request-response-metrics.md | ACTIVE |
CLAIM-ROADMAP-C8-RESPONSE-METRICS |
SERVE-STREAM-USAGE |
Completion/chat stream_options: final and continuous native-ID usage frames, non-stream validation, and force-usage server mode. GATING: the host implementation is CPU/sanitizer-green; void 31d053f 27B execution proved exact native counts on all 2,016 standard timed requests, but fresh passing 27B→35B online and serialization A/B gates remain mandatory |
T1 | vllm/entrypoints/openai/engine/protocol.py:241-243; completion protocol.py:66,471-478, serving.py:298-305,359-454; chat protocol.py:214,731-737, serving.py:459-512,570-760; entrypoints/serve/utils/api_utils.py:276-289; tests/entrypoints/openai/completion/test_completion.py:400-553; tests/entrypoints/openai/chat_completion/test_chat.py:348-445 |
schema/parser include/vllm/entrypoints/openai/protocol.h:62,203,318, src/vllm/entrypoints/openai/protocol.cpp:103,223,278; selection src/vllm/entrypoints/openai/serving_utils.cpp:8; completion SSE src/vllm/entrypoints/openai/serving_completion.cpp:22,160; chat SSE src/vllm/entrypoints/openai/serving_chat.cpp:232,450; force CLI examples/server/main.cpp:123 |
protocol/selection tests/vllm/entrypoints/openai/test_protocol.cpp:130,189; sync completion/chat tests/vllm/entrypoints/openai/test_serving.cpp:484,647; production final/continuous/validation/force/disconnect tests/vllm/entrypoints/openai/test_api_server.cpp:403,442,498,607,652,687,712; help examples/CMakeLists.txt:36. CPU CTest 105/105; focused 63 cases/658 assertions; API repeat 100/100; ASan+UBSan 3/3; TSan 1/1. 31d053f retained all 36 standard 27B raw points / 2,016 successful requests with exact native 128-token usage |
stream-options.md | GATING |
- |
SERVE-UTILITY-ENDPOINTS |
Tokenize, detokenize, ready, ping, server info, prefix reset. LANDED + CPU-GATED 2026-07-27 (CLAIM-ROADMAP-C8, NOT pushed): /tokenize (prompt form → {count,max_model_len,tokens,token_strs?}) + /detokenize ({tokens[]}→{prompt}) over the existing tokenizer, /ping (liveness, mirrors /health), /server_info ({vllm_config,vllm_env,system_env}), /reset_prefix_cache ({"success":bool} via an injected callback). All ADDITIVE + opt-in (tokenize/detokenize/reset registered only when their backing is attached). CHAT-FORM /tokenize LANDED + CPU-GATED 2026-07-28 (CLAIM-C8-CHAT-TOKENIZE, NOT pushed): /tokenize now accepts BOTH arms of the vLLM TokenizeRequest union — the raw prompt form AND the TokenizeChatRequest{messages, add_generation_prompt, continue_final_message, add_special_tokens, tools?}; the chat form renders through chat_.prompt_fn() (the IDENTICAL model chat template create_chat_completion tokenizes through), applies the check_generation_prompt mutual-exclusion (→400), tokenizes with the chat-form add_special_tokens default False (vs completion-form True), returns the same {count,max_model_len,tokens,token_strs?}. /tokenizer_info LANDED + CPU-GATED 2026-07-28 (CLAIM-C8-SERVE-ENDPOINTS, NOT pushed): GET /tokenizer_info gated behind a set_tokenizer_info_enabled flag mirroring vLLM's enable_tokenizer_info_endpoint CLI arg (off by default → the route is not registered → 404; on + tokenizer attached → 200). Surfaces the tokenizer_config.json-equivalent fields our byte-level/SentencePiece BPE tokenizer can GENUINELY back — tokenizer_class (the BPE family name), model_max_length, vocab_size, bos_token_id/eos_token_id (omitted when -1), and added_tokens_decoder (id → {content,special,lstrip,rstrip}); NAMED gaps OMITTED (never fabricated): the raw chat_template string (lives in the ChatPromptFn render seam, not the tokenizer), the HF init_kwargs (clean_up_tokenization_spaces/add_bos_token/model_input_names/padding-truncation defaults — not parsed), and the added-token normalized/single_word flags. PRODUCTION main.cpp WIRING LANDED + CPU-GATED 2026-07-28 (CLAIM-C8-SERVE-PROD-WIRING, NOT pushed): the shipped vllm-server binary now lights /tokenize+/detokenize (on by default when a tokenizer exists) and /tokenizer_info (behind the new --enable-tokenizer-info-endpoint flag, mirroring vLLM's enable_tokenizer_info_endpoint) from the LIVE engine+tokenizer through the shared ConfigureUtilityEndpoints seam — the SAME seam the gate drives over a real socket. /metrics + /reset_prefix_cache stay UNWIRED (named residuals): the production AsyncLLM frontend exposes no live PrometheusStatLogger (async stats deferred; missing LoadedEngine::stat_logger() + a Record() site in AsyncLLM::RunOutputHandler) and no thread-safe prefix-cache reset RPC (reset_prefix_cache() is KVCacheManager-private, mutated only on the EngineCore thread; missing AsyncLLM::reset_prefix_cache), so attaching either would be a fabricated wiring that never reaches the live engine — library handlers+tests retained. RESIDUAL: chat_template_kwargs/continue_final_message full template-render passthrough (the ChatPromptFn seam renders only via the add_generation_prompt gate), /ready, full server_info config dump, live /metrics + /reset_prefix_cache backing on the AsyncLLM path |
T1 | vllm/entrypoints/serve/tokenize/api_router.py:37,63,95-108; production gating vllm/entrypoints/openai/api_server.py:222, vllm/entrypoints/serve/__init__.py:11-31, vllm/entrypoints/openai/cli_args.py:140; vllm/entrypoints/serve/tokenize/protocol.py:24,50,156,166,181,185; vllm/entrypoints/serve/tokenize/serving.py:57,70-124,154-195; vllm/entrypoints/serve/sagemaker/api_router.py:47; vllm/entrypoints/serve/dev/server_info/api_router.py:43; vllm/entrypoints/serve/dev/cache/api_router.py:20 |
handlers src/vllm/entrypoints/openai/api_server.cpp:262 (handle_tokenize, prompt+chat union),:368,404,245,422 (handle_detokenize/handle_reset_prefix_cache/handle_ping/handle_server_info),:438 (handle_tokenizer_info); chat render seam include/vllm/entrypoints/openai/serving_chat.h:197 (prompt_fn()); opt-in setters + routes include/vllm/entrypoints/openai/api_server.h:118 (set_tokenizer_info_enabled); production seam include/vllm/entrypoints/openai/api_server.h (ConfigureUtilityEndpoints) + src/vllm/entrypoints/openai/api_server.cpp (impl); production call + CLI flags examples/server/main.cpp (--enable-tokenizer-info-endpoint, ConfigureUtilityEndpoints(...)) |
tests/vllm/entrypoints/openai/test_api_server.cpp:879 (prompt round-trip+schema+raw-form exact ids),:938 (chat-form renders template + exact tokens, RED-first),:1061 (/tokenizer_info backed fields + named-gap omissions + no-tokenizer 500),:1250 (opt-in route gate: 404 flag-off → 200 flag-on over a real socket, RED-first),:1319 (production ConfigureUtilityEndpoints seam over a real socket: no-seam→404 RED, defaults→tokenize/detokenize 200 + info/abort 404, flags-on→200, exact abort delta-count) — 32/32 / 420-assertion suite |
utility-endpoints.md | ACTIVE |
CLAIM-C8-SERVE-PROD-WIRING |
SERVE-CHAT-TEMPLATE |
Full-surface Jinja chat templates (vendored google/minja 021c229 + documented lstrip guard) |
T0 | vllm/renderers/hf.py:673,986; vllm/entrypoints/chat_utils.py:1248,1335 |
src/vllm/entrypoints/chat_template.cpp:101,168,181,220 |
tests/vllm/entrypoints/test_chat_template.cpp:65,75,84,96 |
planned: specs/chat-templating.md |
ANCHOR-BACKFILL |
- |
SERVE-ASYNC-LLM |
AsyncLLM-equivalent streaming engine API: per-request collectors, concurrent submit/generate/abort, live completion/chat SSE with disconnect abort, additive nonblocking C requests, and enough HTTP delivery capacity for configured concurrent streams. GATING: deterministic c32 capacity is implemented and GPU-classified; broader every-axis parity remains open | T0 | vllm/v1/engine/async_llm.py:70,280,524,637,709; vllm/v1/engine/output_processor.py:45-105; asyncio server path vllm/entrypoints/openai/api_server.py:1; tests/v1/engine/test_async_llm.py:109,157,228,306,340,598 |
existing async path include/vllm/v1/engine/async_llm.h:45, src/vllm/v1/engine/async_llm.cpp:32; fixed/legacy pool API include/vllm/entrypoints/openai/api_server.h:41-57,101-104; capacity selection src/vllm/entrypoints/openai/api_server.cpp:23-62; production max-seqs wiring + VLLM_CPP_HTTP_FIXED_POOL=0 A/B examples/server/main.cpp:230-247; cpp-httplib defect third_party/httplib/httplib.h:161-169,10359-10377 |
persistent 32-client + control reserve, validation and diagnostic-mode cases tests/vllm/entrypoints/openai/test_api_server.cpp:937-1000; focused Release/help pass, API 100/100, ASan+UBSan 1/1, TSan 1/1; known unrelated serial C-API flake isolated. Exact fixed/legacy c32 AB/BA/AB is healthy and steady-state-neutral: 1097.031/1097.290 tok/s = 0.999764×, 8/20 axes, 1,152/1,152 requests and six memory returns; neither legacy arm samples the rare old stall. Exact 4e1d8ca fixed c32 is healthy 3/3 and 0.9910× vLLM |
async-serving.md | GATING |
- |
SERVE-HTTP-TRANSPORT |
Serving-socket transport parity: mirror vLLM's uvicorn/asyncio default TCP_NODELAY on every accepted SSE socket so per-token stream frames are not held by Nagle against the peer's delayed ACK. Implemented + CPU-tested; the non-binding localhost A/B sizing is COMPLETE and NEUTRAL within noise on c1/c2 ITL/TPOT/throughput (loopback ACKs are instant, so Nagle never coalesces ~100 ms-cadence token frames) — no gate-axis credit expected; the mirror stays for real-network parity. Future keep-alive / read-write-timeout / listening-socket option parity noted, not done |
T0 | vLLM serves via uvicorn over asyncio vllm/entrypoints/launcher.py:71,76, vllm/entrypoints/openai/api_server.py:591,630; asyncio disables Nagle per accepted TCP stream socket asyncio/base_events.py:192-197 (_set_nodelay) called from asyncio/selector_events.py:950; cpp-httplib default-off third_party/httplib/httplib.h:142, applied on accept only when set third_party/httplib/httplib.h:12083 |
src/vllm/entrypoints/openai/api_server.cpp:69 (set_tcp_nodelay(true) in the ApiServer setup) |
behavioral accepted-socket getsockopt(TCP_NODELAY) case tests/vllm/entrypoints/openai/test_api_server.cpp:1076 (helper :380); RED accepted TCP_NODELAY 0 → GREEN 1, full test_openai_api_server 22/22 cases / 242 assertions; non-binding sizing root ~/work/vllm.cpp-tcpnodelay-sizing/ff915e8… (raw-set SHA f5b52900…2128) neutral within noise; closure ledger |
serve-tcp-nodelay.md | DONE |
ff915e8 |
SERVE-C-ABI |
Stable LocalAI-style C FFI (19 exported VLLM_API symbols at VLLM_ABI_VERSION 10; blocking and nonblocking request handles. Count corrected 2026-07-24 from a stale 17, which predated ABI v4/v5 adding tool_parser/reasoning_parser and the chat entry points; include/vllm.h is the source of truth and README:231 already said 19). ABI v9 2026-07-28 (CLAIM-CAPI-ENGINE-CONFIG-V9): the ABI carried strictly LESS engine config than EngineParams does - max_num_batched_tokens, the scheduler scheduling_policy (fcfs / priority / lpm), and kv_transfer_config (the external KV connector / LMCache JSON) were reachable from the bundled server's flags and from NO embedder. All three added, inert at their defaults (zero-filled v8 growth == byte-identical pre-v9 engine); the connector NAME is validated against KVConnectorFactory at load, mirroring the server's startup check. tokenizer_config_path stopped being a declared-since-v1 no-op and now selects the chat template's source file. Malformed speculative_config/kv_transfer_config documents now report VLLM_ERR_INVALID_ARGUMENT (the contract vllm.h documented since v6) instead of VLLM_ERR_MODEL_LOAD, via a catch scoped to the parse block so a real FromModelDir failure still reports MODEL_LOAD. Driver: the LocalAI vllm-cpp backend could not expose LMCache or the prefill budget in a model config) |
T0 | Original project ABI; pinned vLLM has no C ABI | include/vllm.h:143,181,207; src/capi/vllm_c.cpp:229,264,327,391 |
tests/capi/test_capi.cpp:320,428,505,574,606,640; tests/capi/test_dlopen.cpp:77,86; tests/capi/c_header_compile.c:1 |
c-api-library.md | ANCHOR-BACKFILL |
CLAIM-SERVE-C-ABI-SPIKE |
SERVE-CPP-API |
Rich LLM and AsyncLLM C++ API |
T1 | vllm/entrypoints/llm.py:66,422; vllm/v1/engine/async_llm.py:70 |
- | - | planned: specs/cpp-api.md |
INVENTORIED |
- |
SERVE-CLI-BENCH |
Serve and latency/throughput/serve benchmark modes | T0 | vllm/entrypoints/cli/serve.py:44; vllm/entrypoints/cli/benchmark/main.py:29 |
separate binaries + explicit scheduler-capacity flags examples/server/main.cpp:63,96,116,170; examples/bench/main.cpp:40,109; examples/bench/bench_core.h:96,468 |
server help contract examples/CMakeLists.txt:34; benchmark tests/examples/test_bench.cpp:15,48 |
planned: specs/cli-serve-bench.md |
PARTIAL |
- |
SERVE-GATE-ONLINE |
Same-corpus online correctness, TTFT/TPOT/ITL, throughput and peak-memory gate vs vLLM v0.25.0 | T0 | vllm/benchmarks/serve.py:1,581-615; v0.25 audit; tests/benchmarks/test_serve_cli.py:1 |
Schema-v5 harness plus trace controller, production component driver, and fail-closed component finalizer | BINDING 9ecd9d0: 114/124 (async default ON; mem 4/4, c1 20/20, c2 20/20, c16 19/20, c4 & c32 18/20, c8 15/20; benchmark_binding refers here, superseding 3f256ab 55/124 and 246a23c 49/124, both retained immutable). Two-grid totality with f0fb727 (111/124) is 115/124 effective parity vs vLLM 0.25.0 (27B). Async CLOSED the c16/c32 ITL tails (ours now BEATS vLLM: c16 p99 1.055, c32 p90 1.034/p99 1.078) and leaves a stable c8 p99_itl ~0.86 residual, ROOT-CAUSED (2026-07-18, CLAIM-C8-P99-TAIL-1, spec) as IRREDUCIBLE-AS-MIRRORED: our deterministic synchronous forward keeps co-admitted c8 requests in byte-identical lockstep where vLLM's async-future jitter de-phases them; the c16/c32 INVERSION proves this is the trailing edge of the per-step determinism that wins c16/c32 + throughput, not a capability gap (scheduler + async placeholder byte-identical, tests/vllm/v1/test_scheduler_wave.cpp, tail spec). Full grid + per-binding forensics: roadmap_v1.md + parity ledger; no packed speed credit |
online serving gate; merged GDN projections; packed decode | ACTIVE |
CLAIM-SERVE-GATE-1 |
SERVE-E2E-NIGHTLY |
Server conformance and real-model nightly suites for all release gates | T0 | tests/entrypoints/openai/; tests/v1/e2e/; .buildkite/test-pipeline.yaml |
current unit/conformance tests only; no scheduled DGX suite | tests/vllm/entrypoints/openai/test_conformance.cpp:1; tests/parity/test_qwen36_paged_engine.cpp:78; tests/parity/test_qwen27_paged_engine.cpp:110 |
planned: specs/server-e2e-nightly.md |
INVENTORIED |
- |
SERVE-CLI-CHAT |
Interactive chat and complete commands against a running OpenAI-compatible server, plus preservation of the existing local-model completion invocation |
T1 | registration vllm/entrypoints/cli/main.py:17-37,73-98; client/model resolution + stream shaping vllm/entrypoints/cli/openai.py:30-100; chat :155-234; complete :237-312 at 5559679229 |
current in-process completion only examples/cli/main.cpp:1-207; remote command implementation absent |
C-ABI stream baseline tests/capi/test_capi.cpp:567-711; chat-template baseline tests/capi/test_chat_prompt.cpp:37-89; command/fake-server tests absent |
cli-chat-complete.md | SPIKE |
CLAIM-SERVE-CLI-CHAT-SPIKE |
SERVE-POOLING-ENDPOINTS |
Embeddings, pooling, score, rerank, classify HTTP surface (/v1/embeddings, /pooling, /score, /rerank, /classify). SPIKED 2026-07-28 (CLAIM-POOLING): the whole pooling task class is scoped in pooling-task-class.md — endpoints depend on the pooling RUNNER (ENG-POOLING-RUNNER, W3 LANDED 2026-07-29 — PoolingRunner returns pooled data via the model Pooler) and a concrete pooling model. Endpoint protocol + handler port is the W4 brick |
T2 | vllm/entrypoints/pooling/embed/api_router.py:28; vllm/entrypoints/pooling/scoring/api_router.py:37,71; vllm/entrypoints/pooling/classify/api_router.py:26 |
- | - | pooling-task-class.md | SPIKE |
CLAIM-POOLING |
ENG-POOLER-SEQ |
The non-generative POOLER OP — turn hidden states into a pooled embedding/logit row instead of a sampled token. W1 LANDED + CPU-GATED 2026-07-28 (CLAIM-POOLING, NOT pushed): the sequence pooling methods CLSPool/LastPool/MeanPool (+ GetSeqPoolingMethod factory) over a packed [num_tokens, hidden] CPU buffer keyed by a minimal PoolingCursor (CLS/MEAN reject partial prefill, LAST allows it, MeanPool upcasts to float32) and the activation heads PoolerIdentity/PoolerNormalize (L2 F.normalize)/PoolerMultiLabelClassify (sigmoid)/PoolerClassify (sigmoid if num_labels<2 else softmax). Unit-gated vs DOUBLE-PRECISION references, RED-first. W2 LANDED + CPU-GATED 2026-07-29 (CLAIM-POOLING, NOT pushed): the pooler HEADS composite (EmbeddingPoolerHead = projector→matryoshka→normalize; ClassifierPoolerHead = classifier→(logit-mean)/sigma→activation), the SequencePooler (method∩head task intersection) + PoolerForEmbed/PoolerForClassify factories, the DispatchPooler groupby-task routing (ForEmbedding/ForSeqCls + a mixed embed+classify batch + ctor task-support validation), and the PoolerConfig/PoolingParams/PoolingParamsUpdate structs; test_pooler_heads 27/27 (240 asserts) vs double-precision refs, RED-first (disable matryoshka slice + logit_mean calibration → 8 cases / 50 asserts fail). RESIDUALS (named, spec §Work breakdown): the endpoints (W4), tokwise AllPool/StepPool (W5), a concrete pooling MODEL + real-oracle cosine gate (W3-model — see ENG-POOLING-RUNNER) |
T2 | vllm/model_executor/layers/pooler/seqwise/methods.py:35-121; vllm/model_executor/layers/pooler/activations.py:106-158; vllm/model_executor/layers/pooler/seqwise/heads.py:19-196; vllm/model_executor/layers/pooler/seqwise/poolers.py:41-138; vllm/model_executor/layers/pooler/special.py:23-140; vllm/model_executor/layers/pooler/common.py:12-30; vllm/pooling_params.py:35-70; vllm/config/pooler.py:16-90; vllm/v1/pool/metadata.py:13-71; tests/model_executor/layers/test_pooler_methods.py, tests/model_executor/layers/test_pooler_activations.py, tests/model_executor/layers/test_pooler_heads.py |
include/vllm/model_executor/layers/pooler/{methods,activations,pooling_metadata,common,pooling_params,pooler_config,heads,poolers,dispatch_pooler}.h + src/vllm/model_executor/layers/pooler/{methods,activations,heads,poolers,dispatch_pooler}.cpp |
tests/vllm/model_executor/layers/pooler/test_pooler.cpp (CLS/LAST/MEAN + factory + activations, 50 asserts) + test_pooler_heads.cpp (Embedding/Classifier heads + SequencePooler + DispatchPooler, 240 asserts) |
pooling-task-class.md | ACTIVE |
CLAIM-POOLING |
ENG-POOLING-RUNNER |
The pooling RUNNER path — where the generation runner SAMPLES a token, the pooling runner applies the model's Pooler to the last hidden state and returns the POOLED DATA (embedding vector / classification logit row). W3 LANDED + CPU-GATED 2026-07-29 (CLAIM-POOLING, NOT pushed): PoolingRunner over a packed [num_tokens, hidden] last-hidden-state buffer + a PoolingMetadata — Pool() delegates to the model pooler (DispatchPooler.ForEmbedding), GetSupportedTasks(), ComputeValid() (seq_lens==prompt_len). GATE: a STRUCTURAL cosine-parity gate — the runner's embedding vs an independent double-precision LAST+normalize reference is cosine≈1 (5 cases / 14 asserts), RED-first (CLS-instead-of-LAST drops cosine <0.5; disable normalize → 2 unit-L2 asserts fail). GENERALIZATION DEVIATION: upstream pooling_runner.py hardcodes LAST+normalize; we route through the model Pooler (the general bert.py path), strictly more capable. HONEST RESIDUAL (named): the REAL-model oracle cosine gate (vllm.LLM(task="embed").encode) needs a registered concrete embedding model's forward — no such model is registered yet (W3-model), so no cosine-vs-oracle number is fabricated. The InputBatch→PoolingMetadata construction (logits_indices gather) rides the endpoint brick (W4) |
T2 | vllm/v1/worker/gpu/pool/pooling_runner.py:18-46; vllm/tasks.py:10; tests/models/language/pooling/test_embedding.py (real-oracle gate, DEFERRED) |
include/vllm/v1/worker/gpu/pool/pooling_runner.h + src/vllm/v1/worker/gpu/pool/pooling_runner.cpp |
tests/vllm/v1/worker/gpu/pool/test_pooling_runner.cpp (runner path + structural cosine gate, 14 asserts, RED-first) |
pooling-task-class.md | ACTIVE |
CLAIM-POOLING |
SERVE-RESPONSES-MESSAGES |
Responses, Anthropic messages, audio | T2 | vllm/entrypoints/openai/responses/api_router.py:48; vllm/entrypoints/anthropic/api_router.py:49; vllm/entrypoints/speech_to_text/transcription/api_router.py:1 |
- | - | planned: specs/responses-messages-endpoints.md |
INVENTORIED |
- |
SERVE-ADMIN |
Abort-requests, sleep, pause/resume, profiling, RL weight updates. /abort_requests LANDED + CPU-GATED 2026-07-28 (CLAIM-C8-SERVE-ENDPOINTS, NOT pushed): POST /abort_requests (from the dev/rlhf admin router) parses {request_ids:[...]} and aborts exactly those (external) ids via an injected abort callback wired to the engine abort path (AsyncLLM::abort); an empty/missing list means "abort all in-flight" (the callback decides). Response {"status":"aborted","aborted":<count>}; malformed JSON → 400 {"detail":"Invalid JSON format"}; abort failure → 500 {"error":...} — all three shapes mirror the upstream router verbatim. ADDITIVE + opt-in (route registered only when the abort callback is attached → 404 otherwise). PRODUCTION main.cpp WIRING LANDED + CPU-GATED 2026-07-28 (CLAIM-C8-SERVE-PROD-WIRING, NOT pushed): the shipped vllm-server binary now wires /abort_requests to the LIVE AsyncLLM::abort through the shared ConfigureUtilityEndpoints seam, DEV-mode gated behind the new --enable-server-dev-mode flag — mirroring vLLM registering the dev/rlhf router only under if envs.VLLM_SERVER_DEV_MODE (api_server.py:238; envs.py:157 default 0). Explicit-id abort tears the request down and reports the exact drop in unfinished requests (before−after); empty request_ids (abort-ALL) reports 0 — NAMED RESIDUAL (AsyncLLM exposes no active-request-id accessor). RESIDUAL: the abort-ALL enumeration (missing AsyncLLM::active_request_ids()); /sleep//wake_up//is_sleeping, /pause//resume, /start_profile//stop_profile, weight-update/EP endpoints still INVENTORIED |
T2/T3 | vllm/entrypoints/serve/dev/rlhf/api_router.py:94-138 (abort_requests); dev-mode gate vllm/entrypoints/openai/api_server.py:238-240, vllm/entrypoints/serve/__init__.py:35, vllm/envs.py:157; vllm/entrypoints/serve/dev/sleep/api_router.py:21; vllm/entrypoints/serve/dev/rlhf/api_router.py:29,74,136; vllm/entrypoints/serve/profile/api_router.py:21 |
handler src/vllm/entrypoints/openai/api_server.cpp:488 (handle_abort_requests); opt-in setter include/vllm/entrypoints/openai/api_server.h:156 (set_abort_requests); production seam src/vllm/entrypoints/openai/api_server.cpp (ConfigureUtilityEndpoints, before/after delta-count) + examples/server/main.cpp (--enable-server-dev-mode); engine abort path include/vllm/v1/engine/async_llm.h:115 (abort) |
tests/vllm/entrypoints/openai/test_api_server.cpp:1104 (shape + callback wiring: explicit ids passthrough, empty→abort-all branch, malformed→400),:1143 (aborts an in-flight AsyncLLM request → has_unfinished_requests() false),:1250 (opt-in route gate: 404 no-callback → 200 attached, RED-first),:1319 (production seam: dev-mode gate 404→200, live abort exact delta-count==1, empty→0) — in the 32/32 / 420-assertion suite |
admin-endpoints.md | ACTIVE |
CLAIM-C8-SERVE-PROD-WIRING |
SERVE-OTLP |
OpenTelemetry traces | T2 | vllm/config/observability.py:18,36,128 |
- | - | planned: specs/otlp-tracing.md |
INVENTORIED |
- |
SERVE-BATCH-API |
Offline OpenAI Batch API runner — read a JSONL of BatchRequestInput (custom_id/method/url/body), dispatch each line to the matching serving handler, collect BatchRequestOutput rows (custom_id echoed, per-line error isolation), write the response JSONL. W0 SPIKE + W1 CPU BRICK LANDED + CPU-GATED 2026-07-29 (CLAIM-BATCH-API, NOT pushed): RunBatch (RunLine/RunLines/Run) + RunBatchFile (local paths) as a pure ORCHESTRATOR over the existing OpenAIServingChat::create_chat_completion (the SAME handler handle_chat_completions drives — NO reimplemented generation), 1:1 with vLLM's endpoint_registry url→handler map. /v1/chat/completions wired; BatchResponseData/BatchRequestOutput schema + vllm-<uuid>/vllm-batch-<uuid> ids; the run_request AllResponse/ErrorResponse/stream branches; the unsupported-endpoint (handler None) + unsupported-url error rows. RECORDED DEVIATION: a malformed input line is ISOLATED into an error row (batch continues) where upstream aborts the job (deviation lives in the library; the abort-on-bad-line CLI exit code is a W2 residual). RESIDUALS (named, spec §Work breakdown): the vllm run-batch CLI + BatchFrontendArgs (W2); embeddings/score/rerank dispatch (W3, rides SERVE-POOLING-ENDPOINTS); audio transcription/translation + media fetch (W4); http(s)/data-URL file I/O, metrics server, overlapped AsyncLLM submission (W5) |
T2 | vllm/entrypoints/openai/run_batch.py:148-228 (schema),:508-570 (run_request/make_error),:722-777,815-847 (dispatch/run loop); tests/entrypoints/openai/test_run_batch.py:375,402,432 |
include/vllm/entrypoints/openai/run_batch.h; src/vllm/entrypoints/openai/run_batch.cpp:70,84,87,142 |
tests/vllm/entrypoints/openai/test_run_batch.cpp (7 cases / 80 assertions, CPU, RED-first: dropping the custom_id echo fails 9 assertions) |
batch-api.md | ACTIVE |
CLAIM-BATCH-API |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
LORA-RUNTIME |
Punica-style batched LoRA apply. W1 CPU BRICK LANDED + CPU-GATED 2026-07-28 (CLAIM-LORA-RUNTIME, NOT pushed): the LoRALayerWeights container (optimize scaling-fold, dummy) + the portable punica shrink/expand ops (BgmvShrink/BgmvExpand/BgmvExpandSlice, -1-slot SKIP semantics mirroring the triton early-exit + the test CPU refs) + AddLoraLinear (buffer=shrink; y+=expand) + a LoRALinear (ReplicatedLinear n_slices=1: create/set/reset/apply, scaling folded into b at SetLora exactly like the manager optimize path). RUNTIME-VERIFIED on CPU: test_punica_cpu 6/6 (101 assertions) vs an independent double-precision per-LoRA matmul reference; RED-first proven (base-only output differs; -1 base token unchanged; ResetLora → identity). Packed/TP/merged layers, mapping metadata, adapter load, LRU manager, GPU kernels + model gate are W2-W7 (see spec). |
T2 | vllm/lora/lora_model.py:60; vllm/lora/lora_weights.py:13; vllm/lora/ops/torch_ops/lora_ops.py:24; vllm/lora/punica_wrapper/punica_cpu.py:265; vllm/lora/layers/base_linear.py:100; vllm/lora/punica_wrapper/punica_gpu.py:33; vllm/v1/worker/lora_model_runner_mixin.py:30 |
include/vllm/lora/lora_weights.h:27; include/vllm/lora/punica.h:42,69,78; src/vllm/lora/punica_cpu.cpp:45,87,104,142; CMakeLists.txt:367 |
tests/vllm/lora/test_punica_cpu.cpp:76,91,109,138,161,219 (6/6, 101 assertions, CPU) |
lora-adapter.md | ACTIVE |
CLAIM-LORA-RUNTIME |
LORA-ENDPOINTS |
Dynamic adapter load and unload (POST /v1/{load,unload}_lora_adapter); builds on the runtime, scoped as W6 of lora-adapter.md |
T2 | vllm/entrypoints/serve/lora/api_router.py:43,60 |
- | - | planned: specs/lora-adapter.md |
INVENTORIED |
- |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
ATTN-YARN |
Typed YaRN RoPE config/factory/cache plus plain and mrope_section supplied-cache apply; CPU/oracle/sanitizer green. GPU-GATED 2026-07-27 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a, clean CUDA build of 489f7771): the shared scaled-RoPE CUDA apply+cache path COMPILES/RUNS on GB10 — test_ops_rope_cache 6/6 (6692 assertions), test_rotary_embedding 14/14, and the same RopeFromCache op is exercised by the Phi-4-mini/Llama-3.2-1B model gates below. Model-level YaRN e2e is REACHABLE-BLOCKED (no cached oracle-runnable YaRN consumer — Nomic-bert/gpt-oss absent, cached Qwen3-4B is default-rope); the YaRN formula itself stays oracle-gated (G3 CPU goldens). SPEED pending |
T1 | vllm/model_executor/layers/rotary_embedding/__init__.py:243-284; vllm/model_executor/layers/rotary_embedding/yarn_scaling_rope.py:10-84; vllm/model_executor/layers/rotary_embedding/mrope.py:201-340; tests/kernels/core/test_mrope.py:47-235; tests/models/language/pooling/test_nomic_max_model_len.py:93-113 |
include/vllm/transformers_utils/hf_config.h:21; src/vllm/transformers_utils/hf_config.cpp:129; src/vllm/model_executor/layers/rotary_embedding/base.cpp:182; src/vllm/model_executor/layers/rotary_embedding/yarn_scaling_rope.cpp:16; src/vllm/model_executor/layers/rotary_embedding/mrope.cpp:29; include/vt/ops.h:723; src/vt/cpu/cpu_ops.cpp:462; src/vt/cuda/cuda_ops.cu:422 |
tests/vllm/test_hf_config.cpp:424; tests/vllm/model_executor/layers/rotary_embedding/test_rotary_embedding.cpp:47; tests/vt/test_ops_rope_cache.cpp:76; tests/parity/test_op_parity.cpp:357,387,1578; tests/parity/goldens/long_rope_yarn_neox_f32_truncate/manifest.json:1 |
sliding-local-yarn-long-context.md | ACTIVE |
CLAIM-ROADMAP-C5 |
ATTN-ROPE-FAMILY |
Block row (claim the three leaves below, not this row): Llama 3, LongRoPE, and dynamic-NTK scaling | T1/T2 | vllm/model_executor/layers/rotary_embedding/__init__.py:155-171,200-230,315-335; tests/kernels/core/test_pos_encoding.py:66-193 |
- | - | sliding-local-yarn-long-context.md | READY |
- |
ATTN-ROPE-LLAMA3 |
Llama 3 typed config/factory/cache with exact unchanged, smoothed and scaled frequency bands including equal factors; CPU/oracle/sanitizer green. GPU-GATED 2026-07-27 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a, clean CUDA build of 489f7771, oracle vLLM 0.26.0.dev0+g5559679): feature-positive model SACRED gate test_llama_paged_engine (Llama-3.2-1B, unsloth/Llama-3.2-1B) 16/16 (12 STRICT token-exact + 4 near-tie gap 0.0 nats, 0 forward-divergent) — llama3 band-rescale is applied at ALL positions (not thresholded). 0.26 oracle recapture ALL-DETERMINISTIC over K=5 and BIT-IDENTICAL to the committed golden ⇒ zero drift ⇒ STRICT-valid vs the 0.26 oracle. SPEED pending |
T1 | vllm/model_executor/layers/rotary_embedding/__init__.py:155-171; vllm/model_executor/layers/rotary_embedding/llama3_rope.py:11-54 |
include/vllm/transformers_utils/hf_config.h:21,32; src/vllm/transformers_utils/hf_config.cpp:184,215; include/vllm/model_executor/layers/rotary_embedding/base.h:80; src/vllm/model_executor/layers/rotary_embedding/base.cpp:148,220; include/vllm/model_executor/layers/rotary_embedding/llama3_rope.h:14; src/vllm/model_executor/layers/rotary_embedding/llama3_rope.cpp:11,33 |
tests/vllm/test_hf_config.cpp:467,539; tests/vllm/model_executor/layers/rotary_embedding/test_rotary_embedding.cpp:202,244,294; tests/parity/test_op_parity.cpp:357,395,1586; tests/parity/goldens/long_rope_llama3_neox_f32_bands/manifest.json:1 |
sliding-local-yarn-long-context.md | ACTIVE |
CLAIM-ROADMAP-C5 |
ATTN-ROPE-LONGROPE |
Phi-3 LongRoPE typed factor arrays, optional mscales, concatenated short/long caches and one runtime-length-global selection; CPU/oracle/sanitizer green. GPU-GATED 2026-07-27 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a, clean CUDA build of 489f7771, oracle vLLM 0.26.0.dev0+g5559679): feature-positive model SACRED gate test_phi3_paged_engine (Phi-4-mini-instruct) 16/16 (7 STRICT + 9 near-tie <=0.5 nats, 0 forward-divergent; RED-first proven — disabling the LongRoPE mscale flips 11 roots the gate catches); the LONG cache is selected globally (max_model_len 131072 > original 4096). 0.26 oracle recapture ALL-DETERMINISTIC over K=5 and BIT-IDENTICAL to the committed golden ⇒ zero drift ⇒ STRICT-valid vs the 0.26 oracle. SPEED pending |
T2 | vllm/model_executor/layers/rotary_embedding/__init__.py:315-335; vllm/model_executor/layers/rotary_embedding/phi3_long_rope_scaled_rope.py:16-159 |
include/vllm/transformers_utils/hf_config.h:35; src/vllm/transformers_utils/hf_config.cpp:196,247; include/vllm/model_executor/layers/rotary_embedding/base.h:98; src/vllm/model_executor/layers/rotary_embedding/base.cpp:193,256; include/vllm/model_executor/layers/rotary_embedding/phi3_long_rope_scaled_rope.h:16; src/vllm/model_executor/layers/rotary_embedding/phi3_long_rope_scaled_rope.cpp:13,60,77,88 |
tests/vllm/test_hf_config.cpp:497,599; tests/vllm/model_executor/layers/rotary_embedding/test_rotary_embedding.cpp:281,335,411; tests/parity/test_op_parity.cpp:357,399,1604; tests/parity/goldens/long_rope_phi3_neox_f32_short/manifest.json:1 |
sliding-local-yarn-long-context.md | ACTIVE |
CLAIM-ROADMAP-C5 |
ATTN-ROPE-DYNAMIC-NTK |
Dynamic-NTK typed alpha/factor dispatch with alpha precedence, optional trained length, exact base transforms and dimension guard; CPU/oracle/sanitizer green. GPU-GATED 2026-07-27 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a, clean CUDA build of 489f7771, oracle vLLM 0.26.0.dev0): model SACRED gate test_internlm2_paged_engine (internlm2-chat-1_8b, rope_scaling type dynamic factor 2.0) 16/16 (12 STRICT + 4 near-tie, 0 forward-divergent) proves the dynamic config-parse + cache build on GB10. HONEST: dynamic-NTK is IDENTITY at gate-battery lengths (seq < trained length ⇒ unchanged base — exactly mirrors vLLM); the nontrivial NTK base transform needs a >trained-length prompt (unreached at the gate battery) and stays oracle-gated by G3 CPU goldens. SPEED pending |
T1/T2 | vllm/model_executor/layers/rotary_embedding/__init__.py:200-230; vllm/model_executor/layers/rotary_embedding/dynamic_ntk_scaling_rope.py:30-73; vllm/model_executor/layers/rotary_embedding/dynamic_ntk_alpha_rope.py:9-43; tests/test_config.py:543-587 |
include/vllm/transformers_utils/hf_config.h:42; src/vllm/transformers_utils/hf_config.cpp:200,265; src/vllm/model_executor/layers/rotary_embedding/base.cpp:243; include/vllm/model_executor/layers/rotary_embedding/dynamic_ntk_scaling_rope.h:15; src/vllm/model_executor/layers/rotary_embedding/dynamic_ntk_scaling_rope.cpp:12,31; include/vllm/model_executor/layers/rotary_embedding/dynamic_ntk_alpha_rope.h:15; src/vllm/model_executor/layers/rotary_embedding/dynamic_ntk_alpha_rope.cpp:12,26 |
tests/vllm/test_hf_config.cpp:532,650; tests/vllm/model_executor/layers/rotary_embedding/test_rotary_embedding.cpp:389,411,429,500; tests/parity/test_op_parity.cpp:357,399,1611; tests/parity/goldens/long_rope_dynamic_factor1_neox_f32/manifest.json:1 |
sliding-local-yarn-long-context.md | ACTIVE |
CLAIM-ROADMAP-C5 |
ATTN-SLIDING-WINDOW |
Sliding-window attention semantics and backend dispatch; generic config/window seam plus CPU, portable-CUDA, and vendored-FA2 local masks are implemented. GPU-GATED 2026-07-27 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a, clean CUDA build of 489f7771, oracle vLLM 0.26.0.dev0): the local-mask CUDA kernel is POSITIVELY gated at operator level — test_ops_paged_attn 25/25 (454474 assertions) incl. WMMA sliding-window max_abs_err 2.10e-6 vs f32 ref + FA-2, and test_attention_window 3/3 (window-straddling (left,right) masks, mask-exact). Model consumers gated on GB10: Gemma-2 (test_gemma2_forward) 48/48 + Gemma-3 (test_gemma3_forward) 48/48 CUDA greedy tokens vs the vLLM oracle. Correctness-complete, SPEED pending; the window is INERT at the short gate battery (ctx < W) so the long-context positive-mask (prompt > W) model e2e + the KV-memory-optimization G8 are the honest tail |
T1 | vllm/v1/attention/backends/flash_attn.py:255-300,674-717,840-955; tests/v1/attention/test_attention_backends.py:745-867; tests/v1/e2e/general/test_correctness_sliding_window.py:19-78 |
include/vllm/model_executor/layers/attention/attention.h:24,33; src/vllm/model_executor/layers/attention/attention.cpp:12,33; include/vt/ops.h:239,260; src/vt/cpu/cpu_paged_attn.cpp:86; src/vt/cuda/cuda_paged_attn.cu:85,1892,2051; src/vt/cuda/cuda_flash_attn_fa2.cu:234 |
tests/vllm/model_executor/layers/attention/test_attention.cpp:22,67,81,100; tests/vt/test_ops_paged_attn.cpp:454,500,779,1145; tests/vllm/test_hf_config.cpp:342; tests/vllm/models/test_gemma2_forward.cpp; tests/vllm/models/test_gemma3_forward.cpp |
sliding-local-yarn-long-context.md | ACTIVE |
CLAIM-ROADMAP-C5 |
ATTN-CHUNKED-LOCAL |
Chunked-local cached wrapper, virtual Q/K batches, reusable block-table gather/update plan, cudagraph rejection, spec emission and ordinary-backend delegation; CPU G1/reference G5 green. On the clean CUDA build of 489f7771 (CLAIM-ROADMAP-C5, dgx GB10 sm_121a) it compiles -Werror-clean and test_chunked_local_attention 5/5 (18849 assertions, all six pinned virtual-batch vectors) passes; delegation runs the ordinary paged backend (already GPU-gated). Model-level feature-positive e2e is REACHABLE-BLOCKED: no Llama4ForCausalLM model row is implemented (the representative chunked-local consumer). Row stays GATING on that vehicle |
T1 | vllm/model_executor/layers/attention/chunked_local_attention.py:30-128; vllm/v1/attention/backends/utils.py:225-420; tests/v1/attention/test_chunked_local_attention.py:28-204 |
include/vllm/model_executor/layers/attention/chunked_local_attention.h:37,82,113; src/vllm/model_executor/layers/attention/chunked_local_attention.cpp:82,108,126; include/vllm/v1/attention/backends/utils.h:17; src/vllm/v1/attention/backends/utils.cpp:29,74; include/vllm/v1/attention/backend.h:58,79,94 |
tests/vllm/v1/attention/test_chunked_local_attention.cpp:224,265,343,370,416,436; tests/vllm/v1/attention/test_common_attn_metadata.cpp:42 |
sliding-local-yarn-long-context.md | GATING |
- |
ATTN-MLA |
MLA prefill/decode backends and latent KV | T2 | vllm/v1/attention/backends/mla/flashinfer_mla.py:1; vllm/v1/attention/backends/mla/triton_mla.py:1 |
- | - | planned: specs/mla-backends.md |
INVENTORIED |
- |
ATTN-MAMBA |
Mamba1/Mamba2, short-conv, linear backends | T2 | vllm/v1/attention/backends/mamba_attn.py:30,79; vllm/model_executor/layers/mamba/short_conv.py:1 |
- | - | planned: specs/mamba-backends.md |
INVENTORIED |
- |
ATTN-ENCODER-CROSS |
Encoder and cross-attention | T2 | vllm/model_executor/layers/attention/attention.py:1; vllm/v1/attention/backends/utils.py:1 |
- | - | planned: specs/encoder-cross-attention.md |
INVENTORIED |
- |
| ID | Item | Tier | Upstream code/tests | Our code | Our tests/evidence | Spike/spec | State | Owner |
|---|---|---|---|---|---|---|---|---|
LOAD-SAFETENSORS |
General safetensors loading, stacked parameters, weights mapping. Binding memory scan finds all source mappings live through load plus a persistent CPU mirror of selected tensors; windowed progressive madvise(DONTNEED) release now drops each copied-then-dead source range during the copy loop so the mirror build no longer double-resides with the full source mmap |
T0 | streamed target-device construction vllm/model_executor/model_loader/base_loader.py:43-82; incremental safe_open yield vllm/model_executor/model_loader/weight_utils.py:905-954; immediate parameter copy vllm/model_executor/models/utils.py:170-180,252-279 |
all mappings opened/retained src/vllm/entrypoints/model_loader.cpp:47-63,303-312; MAP_PRIVATE reader src/vllm/model_executor/model_loader/safetensors_reader.cpp:43-70; windowed release primitive/gate src/vllm/model_executor/model_loader/safetensors_reader.cpp:285-340; copy-helper instrumentation src/vllm/model_executor/models/qwen3_5_weights.cpp:88,104,119,171,190,226,231 + qwen3_5_dense_weights.cpp:63,79,108,158,163,262,322 |
reader contracts + windowed-release cases green (smaps-Rss drop, byte-identity, gate semantics, neighbor safety); exact 27B accounting finds 1,155 tensors / 24,610,136,064 B (22.920 GiB) persistent host bytes. VmHWM A/B MEASURED at cb2d310 (root ~/work/vllm.cpp-windowed-load/cb2d310c…518/evidence, one flock, single 27B load per arm): OFF 48,285,916 kB vs ON 24,750,704 kB = VmRSS (−23.54 GB, load transient eliminated), ON-arm smoke 6/6; ledger row 2026-07-15. Exact-grid memory axes remain FAILED (projected PASS); direct-device streaming gate still open |
specs/safetensors-windowed-load.md | PARTIAL |
- |
LOAD-SAFETENSORS-DIRECT-DENSE |
Layer-bounded target-device loading for ordinary plain-BF16 Qwen3.5 dense safetensors on discrete CUDA; additive leaf below LOAD-SAFETENSORS, preserving windowed source release and owned-shard lifetime. Plain weights, stacked raw-NK owners, tied logits, logical resident state and same-queue layer staging are implemented and locally CUDA-gated. H32 Triton AOT, plain-BF16 decode graphs and ratio-4 FA2 repair the transplanted hot path; the broader row remains speed-gating |
T1 | target-device model construction vllm/model_executor/model_loader/base_loader.py:43-82; incremental safetensors yield vllm/model_executor/model_loader/weight_utils.py:820-954; stacked/tied parameters vllm/model_executor/models/qwen3_5.py:276-303,483-492 |
load/dispatch src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:51-132,186-245,333-471; discrete/unified classifier src/vt/cuda/cuda_backend.cu:251-265; plain execution/residency and graph selection src/vllm/model_executor/models/qwen3_5.cpp, src/vllm/model_executor/models/qwen3_5_dense.cpp; H32 recurrence src/vt/cuda/cuda_gdn.cu; ratio-4 FA2 src/vt/cuda/cuda_flash_attn_fa2.cu, src/vt/cuda/cuda_paged_attn.cu; exact benchmark corpus/output capture examples/bench/bench_core.h:61-68,357-386,409-589; reference metrics/token collector tools/bench/vllm_closed_loop_metrics.py; guarded driver and summarizer tools/bench/run_qwen35_4b_compare.sh, tools/bench/summarize_qwen35_4b_compare.py |
Real 4B graph/direct ON/OFF/eager gate passes 3/3, 1672/1672; H32/GDN tests 10/10 flag and 66/66, 4242/4242 full; paged-attention tests 25/25, 454474/454474. Final 18-leg root /tmp/qwen35-main-final-fa2-20260725 plus stable vLLM confirmation: direct ON/OFF/vLLM-0.25 total 5769.99/5660.70/5849.80 tok/s, output 638.03/625.94/646.85, TPOT/ITL 43.72/43.84/38.55 ms, peak PSS 2.406/8.592/7.662 GiB, stable PSS 0.759/8.589/4.029 GiB, VRAM 12850.7/12843.3/12942.7 MiB. ON=OFF output IDs 128/128 every pair; ON is +1.93% total and cuts peak/stable PSS 72.0%/91.2%. H32 AOT/graph/FA2 A/B gains are +4.59%/+0.39%/+1.60%. Graph-node trace has 453 launches and 200972 child kernels; local FA2 is 180.28 us/call vs vLLM 178.40, so the remaining 0.9864x throughput and TPOT gap is host/engine-side. Sanitizer availability and external 27B/35B remain open |
plain-BF16 direct-load spike; 2026-07-25 evidence | GATING |
- |
ENG-MOE-HOSTFREE |
MoE Marlin resident host-weight release: after BuildMoeMarlinResident uploads+repacks the routed experts to the device Marlin resident, free the per-expert fp4 HOST mirror (OwnedTensor packed+scale bytes) and madvise(MADV_DONTNEED) the pages back to the OS — returns the ~16.9 GiB steady 35B host double-store (LoadNvfp4Raw MakeOwned copies kept resident forever). Guarded to the committed Marlin path (MarlinMoeEnabled(); retained for the VT_NVFP4_MARLIN=0 wmma fallback that re-reads them); VT_MOE_HOST_FREE=0 A/B rollback. Realizes release_host_weights_after_upload for the dominant host consumer. Item-2 (2026-07-19, CLAIM-BACKEND-PLATFORM-2): the host-free decision is now CONSUMED from GetPlatform(d.q.device.type).residency_policy() via vllm::platforms::ShouldReleaseHostWeights(policy, MarlinMoeEnabled()/*kernel-path*/, VT_MOE_HOST_FREE/*env*/) — CudaPlatform flag flipped false→true (reproduces today EXACTLY); MarlinMoeEnabled() stays the orthogonal KERNEL-PATH safety gate. |
T0 | streamed target-device weight construction vllm/model_executor/model_loader/base_loader.py:43-82; residency capability vllm/platforms/interface.py:134-229; Marlin MoE in-place repack (no host mirror) vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py:375-434 |
free region src/vllm/model_executor/models/qwen3_5.cpp:3743-3781; OwnedTensor::ReleaseHost decl include/vllm/model_executor/models/qwen3_5_weights.h:65 + impl (madvise+swap) src/vllm/model_executor/models/qwen3_5_weights.cpp:24; public hook Qwen3_5Model::PrepareMarlinResident src/vllm/model_executor/models/qwen3_5.cpp:4472 |
tests/vllm/test_qwen36_weights.cpp:273 (ReleaseHost frees buffer+capacity) + :314 (PrepareMarlinResident release under Marlin / retention under VT_NVFP4_MARLIN=0, DGX release 27/27 + retention 15/15); DGX A/B (VT_MOE_HOST_FREE) 35B STEADY serving PSS 20.17→3.53 GiB (root dgx:~/work/mem35-hostfree); token-neutral 315/315 + 235/235; c2 smoke clean; memcheck clean. Whole-window load-phase PEAK bounded by the ENG-MOE-LOADSTREAM follow-up. Ledger parity-ledger.md#L521 |
moe-marlin-host-free.md | DONE |
ac77bec |
ENG-MOE-LOADSTREAM |
35B load-phase PEAK-PSS interleave (follow-up to ENG-MOE-HOSTFREE): the steady free returns the routed-expert host mirror only AFTER the whole model loads, so whole-window peak_pss/peak_rss (~19.8 GiB) is still set by all N layers' ~256 experts host-coexisting at load. DEFER the routed-expert host copies (LoadQwen3_5Moe loads each layer WITHOUT experts + installs a per-layer load_layer_experts streaming closure that owns the mmap'd shards) and materialize ONE layer's experts inside PrepareMarlinResident immediately before that layer's device Marlin build + host free — so at most one layer's experts coexist on the host (peak ~ one layer, not all N). Device residents byte-identical (same source bytes, same per-layer build order). Non-CUDA/VT_NVFP4_MARLIN=0/no-Marlin build falls back to bulk host materialization (their forward reads the host bytes). 27B is a different loader (LoadQwen3_5Dense, true-W4A4) → unaffected; GGUF/synthetic/borrowed pass no shards owner → eager. Item-2 (2026-07-19, CLAIM-BACKEND-PLATFORM-2): the per-layer interleave gate is now CONSUMED from GetPlatform(queue.device.type).residency_policy() via vllm::platforms::ShouldInterleaveLoadStream(policy, MarlinMoeEnabled()) — reproduces the old queue.device.type != kCUDA OR !MarlinMoeEnabled() gate EXACTLY (unified/CPU retain-host ⇒ policy false ⇒ materialize-all fallback); the ~4 GiB load-peak win is preserved. |
T0 | streamed target-device construction vllm/model_executor/model_loader/base_loader.py:43-82; in-place Marlin repack, no host mirror vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py:375-434; residency capability vllm/platforms/interface.py:134-229 |
deferred field include/vllm/model_executor/models/qwen3_5_weights.h:307 (Qwen3_5MoeWeights::load_layer_experts); loader defer + closure src/vllm/model_executor/models/qwen3_5_weights.cpp:331,399 (LoadMoeExpertsInto/LoadQwen3_5Moe); shared shards owner include/vllm/model_executor/models/model_registry.h:60 + src/vllm/model_executor/models/model_registry.cpp:411 (ModelSource::FromSafetensorsOwned) + src/vllm/entrypoints/model_loader.cpp:365 (LoadFromDir); per-layer interleave + MaterializeAllDeferredExperts src/vllm/model_executor/models/qwen3_5.cpp:4498,4508 (PrepareMarlinResident) |
CPU coexistence-bound contract tests/vllm/test_qwen36_weights.cpp:324 ("deferred routed-expert load: move-safe closure + bounded coexistence", peak==1 across the per-layer materialize→free loop, move-safe closure); clean -Werror CPU build 0 warn, full ctest (3 HTTP/engine-proc parallel-port flakes pass isolated), tools 164/164. DGX PROVEN (~/work/vllm.cpp-mem35-loadstream new vs -parent 7a1a6d6 eager, production flags CUTLASS sm120a+Marlin+FA2 sm_121a, one flock): 35B load-to-ready peak RSS (VmHWM) 21.43 GiB → 4.19 GiB (−17.24 GiB / −80%, below vLLM 13.3 GiB); token BYTE-IDENTICAL both binaries — 35B test_qwen36_paged_engine 315/315 + 27B test_qwen27_paged_engine 235/235; 27B UNAFFECTED (peak RSS 24.8 GiB, matches its baseline — dense loader, no deferral); compute-sanitizer memcheck on the deferred load path 0 errors / 315 assertions (no use-after-free of the freed host bytes); weights unit 127 assertions (CPU coexistence peak==1 + DGX residency). benchmark_binding=false — the orchestrator re-grids the binding peak_pss/peak_rss axes to confirm the FAIL→PASS flip |
moe-expert-load-stream.md | ACTIVE |
CLAIM-MEM35-LOADSTREAM |
LOAD-GGUF |
GGUF reader, dequantization, Qwen name transforms, embedded vocabulary | T0 | Pinned vLLM has no GGUF loader: vllm/model_executor/model_loader/__init__.py:31-65; compatibility reference is llama.cpp |
src/vllm/model_executor/model_loader/gguf_reader.cpp:302; src/vllm/model_executor/model_loader/gguf_dequant.cpp:223; src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:432; src/vllm/entrypoints/model_loader.cpp:240 |
tests/vllm/test_gguf.cpp:53; tests/vllm/test_gguf_dequant.cpp:25; tests/vllm/test_gguf_qwen36_loader.cpp:153; real 35B tests/parity/test_qwen36_gguf_engine.cpp:145 |
planned: specs/gguf-loader.md |
PARTIAL |
- |
LOAD-HF-BPE |
HF tokenizer.json byte-level BPE and incremental detokenization | T0 | vllm/tokenizers/registry.py:176; vllm/tokenizers/hf.py:163; tests/tokenizers_/test_hf.py:18; tests/tokenizers_/test_detokenize.py:148 |
src/vllm/tokenizer/tokenizer.cpp:240,484,512; src/vllm/v1/engine/detokenizer.cpp:409 |
tests/vllm/test_bpe.cpp:226; tests/vllm/test_tokenizer_parity.cpp:66,74,82,90; tests/vllm/test_pretokenizer.cpp:130 |
planned: specs/hf-tokenizer.md |
ANCHOR-BACKFILL |
- |
LOAD-SENTENCEPIECE |
SentencePiece (Metaspace + byte-fallback) BPE tokenizer family — the gate to Mistral/Gemma/SentencePiece tokenizer.json. Tokenizer::FromHfJson now dispatches on the pre_tokenizer FAMILY: a bare Metaspace node selects the SP path (space→▁ U+2581; prepend_scheme first/always/never with the starts-with-▁ guard; split=false; out-of-vocab char → <0xNN> byte-fallback tokens; merge-ranked BPE over the raw-UTF-8 string), else the BYTE-IDENTICAL byte-level path (DetectPattern still fails loudly on Metaspace, so families never overlap). Decode + the incremental detokenizer mirror HF's Sequence decoder (Replace ▁→space, ByteFallback, Fuse, Strip 1 leading space). UNBLOCKS the Mistral (MODEL-TEXT-mistral-mistral-for-causal-lm) paged-engine SACRED gate. split=true fails loudly (no golden in scope). Mirrors HF tokenizers 0.22 |
T1 | vllm/tokenizers/hf.py:163; vllm/tokenizers/mistral.py:235,467; HF tokenizers 0.22 pre_tokenizers/metaspace.rs, models/bpe/model.rs::merge_word, decoders/{replace,byte_fallback,fuse,strip}.rs |
dispatch src/vllm/tokenizer/tokenizer.cpp:184 (DetectMetaspace), :732 (EncodePlainSp), :878 (SpDecodeTokens), :927 (Decode SP branch); merge-loop factor src/vllm/tokenizer/bpe.cpp:91 (BpeMerge); family/params/accessors include/vllm/tokenizer/tokenizer.h:88,98; incremental dispatch src/vllm/v1/engine/detokenizer.cpp:357; generator tools/parity/dump_tokenizer_mistral.py |
tests/vllm/test_tokenizer_parity_mistral.cpp 6/6, 421 assertions byte-exact vs HF tokenizers 0.22.2 (= vLLM 0.25.0 backend) over a 45-entry Metaspace/byte-fallback/special-token corpus (goldens tests/parity/goldens/tokenizer_mistral/{tokenizer.json,encodings.json}); SACRED cross-check vLLM AutoTokenizer (transformers 5.13.1) 0/45 mismatch (±BOS); byte-level suites byte-identical tests/vllm/test_bpe.cpp 852, tests/vllm/test_detokenizer.cpp 221, tests/vllm/test_tokenizer_parity.cpp 1175, tests/vllm/test_tokenizer_parity_deepseek.cpp 2461 |
specs/sentencepiece.md | ACTIVE |
CLAIM-LOAD-SENTENCEPIECE |
LOAD-CONFIG-SURFACE |
Dataclass-for-dataclass config and serve-compatible flags | T0/T1 | vllm/config/scheduler.py:26; vllm/config/cache.py:25; vllm/config/compilation.py:378 |
include/vllm/config/scheduler.h:67; src/vllm/config/scheduler.cpp:11; src/vllm/transformers_utils/hf_config.cpp:83; limited flags incl. max_num_seqs/max_num_batched_tokens examples/server/main.cpp:63,116,170 |
tests/vllm/test_scheduler_config.cpp:10; tests/vllm/test_hf_config.cpp:131,224,265; examples/CMakeLists.txt:34 |
planned: specs/config-surface.md |
PARTIAL |
- |
LOAD-LONGTAIL |
Sharded-state, tensorizer, RunAI, BitsAndBytes loaders | T3 | vllm/model_executor/model_loader/__init__.py:33-65 |
- | - | planned: specs/loader-longtail.md |
INVENTORIED |
- |
Claims use the stable ID above. An agent first commits the row's leaf spike,
updates this matrix to READY, then claims implementation in
coordination.md. The umbrella anchor-backfill inventory does
not replace any leaf spike. DONE additionally requires merged code, ported
upstream tests, end-to-end oracle evidence where applicable, and same-change
roadmap/README/ledger/state updates.