spec: add DSpark speculative decoding

[wjinxu]

This PR adds DSpark speculative decoding, layered on the merged DFlash drafter. DSpark (DeepSeek + PKU, 2026 — "Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation", the DeepSpec repo) is DFlash plus a small semi-autoregressive Markov head: where DFlash takes an independent argmax at each block position (every position marginalizes over all possible predecessors, so acceptance decays along the block), DSpark adds a low-rank, previous-token-conditioned logit bias and samples the block left-to-right, so each draft conditions on the one actually sampled before it. This lifts accepted length at near-zero extra draft cost.

DSpark reuses the entire DFlash machinery unchanged — the encoder/decoder graph, target-layer feature extraction (llama_set_embeddings_layer_inp / _nextn), KV-cache injection, and the verify/accept path. The only additions are:

• a new draft architecture dspark (llama_model_dspark : llama_model_dflash) that reuses the DFlash graph and additionally loads the Markov head (markov_w1, markov_w2) and an optional confidence head; it shares the target's token-embeddings / lm_head (same as DFlash);
• a new speculative type draft-dspark (common_speculative_impl_draft_dspark : common_speculative_impl_draft_dflash) that reuses process() (extraction + injection) and overrides only draft(): the block is anchor-first (position 0 already predicts the first draft token) and sampled with the Markov bias bias(prev) = markov_w2 · markov_w1[prev], computed on-device (llama_dspark_markov_bias);
• a Qwen3DSparkModel converter.

Greedy decoding is lossless: the Markov bias only changes which tokens are proposed; every draft is still verified against the target, so the output is identical to non-speculative greedy.

The confidence head is converted/loaded but not used at inference in this PR (phase 1); the draft-quality win from the Markov head is self-contained and is what the numbers below measure.

How to run

Complete example from scratch (Qwen3-8B). Drafts for other sizes are on the same org: deepseek-ai/dspark_qwen3_{4b,8b,14b}_block7.

1. Get the models — target + its DSpark draft:

huggingface-cli download Qwen/Qwen3-8B --local-dir Qwen3-8B
huggingface-cli download deepseek-ai/dspark_qwen3_8b_block7 --local-dir dspark_qwen3_8b

2. Convert to GGUF — the draft ships no tokenizer and reuses the target's, so pass --target-model-dir:

python convert_hf_to_gguf.py Qwen3-8B --outtype bf16 --outfile Qwen3-8B.gguf
python convert_hf_to_gguf.py dspark_qwen3_8b --outtype bf16 \
    --target-model-dir Qwen3-8B --outfile Qwen3-8B-DSpark.gguf

You may quantize the target (e.g. llama-quantize Qwen3-8B.gguf Qwen3-8B-Q4_K_M.gguf Q4_K_M); keep the draft bf16 — it's tiny, and acceptance is unaffected by target quant.

3. Build with CUDA:

cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

4. Run — the only DSpark-specific flags are -md <draft> and --spec-type draft-dspark
(--spec-draft-n-max = draft tokens per step; the released checkpoints use block size 7):

./build/bin/llama-server -m Qwen3-8B.gguf -md Qwen3-8B-DSpark.gguf \
    --spec-type draft-dspark --spec-draft-n-max 7 \
    --temp 0 --top-k 1 -np 1 -c 4096 -ngl 99 -fa on --jinja

5. Send a request (the server logs draft acceptance = ... per request):

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": "Explain the Pythagorean theorem."}],
  "temperature": 0, "max_tokens": 256
}'

llama-cli works the same way (-m ... -md ... --spec-type draft-dspark). Note: draft-dspark needs the target's hidden states (KV-cache injection), so use llama-server / llama-cli — the speculative-simple example does not drive that path.

Performance

SpeedBench (llama.cpp's own tools/server/bench/speed-bench)

Qwen3-8B (bf16), matched --spec-draft-n-max 7, qualitative split (11 categories), greedy. Baseline is the same server with no draft model. DSpark reaches 1.88× overall decode speedup vs baseline (DFlash is 1.55×), and beats the merged DFlash on every one of the 11 categories (overall 1.21×).

DSpark vs baseline:

category       base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
-------------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
coding         58.16              123.57             2.12x           11.172s           5.458s            2.05x            0.3219
humanities     58.22              99.06              1.70x           9.573s            5.646s            1.70x            0.2340
math           58.21              109.86             1.89x           10.313s           5.409s            1.91x            0.2840
qa             58.23              107.32             1.84x           8.313s            4.486s            1.85x            0.2659
rag            57.91              123.54             2.13x           9.521s            4.639s            2.05x            0.3264
reasoning      58.21              99.29              1.71x           9.570s            5.622s            1.70x            0.2347
stem           58.19              98.92              1.70x           8.827s            5.205s            1.70x            0.2332
writing        57.82              111.32             1.93x           9.765s            5.282s            1.85x            0.2807
multilingual   58.18              121.96             2.10x           8.691s            4.250s            2.05x            0.3187
summarization  58.36              102.74             1.76x           5.309s            3.001s            1.77x            0.2530
roleplay       58.20              102.56             1.76x           14.139s           8.274s            1.71x            0.2454
overall        58.15              109.10             1.88x           9.563s            5.207s            1.84x            0.2698

DSpark vs the merged DFlash (same --spec-draft-n-max 7):

category       dflash_avg_pred_t/s  dspark_avg_pred_t/s  decode_speedup  dflash_avg_latency  dspark_avg_latency  latency_speedup  accept_rate
-------------  -------------------  -------------------  --------------  ------------------  ------------------  ---------------  -----------
coding         106.00               123.57               1.17x           6.343s              5.458s              1.16x            0.3219
humanities     83.61                99.06                1.18x           6.674s              5.646s              1.18x            0.2340
math           90.48                109.86               1.21x           6.529s              5.409s              1.21x            0.2840
qa             85.20                107.32               1.26x           5.650s              4.486s              1.26x            0.2659
rag            98.61                123.54               1.25x           5.733s              4.639s              1.24x            0.3264
reasoning      83.51                99.29                1.19x           6.681s              5.622s              1.19x            0.2347
stem           83.60                98.92                1.18x           6.154s              5.205s              1.18x            0.2332
writing        90.28                111.32               1.23x           6.443s              5.282s              1.22x            0.2807
multilingual   102.94               121.96               1.18x           5.016s              4.250s              1.18x            0.3187
summarization  85.85                102.74               1.20x           3.606s              3.001s              1.20x            0.2530
roleplay       79.96                102.56               1.28x           10.451s             8.274s              1.26x            0.2454
overall        90.00                109.10               1.21x           6.298s              5.207s              1.21x            0.2698

Hardware: RTX 4090. Target Qwen/Qwen3-8B (bf16), draft deepseek-ai/dspark_qwen3_8b_block7 (bf16). Greedy (--temp 0 --top-k 1), no-thinking, --spec-draft-n-max 7. Baseline = same llama-server with no draft model. DFlash is the merged drafter (z-lab/Qwen3-8B-DFlash, b16), run at the same matched draft size for an apples-to-apples comparison. Per-domain aggregate over the listed prompt counts.

Losslessness

Greedy decoding is lossless by construction (the draft is verified against the target). Output is coherent and matches non-speculative greedy.

Qwen3-4B, target bf16

DSpark vs baseline (DFlash was not benchmarked at 4B — no nested-schema 4B DFlash draft available):

Domain	Baseline t/s	DSpark t/s (accept)	DSpark
GSM8K (40)	103.1	354.0 (75.3%)	3.43×
MATH500 (30)	102.9	341.3 (71.7%)	3.32×
HumanEval (30)	103.9	340.0 (72.9%)	3.27×
MBPP (30)	103.6	281.4 (57.2%)	2.72×
MT-Bench (30)	102.8	190.4 (31.7%)	1.85×
geomean			2.85×

Qwen3-8B, target bf16

Domain	Baseline t/s	DFlash t/s (accept)	DSpark t/s (accept)	DFlash	DSpark
GSM8K (40)	58.5	182.4 (53.7%)	237.3 (78.9%)	3.12×	4.06×
MATH500 (30)	58.5	195.7 (59.2%)	223.2 (72.8%)	3.35×	3.82×
HumanEval (30)	59.1	238.8 (77.2%)	241.4 (81.7%)	4.04×	4.08×
MBPP (30)	59.6	177.3 (53.3%)	193.1 (63.7%)	2.98×	3.24×
MT-Bench (30)	58.6	93.5 (19.7%)	120.4 (31.3%)	1.60×	2.05×
geomean				2.89×	3.35×

Qwen3-8B, target Q8_0

Domain	Baseline t/s	DFlash t/s (accept)	DSpark t/s (accept)	DFlash	DSpark
GSM8K (40)	100.6	246.4 (53.2%)	322.9 (77.8%)	2.45×	3.21×
MATH500 (30)	100.5	266.2 (59.2%)	305.7 (72.2%)	2.65×	3.04×
HumanEval (30)	101.3	319.5 (76.5%)	327.9 (81.4%)	3.15×	3.24×
MBPP (30)	102.2	242.4 (54.3%)	268.0 (64.3%)	2.37×	2.62×
MT-Bench (30)	100.7	126.7 (19.3%)	167.8 (31.4%)	1.26×	1.67×
geomean				2.28×	2.68×

Qwen3-8B, target Q4_K_M

Domain	Baseline t/s	DFlash t/s (accept)	DSpark t/s (accept)	DFlash	DSpark
GSM8K (40)	155.4	259.0 (52.9%)	340.7 (77.4%)	1.67×	2.19×
MATH500 (30)	155.2	284.9 (60.4%)	326.1 (73.9%)	1.84×	2.10×
HumanEval (30)	156.5	314.3 (71.1%)	332.0 (78.4%)	2.01×	2.12×
MBPP (30)	157.5	257.5 (55.5%)	281.3 (66.0%)	1.63×	1.79×
MT-Bench (30)	155.5	135.2 (19.7%)	174.4 (30.6%)	0.87×	1.12×
geomean				1.54×	1.81×

DSpark beats the merged DFlash on every domain (higher accept rate and higher throughput), for a ~1.16× geomean speedup over DFlash. The gains are largest on reasoning (GSM8K +25pp accept, 1.30× over DFlash) and open chat (MT-Bench, 1.29×); on code (HumanEval) the two are close as both already accept ~80%.

Confidence Evaluation

Qwen3-8B Q4_K_M target, SPEED-Bench qualitative, 132 completed samples, OSL 512.

Concurrency	Unified KV	conf_min	Average decode t/s	Average latency	Acceptance	Total elapsed
1	No	0.0	198.14	2.171 s	36.6%	286.58 s
1	Yes	0.0	195.64	2.193 s	36.6%	289.53 s
1	Yes	0.3	190.16 (-2.8%)	2.276 s (+3.8%)	41.6%	300.52 s
1	Yes	0.6	191.74 (-2.0%)	2.189 s (-0.2%)	62.5%	289.04 s
8	No	0.0	61.43	7.014 s	35.7%	119.68 s
8	Yes	0.0	54.81	7.621 s	35.1%	129.49 s
8	Yes	0.3	56.98 (+4.0%)	7.425 s (-2.6%)	42.4%	126.65 s
8	Yes	0.6	59.01 (+7.7%)	7.360 s (-3.4%)	62.7%	125.59 s
32	No	0.0	16.20	24.662 s	31.3%	122.29 s
32	Yes	0.0	15.17	28.326 s	32.1%	133.05 s
32	Yes	0.3	15.53 (+2.4%)	27.341 s (-3.5%)	40.9%	129.85 s
32	Yes	0.6	17.26 (+13.8%)	24.511 s (-13.5%)	59.8%	117.40 s

Percentage changes on the unified-KV rows are relative to the same-concurrency unified-KV conf_min=0.0 baseline.

Confidence pruning has no benefit at concurrency 1, begins to help at concurrency 8.The intended operating environment is high-concurrency serving with packed/unified KV batching.

Do not enable confidence pruning with non-unified KV at high concurrency. Ragged verification causing CUDA Graph reuse to collapse.

Future work

• Confidence head (phase 2): wire the confidence-scheduled prefix pruning, with the paper's Sequential Temperature Scaling calibration. The big serving win in the paper comes from the batched scheduler, which is a separate, larger change.
• Markov-bias graph reuse: the bias is computed as a tiny per-step ggml graph on the draft context's scheduler; building it once per block and re-running with new inputs would cut overhead. A fused bias+argmax kernel is a further option but would add a backend-specific op (the current path is pure ggml, no new operator).

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, use Claude to help discuss and design the DSpark architecture, ask clarifying questions, and assist with writing tests. Everything remains under my control, and I reviewed every single line of AI-generated code.

[wjinxu]

Hi @CISC @ggerganov , this adds DSpark speculative decoding on top of the merged DFlash drafter (#22105). It's a small change — a new dspark draft arch and draft-dspark spec type that reuse DFlash's graph, feature extraction, KV-cache injection and verify path unchanged; the only new logic is the semi-autoregressive Markov head in draft(). Greedy stays lossless.

I benchmarked it against the merged DFlash using DeepSeek's released Qwen3 DSpark drafts. On Qwen3-8B at bf16 / Q8_0 / Q4_K_M, DSpark beats DFlash on every domain (e.g. GSM8K bf16 4.06× vs 3.12×; full per-domain tables in the PR description).

I believe it's ready for review and I'm happy to walk through any part of it.

[ruixiang63]

Can you run SpeedBench to do the full comparison between DFlash and DSpark with the same --spec-draft-n-max? https://github.com/ggml-org/llama.cpp/tree/master/tools/server/bench/speed-bench

[wjinxu]

@ruixiang63 I've run the SpeedBench test set as you suggested, and updated the results in the PR description. DSpark does outperform DFlash across the board.

[nipeone]

could you give some examples how to use?

[wjinxu]

could you give some examples how to use?

Good point — I've updated the PR description with a more detailed, copy-pasteable end-to-end example (download → convert → build → run → curl). Let me know if anything's unclear.

[am17an]

DSV4 support was merged in #24162, ideally this PR should cover that model as well and try to replicate a similar speedup

[wjinxu]

DSV4 support was merged in #24162, ideally this PR should cover that model as well and try to replicate a similar speedup

Thanks! DeepSeek hasn't open-sourced the DSpark weights for DeepSeek-V4 though — only the Qwen3 and Gemma4 drafts are released. So this PR covers Qwen3 for now, and I'll add Gemma4 as a small follow-up.

[am17an]

I think they're a part of the spec decoding module https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark i.e not distributed separately

[wjinxu]

I think they're a part of the spec decoding module https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark i.e not distributed separately

Sorry, and thanks for the heads-up. For this PR I'd like to keep the scope a bit narrower for now - land the Qwen3 DSpark path first and get it solid, then add Gemma4 and DSV4 as follow-ups. Does that sound ok?

[am17an]

Okay, will try to review. From a cursory look it does not look like adding llama_dspark* is the right thing to do. Honestly don't have a good feeling about the PR, too much AI code.

[wjinxu]

Okay, will try to review. From a cursory look it does not look like adding llama_dspark* is the right thing to do. Honestly don't have a good feeling about the PR, too much AI code.

I agree that llama_dspark_* shouldn't be part of the API. The issue is that the Markov bias computation (W2 @ W1[prev]) needs the draft weights and has to run on the backend - I measured it, and doing it on the host is too slow. But the DSpark drafter lives in common/, which can't reach the draft weights, so once the API is removed there's no way to trigger that computation from common/.

[wjinxu]

I folded the DSpark impl into the DFlash one: the duplicated draft() is gone, and a shifted member (derived from the existing spec type, i.e. draft-dspark vs draft-dflash) now selects the block layout — anchor-first blocks read predictions from position 0 and always submit a whole block, in-place blocks keep the old behavior. Happy to adjust if you'd prefer a different approach.

I also wired up the confidence head behind a new launch-time flag (--spec-draft-conf-min): it prunes each drafted block at the first position whose predicted acceptance falls below the threshold, default 0 = disabled. This is the static-threshold variant. I'll update the PR description with benchmark numbers shortly.

[ruixiang63]

It would be better to touch the current DFlash code as little as possible to avoid any potential breakage, and use is_dspark as the condition where needed.

[AndrewDBN]

I have rebuilt wjinxu's llama.cpp (dspark-upstream branch) and model loading broke after the latest commits :
error loading model: unknown model architecture: 'dspark'.
See below for complete logs.

1. No errors during compilation.
2. No errors loading qwen3-8b without the dspark draft model.
3. Error loading qwen3-8b with the dspark draft model:

andrew@andrew-workstation:~$ CUDA_VISIBLE_DEVICES=1 ~/projects/llama.cpp/build/bin/llama-server   -m ~/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf -md ~/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 7 --port 8081  -fitt 768 -fa on -np 1 --cache-type-k q4_0   --cache-type-v q4_0   --threads 12   --batch-size 2048   --metrics   --no-mmap   --mlock   --no-warmup  --temp 0.6   --top-p 0.95   --top-k 20   --min-p 0.0   --presence-penalty 0.0   --repeat-penalty 1.0 --tools all --webui-mcp-proxy --jinja
0.00.157.570 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.233.092 W srv  llama_server: -----------------
0.00.233.093 W srv  llama_server: CORS proxy is enabled, do not expose server to untrusted environments
0.00.233.094 W srv  llama_server: This feature is EXPERIMENTAL and may be removed or changed in future versions
0.00.233.094 W srv  llama_server: -----------------
0.00.233.100 W srv  llama_server: -----------------
0.00.233.100 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.233.100 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.233.100 W srv  llama_server: -----------------
0.00.234.293 I srv    load_model: loading model '/home/andrew/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf'
0.00.282.487 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.00.282.493 E llama_model_load_from_file_impl: failed to load model
0.00.282.526 W srv    load_model: [spec] failed to measure draft model memory: failed to load model
0.00.514.154 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.475.396 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.01.475.399 E llama_model_load_from_file_impl: failed to load model
0.01.475.400 E common_speculative_init_result: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.475.405 E srv    load_model: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.475.408 I srv    operator(): operator(): cleaning up before exit...
0.01.476.031 E srv  llama_server: exiting due to model loading error

4. Error also with -fit off:

andrew@andrew-workstation:~$ CUDA_VISIBLE_DEVICES=1 ~/projects/llama.cpp/build/bin/llama-server   -m ~/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf -md ~/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 7 --port 8081 -fit off -fa on -np 1 --cache-type-k q4_0   --cache-type-v q4_0   --threads 12   --batch-size 2048   --metrics   --no-mmap   --mlock   --no-warmup  --temp 0.6   --top-p 0.95   --top-k 20   --min-p 0.0   --presence-penalty 0.0   --repeat-penalty 1.0 --tools all --webui-mcp-proxy --jinja
0.00.126.695 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.209.392 W srv  llama_server: -----------------
0.00.209.394 W srv  llama_server: CORS proxy is enabled, do not expose server to untrusted environments
0.00.209.394 W srv  llama_server: This feature is EXPERIMENTAL and may be removed or changed in future versions
0.00.209.394 W srv  llama_server: -----------------
0.00.209.400 W srv  llama_server: -----------------
0.00.209.400 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.209.400 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.209.400 W srv  llama_server: -----------------
0.00.210.682 I srv    load_model: loading model '/home/andrew/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf'
0.00.306.346 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.206.247 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.01.206.253 E llama_model_load_from_file_impl: failed to load model
0.01.206.254 E common_speculative_init_result: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.206.258 E srv    load_model: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.206.262 I srv    operator(): operator(): cleaning up before exit...
0.01.207.146 E srv  llama_server: exiting due to model loading error

[wjinxu]

I have rebuilt wjinxu's llama.cpp (dspark-upstream branch) and model loading broke after the latest commits :我已重新构建了 wjinxu 的 llama.cpp（dspark-upstream 分支），但在最新提交后模型加载出现了问题： error loading model: unknown model architecture: 'dspark'. See below for complete logs.完整日志见下方。

1. No errors during compilation.编译过程中未出现错误。
2. No errors loading qwen3-8b without the dspark draft model.加载 qwen3-8b 时，若未使用 dspark 草稿模型，则无错误。
3. Error loading qwen3-8b with the dspark draft model:加载 qwen3-8b 时，若使用 dspark 草稿模型，则出现错误：

andrew@andrew-workstation:~$ CUDA_VISIBLE_DEVICES=1 ~/projects/llama.cpp/build/bin/llama-server   -m ~/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf -md ~/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 7 --port 8081  -fitt 768 -fa on -np 1 --cache-type-k q4_0   --cache-type-v q4_0   --threads 12   --batch-size 2048   --metrics   --no-mmap   --mlock   --no-warmup  --temp 0.6   --top-p 0.95   --top-k 20   --min-p 0.0   --presence-penalty 0.0   --repeat-penalty 1.0 --tools all --webui-mcp-proxy --jinja
0.00.157.570 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.233.092 W srv  llama_server: -----------------
0.00.233.093 W srv  llama_server: CORS proxy is enabled, do not expose server to untrusted environments
0.00.233.094 W srv  llama_server: This feature is EXPERIMENTAL and may be removed or changed in future versions
0.00.233.094 W srv  llama_server: -----------------
0.00.233.100 W srv  llama_server: -----------------
0.00.233.100 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.233.100 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.233.100 W srv  llama_server: -----------------
0.00.234.293 I srv    load_model: loading model '/home/andrew/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf'
0.00.282.487 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.00.282.493 E llama_model_load_from_file_impl: failed to load model
0.00.282.526 W srv    load_model: [spec] failed to measure draft model memory: failed to load model
0.00.514.154 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.475.396 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.01.475.399 E llama_model_load_from_file_impl: failed to load model
0.01.475.400 E common_speculative_init_result: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.475.405 E srv    load_model: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.475.408 I srv    operator(): operator(): cleaning up before exit...
0.01.476.031 E srv  llama_server: exiting due to model loading error

4. Error also with -fit off:即使关闭 -fit 选项，错误依然存在：

andrew@andrew-workstation:~$ CUDA_VISIBLE_DEVICES=1 ~/projects/llama.cpp/build/bin/llama-server   -m ~/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf -md ~/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 7 --port 8081 -fit off -fa on -np 1 --cache-type-k q4_0   --cache-type-v q4_0   --threads 12   --batch-size 2048   --metrics   --no-mmap   --mlock   --no-warmup  --temp 0.6   --top-p 0.95   --top-k 20   --min-p 0.0   --presence-penalty 0.0   --repeat-penalty 1.0 --tools all --webui-mcp-proxy --jinja
0.00.126.695 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.209.392 W srv  llama_server: -----------------
0.00.209.394 W srv  llama_server: CORS proxy is enabled, do not expose server to untrusted environments
0.00.209.394 W srv  llama_server: This feature is EXPERIMENTAL and may be removed or changed in future versions
0.00.209.394 W srv  llama_server: -----------------
0.00.209.400 W srv  llama_server: -----------------
0.00.209.400 W srv  llama_server: Built-in tools are enabled, do not expose server to untrusted environments
0.00.209.400 W srv  llama_server: This feature is EXPERIMENTAL and may be changed in the future
0.00.209.400 W srv  llama_server: -----------------
0.00.210.682 I srv    load_model: loading model '/home/andrew/Qwen3-8B/Qwen3-8B-Q4_K_M.gguf'
0.00.306.346 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.206.247 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.01.206.253 E llama_model_load_from_file_impl: failed to load model
0.01.206.254 E common_speculative_init_result: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.206.258 E srv    load_model: failed to load draft model, '/home/andrew/dspark_qwen3_8b/Qwen3-8B-DSpark.gguf'
0.01.206.262 I srv    operator(): operator(): cleaning up before exit...
0.01.207.146 E srv  llama_server: exiting due to model loading error

The latest commits may have made older GGUF files incompatible; could you try reconverting the model and let me know if the issue persists?

[AndrewDBN]

The latest commits may have made older GGUF files incompatible; could you try reconverting the model and let me know if the issue persists?

@wjinxu Yes! Thank you! Reconverting the hf draft model file to GGUF fixed the problem!

./python ~/projects/llama.cpp/convert_hf_to_gguf.py ~/dspark_qwen3_8b --outtype bf16 --target-model-dir ~/Qwen3-8B --outfile Qwen3-8B-DSpark.gguf
INFO:hf-to-gguf:Loading model: dspark_qwen3_8b
INFO:hf-to-gguf:Model architecture: Qwen3DSparkModel
...

Please let me know if there is anything I can help you with to implement DSpark speculative decoding for the Gemma-4 LLMs.

[wjinxu]

Please let me know if there is anything I can help you with to implement DSpark speculative decoding for the Gemma-4 LLMs.

I plan to open a draft PR tomorrow with Gemma 4 support for testing.

[ruixiang63]

It looks good to me now. I made a few cleanup changes, please test on your side to see if anything breaks. @wjinxu

@ggerganov Could you take another look at the current changes, especially build_dspark_markov_head in dflash.cpp?

[wjinxu]

It looks good to me now. I made a few cleanup changes, please test on your side to see if anything breaks.

I reran the validation for both DSpark and DFlash on the GSM8K dataset, and both produced the expected results.

[FHRacing]

What kind of actual performance gains will we see from Dspark spec-coding? Obviously we won't be seeing their gains, although i have seen some demos going over the % they showed, but if it's anywhere close to the claim, that would be insane.

[Ankk98]

Tested llama.cpp PR #25173 on AMD Strix Halo iGPU (Vulkan): Qwen3-8B Q4_K_M target vs DSpark draft (bf16 and Q4_K_M), --spec-draft-n-max 2/3/7, greedy decode, thinking on/off. Script: https://gist.github.com/Ankk98/ee29401e4dcdcae575e649c4a0b75d9e

Speed: No config beat vanilla (~45 t/s). Best geomean was ~0.81x (dspark-q4-n2/n3). Even ~93–98% accept only reached ~0.8–0.9x. Larger n-max (7) hurt when accept dropped (~23–45%). Q4 draft helped vs bf16 but still slower overall.

Correctness: Repeated greedy mismatches vs vanilla (BUG / TRUNC_BUG), especially open-ended prompts and thinking, probably a bug in llama.cpp master itself.

Learnings: Spec decode wins when draft < target. On unified-memory Vulkan that ratio is weak (heavy draft + target share bandwidth), so RTX 4090 CUDA gains do not carry over. Prefer smaller n-max and quantized drafts on iGPU; validate losslessness under greedy before chasing speed.

[AndrewDBN]

What kind of actual performance gains will we see from Dspark spec-coding?

@FHRacing
From my experiments, the gains in performance (we are talking gains in token generation speed during inference, measured in tok/s) from speculative decoding of any type (MTP, DFlash, DSpark, etc) are highly variable, depending on:

• The LLM, obviously.
• The hardware configuration, also obviously.
• The benchmarks/prompts/tasks submitted to the LLM.
• The level of optimization of the speculative draft model (or assistant model in Google-speak).
• The speculative decoding algorithm/mechanism.
• Finally, the implementation (coding) of the speculative decoding algorithm.

In other words, imho it's impossible to generalize, each individual case will see different performance gains (or no gains at all, or losses). You just have to try it for yourself and see what works for you.

[wjinxu]

Correctness: Repeated greedy mismatches vs vanilla (BUG / TRUNC_BUG), especially open-ended prompts and thinking, probably a bug in llama.cpp master itself.

I also ran tests on CUDA and encountered the same issue, but I didn't see this problem when using BF16 precision. I suspect it might be a precision issue specific to Q4_K_M — you could try testing with BF16 precision as well to confirm.

Speed: No config beat vanilla (~45 t/s). Best geomean was ~0.81x (dspark-q4-n2/n3). Even ~93–98% accept only reached ~0.8–0.9x. Larger n-max (7) hurt when accept dropped (~23–45%). Q4 draft helped vs bf16 but still slower overall.

I also noticed that on Q4_K_M, the speedup isn't very noticeable, because the DSpark draft model isn't actually that small, so the benefit it brings on Q4_K_M is limited.

[Ankk98]

@wjinxu Did some more investigation and filed a separated issue for divergence bug. This bug threw me offcourse in my attempt to implement dspark speculation as well. I think I am close to finding the root cause. I will create a separate PR for this.

[YanissAmz]

Hi @wjinxu — I've been building on this branch to support the DSpark drafter that ships with a DeepSeek-V4 backbone (drafting for DeepSeek-V4-Flash), and I hit a bug that lives in the shared dflash path rather than in my backbone, so it seemed worth reporting here before I open anything.

llama_model_rope_type() returns LLAMA_ROPE_TYPE_NEOX for LLM_ARCH_DFLASH. That is correct for the Qwen3-style drafter this PR targets, but it is baked into the architecture rather than into the backbone, and a drafter has to rotate exactly like the target it drafts for. A DeepSeek-V4 backbone pairs the rotary dimensions the interleaved way (LLAMA_ROPE_TYPE_NORM, as deepseek2/deepseek4 do), so it silently gets the wrong pairing.

What makes it nasty is the failure mode. It does not crash and it does not produce garbage output: block position 0 stays correct, because it is carried by the injected target hidden state and the shared lm_head, so the model keeps generating perfectly reasonable text. The deeper block positions quietly lose their positional signal and start repeating the earlier ones, and acceptance just stalls at ~2 tokens — which reads like "the drafter is mediocre" rather than "the drafter is broken". Selecting the pairing from the loaded backbone takes mean accepted length from 2.0 to 5.2–6.0 (block size 5) on my setup, with the flat per-position acceptance curve the paper describes.

Three other things I had to fix in the shared path while getting this working, all of which apply to the Qwen drafter too:

• the draft sampler samples a token and then discards it, reading cur_p->data[0] instead — which silently forces greedy regardless of the sampler chain;
• the block's noise K/V from the previous iteration is not purged from the drafter's cache, and since the block runs non-causally the next block attends to it;
• the target hidden states are injected on top of a draft region that has not been purged.

Happy to send these as a small PR against your branch if that is easiest for you, or to fold them into the DeepSeek-V4 PR I'm preparing (which is stacked on this one) and let you cherry-pick — whichever you prefer. Nice work on this, the speculator structure made the backbone swap much easier than I expected.

[pwilkin]

@YanissAmz that's actually a subtle one and I'm wondering if it doesn't affect the other "derived architectures": EAGLE and DFlash. @ruixiang63 maybe you have an idea?

[AndrewDBN]

The deeper block positions quietly lose their positional signal and start repeating the earlier ones, and acceptance just stalls at ~2 tokens — which reads like "the drafter is mediocre" rather than "the drafter is broken".

@YanissAmz
I tested wjinxu's DSpark implementation for Gemma4 (see PR #25549 and my comments in it) and I observed very low acceptance rates, which I indeed attributed to a low-quality drafter model. Your explanation however makes a lot of sense.

[wjinxu]

llama_model_rope_type() returns LLAMA_ROPE_TYPE_NEOX for LLM_ARCH_DFLASH. That is correct for the Qwen3-style drafter this PR targets, but it is baked into the architecture rather than into the backbone, and a drafter has to rotate exactly like the target it drafts for. A DeepSeek-V4 backbone pairs the rotary dimensions the interleaved way (LLAMA_ROPE_TYPE_NORM, as deepseek2/deepseek4 do), so it silently gets the wrong pairing.

What makes it nasty is the failure mode. It does not crash and it does not produce garbage output: block position 0 stays correct, because it is carried by the injected target hidden state and the shared lm_head, so the model keeps generating perfectly reasonable text. The deeper block positions quietly lose their positional signal and start repeating the earlier ones, and acceptance just stalls at ~2 tokens — which reads like "the drafter is mediocre" rather than "the drafter is broken". Selecting the pairing from the loaded backbone takes mean accepted length from 2.0 to 5.2–6.0 (block size 5) on my setup, with the flat per-position acceptance curve the paper describes.

This is indeed an issue that neither Gemma4 nor Qwen3 has, but perhaps we could unify it to NEOX during the DeepSeekV4 conversion? I noticed that eagle also uses the same approach.

• the block's noise K/V from the previous iteration is not purged from the drafter's cache, and since the block runs non-causally the next block attends to it;
• the target hidden states are injected on top of a draft region that has not been purged.

I found that there is cleanup in the server.

[ruixiang63]

the draft sampler samples a token and then discards it, reading cur_p->data[0] instead — which silently forces greedy regardless of the sampler chain;

This is expected as draft model is alwasy greedy.

the block's noise K/V from the previous iteration is not purged from the drafter's cache, and since the block runs non-causally the next block attends to it;

the target hidden states are injected on top of a draft region that has not been purged.

This shouldn't be the case. All past KV caches are either purged or restored.

This is indeed an issue that neither Gemma4 nor Qwen3 has, but perhaps we could unify it to NEOX during the DeepSeekV4 conversion? I noticed that eagle also uses the same approach.

I agree. This can be handled as the same way as eagle3 does.

[pwilkin]

Ah, okay, you reorder the tensors to NEOX during the conversion process.

[wjinxu]

Hi @ggerganov, could you review this PR whenever you get a chance? Thanks in advance!