Fix Minimax 2 EAGLE3 support

[adrianisk]

Overview

Fixes a nullptr assert when trying to use an EAGLE3 draft model with MiniMax-M2.7.

I wanted to try using MiniMax-M2.7-EAGLE3-draft-vocab32k with AesSedai's MiniMax-M2.7-GGUF IQ4_XS quant, but got the bellow error.

/llama.cpp/src/llama-graph.cpp:1247: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null") failed

Searching for the error led me to #24541 which led me to #24593. I tried making the same one-line change in llama_model_minimax_m2::graph::graph and it seems to have worked and I'm able to use the EAGLE3 model. Details below, let me know if you want me to upload the EAGLE3 gguf file somewhere to test.

11.53.410.120 I slot print_timing: id  0 | task 0 | n_decoded =   2976, tg =  17.17 t/s, tg_3s =  16.74 t/s
11.56.545.919 I slot print_timing: id  0 | task 0 | n_decoded =   3028, tg =  17.16 t/s, tg_3s =  16.58 t/s
11.57.461.271 I slot print_timing: id  0 | task 0 | prompt eval time =    7445.92 ms /    94 tokens (   79.21 ms per token,    12.62 tokens per second)
11.57.461.273 I slot print_timing: id  0 | task 0 |        eval time =  177357.53 ms /  3043 tokens (   58.28 ms per token,    17.16 tokens per second)
11.57.461.274 I slot print_timing: id  0 | task 0 |       total time =  184803.45 ms /  3137 tokens
11.57.461.275 I slot print_timing: id  0 | task 0 |    graphs reused =       1057
11.57.461.278 I slot print_timing: id  0 | task 0 | draft acceptance = 0.87084 ( 1072 accepted /  1231 generated), mean len =  2.53
11.57.461.279 I slot print_timing: id  0 | task 0 |      acc per pos = (0.904, 0.356, 0.139, 0.079, 0.036, 0.014, 0.004, 0.000)
11.57.461.297 I spec common_specu: statistics     draft-eagle3: #calls(b,g,a) =    1   1970    700, #gen drafts =    700, #acc drafts =   633, #gen tokens =   1231, #acc tokens =  1072, #mean acc len = 2.53, #acc rate/pos = (0.904, 0.356, 0.139, 0.079, 0.036, 0.014, 0.004), dur(b,g,a) = 0.001, 4519.836, 0.829 ms
11.57.461.317 I slot      release: id  0 | task 0 | stop processing: n_tokens = 3136, truncated = 0

Additional information

EAGLE3 Conversion

I converted https://huggingface.co/asherszhang/MiniMax-M2.7-EAGLE3-draft-vocab32k to gguf with this command. I haven't tried quantizing the draft model yet, but I can if that would be helpful.

python3 ./llama.cpp/convert_hf_to_gguf.py mm2.7eagle \
  --target-model-dir mm2.7 \
  --outfile ./MiniMax-M2.7-EAGLE3-draft-vocab32k.gguf \
  --outtype f16

Full Error

0.00.125.717 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.558.349 I srv    load_model: loading model '/models/MiniMaxAI_MiniMax-M2.7-IQ4_XS-00001-of-00004.gguf'
0.13.387.540 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
7.52.633.969 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
8.01.300.258 I srv    load_model: initializing, n_slots = 1, n_ctx_slot = 128000, kv_unified = 'false'
8.01.701.904 I srv  llama_server: model loaded
8.01.701.909 I srv  llama_server: listening on http://0.0.0.0:10002
13.14.321.893 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
13.14.323.206 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
/llama.cpp/src/llama-graph.cpp:1247: GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null") failed
/app/libggml-base.so.0(+0x1b236)[0x7c116bfc2236]
/app/libggml-base.so.0(ggml_print_backtrace+0x21a)[0x7c116bfc26ba]
/app/libggml-base.so.0(ggml_abort+0x15b)[0x7c116bfc289b]
/app/libllama.so.0(+0x12732a)[0x7c116c1a732a]
/app/libllama.so.0(_ZNK11llama_model11build_graphERK16llm_graph_params+0x95)[0x7c116c2222a5]
/app/libllama.so.0(_ZN13llama_context13graph_reserveEjjjPK22llama_memory_context_ibPm+0x1ea)[0x7c116c17107a]
/app/libllama.so.0(_ZN13llama_context13sched_reserveEv+0x717)[0x7c116c172677]
/app/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x23f)[0x7c116c17677f]
/app/libllama.so.0(llama_decode+0xf)[0x7c116c1786ef]
/app/libllama-server-impl.so(_ZN19server_context_impl6decodeERiiR11llama_batch+0x192)[0x7c116d04ecd2]
/app/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0xb72)[0x7c116d0516f2]
/app/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x221)[0x7c116cfef541]
/app/libllama-server-impl.so(_Z12llama_serverR13common_paramsiPPc+0x36c9)[0x7c116cf8f8e9]
/app/libllama-server-impl.so(_Z12llama_serveriPPc+0xf6b)[0x7c116cf919db]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7c116c9ff1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7c116c9ff28b]
./llama-server(+0x1315)[0x5dcb2bfd4315]

Results

Note: I haven't really tried tuning the --spec-draft-* options yet, this was just from my initial test (full commands below)

# Baseline
15.37.586.422 I slot print_timing: id  0 | task 0 | n_decoded =   4964, tg =  14.21 t/s, tg_3s =  12.94 t/s
15.40.596.398 I slot print_timing: id  0 | task 0 | n_decoded =   5003, tg =  14.20 t/s, tg_3s =  12.96 t/s
15.43.621.239 I slot print_timing: id  0 | task 0 | n_decoded =   5042, tg =  14.19 t/s, tg_3s =  12.89 t/s
15.44.011.285 I slot print_timing: id  0 | task 0 | prompt eval time =    5228.10 ms /    94 tokens (   55.62 ms per token,    17.98 tokens per second)
15.44.011.288 I slot print_timing: id  0 | task 0 |        eval time =  355730.84 ms /  5047 tokens (   70.48 ms per token,    14.19 tokens per second)
15.44.011.288 I slot print_timing: id  0 | task 0 |       total time =  360958.94 ms /  5141 tokens
15.44.011.293 I slot print_timing: id  0 | task 0 |    graphs reused =       5026
15.44.011.321 I slot      release: id  0 | task 0 | stop processing: n_tokens = 5140, truncated = 0


# EAGLE3
11.44.144.849 I slot print_timing: id  0 | task 0 | n_decoded =   2826, tg =  17.23 t/s, tg_3s =  17.26 t/s
11.47.272.058 I slot print_timing: id  0 | task 0 | n_decoded =   2878, tg =  17.22 t/s, tg_3s =  16.63 t/s
11.50.303.042 I slot print_timing: id  0 | task 0 | n_decoded =   2924, tg =  17.18 t/s, tg_3s =  15.18 t/s
11.53.410.120 I slot print_timing: id  0 | task 0 | n_decoded =   2976, tg =  17.17 t/s, tg_3s =  16.74 t/s
11.56.545.919 I slot print_timing: id  0 | task 0 | n_decoded =   3028, tg =  17.16 t/s, tg_3s =  16.58 t/s
11.57.461.271 I slot print_timing: id  0 | task 0 | prompt eval time =    7445.92 ms /    94 tokens (   79.21 ms per token,    12.62 tokens per second)
11.57.461.273 I slot print_timing: id  0 | task 0 |        eval time =  177357.53 ms /  3043 tokens (   58.28 ms per token,    17.16 tokens per second)
11.57.461.274 I slot print_timing: id  0 | task 0 |       total time =  184803.45 ms /  3137 tokens
11.57.461.275 I slot print_timing: id  0 | task 0 |    graphs reused =       1057
11.57.461.278 I slot print_timing: id  0 | task 0 | draft acceptance = 0.87084 ( 1072 accepted /  1231 generated), mean len =  2.53
11.57.461.279 I slot print_timing: id  0 | task 0 |      acc per pos = (0.904, 0.356, 0.139, 0.079, 0.036, 0.014, 0.004, 0.000)
11.57.461.297 I spec common_specu: statistics     draft-eagle3: #calls(b,g,a) =    1   1970    700, #gen drafts =    700, #acc drafts =   633, #gen tokens =   1231, #acc tokens =  1072, #mean acc len = 2.53, #acc rate/pos = (0.904, 0.356, 0.139, 0.079, 0.036, 0.014, 0.004), dur(b,g,a) = 0.001, 4519.836, 0.829 ms
11.57.461.317 I slot      release: id  0 | task 0 | stop processing: n_tokens = 3136, truncated = 0
11.57.461.336 I srv  update_slots: all slots are idle

Other Info

Hardware

• 1x NVIDIA V100
• 6x NVIDIA P100
• 1x NVIDIA GTX 1080

Baseline command

./llama-server
        --timeout 1800
        --port  ${PORT}
        --model /models/MiniMaxAI_MiniMax-M2.7-IQ4_XS-00001-of-00004.gguf
        --device CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7
        --tensor-split 26,14,14,14,14,14,14,5
        --flash-attn on
        --ctx-size 128000
        -sm tensor 
        --n-cpu-moe 22
        -lv 3
        --n-gpu-layers all
        --no-mmap
        -np 1
        --jinja
        --cache-type-k q8_0
        --cache-type-v q8_0
        --top-k 40
        --temperature 1.0
        --top-p 0.95
        --min-p 0.0

--spec-type draft-eagle3 command

./llama-server
        --timeout 1800
        --port  ${PORT}
        --model /models/MiniMaxAI_MiniMax-M2.7-IQ4_XS-00001-of-00004.gguf
        --device CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7
        --tensor-split 26,14,14,14,14,14,14,5
        --flash-attn on
        --ctx-size 128000
        -sm tensor 
        --n-cpu-moe 22
        -lv 3
        --n-gpu-layers all
        --no-mmap
        -np 1
        --jinja
        --cache-type-k q8_0
        --cache-type-v q8_0
        --top-k 40
        --temperature 1.0
        --top-p 0.95
        --min-p 0.0
        --spec-type draft-eagle3
        --spec-draft-device CUDA0
        --spec-draft-ngl all
        -md /models/MiniMax-M2.7-EAGLE3-draft-vocab32k.gguf
        --spec-draft-n-max 8
        --spec-draft-p-min 0.75
        --spec-draft-type-k q4_0
        --spec-draft-type-v q4_0

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: No, but full disclosure - I'm just copying a change from spec: support eagle3 for qwen3.5 & 3.6 #24593 and I don't fully understand the implications of the change.

[ruixiang63]

Can you also post the speedup numbers here? @adrianisk

[adrianisk]

These are just preliminary numbers from my initial run, I haven't done any sort of tuning to the --spec-draft-* config options. I can post additional results once I have a chance to test things out more

Metric	Baseline	EAGLE3	% Change
Prompt Eval Speed	17.98 t/s	12.62 t/s	-29.8%
Generation (Eval) Speed	14.19 t/s	17.16 t/s	+20.9%