CUDA: tighter MMQ src1 buffer size for native fp4#25613
Draft
leonardHONG wants to merge 1 commit into
Draft
Conversation
github-actions
Bot
added
ggml
am17an
reviewed
Jul 13, 2026
| const bool use_native_fp4 = blackwell_mma_available(cc) && (src0->type == GGML_TYPE_MXFP4 || src0->type == GGML_TYPE_NVFP4); | ||
| // Native fp4 packs twice as many values per MMQ block as q8_1. | ||
| const size_t y_block_size = use_native_fp4 ? sizeof(block_fp4_mmq) : sizeof(block_q8_1_mmq); | ||
| const size_t y_values_per_block = use_native_fp4 ? QK_K : 4*QK8_1; |
Contributor
There was a problem hiding this comment.
QK_K is probably overloaded here, we should use a better term
Contributor
Author
There was a problem hiding this comment.
Done, added QK_FP4_MMQ.
leonardHONG
force-pushed
the
cuda-mmq-fp4-pool-size
branch
from
5b50dee to
4e896d2
Compare
Contributor
|
This should be scheduled after #24127 . |
Contributor
Author
|
OK, I’ll rebase after it lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The native FP4 path still sizes the
src1quantization buffer using the Q8_1 layout.block_fp4_mmqhas the same size asblock_q8_1_mmq, but stores twice as many values, so the buffer is over-allocated by 2x in bothMUL_MATandMUL_MAT_ID.This PR uses the actual block layout of the selected quantizer. The extra tail padding is unchanged.
Additional information
Tested on RTX 5090 with MXFP4 and NVFP4 for both
MUL_MATandMUL_MAT_ID.All
test-backend-opscases passed. Around 300 FP4 cases also passed undercompute-sanitizer --tool memcheckwith 0 errors.Requirements