sycl: fix UE4M3 parsing#25608
Conversation
|
Reproduced here. Opens up an opportunity for further optimizations. Before:
After (oneDNN Weight Decompression):
(preliminary, don't hold me to it.) |
Backport of upstream PR ggml-org#25608. NVFP4 stores per-16-weight scales as unsigned E4M3; the SYCL path decoded them via the signed E4M3 helper, producing gibberish output for NVFP4 models on Intel Arc (B70). Decode UE4M3 explicitly (exp==0xF is a valid exponent, not NaN). Assisted-by: Fable 5 (Cursor) Co-authored-by: Cursor <[email protected]>
…org#25550 (XIELU) Assisted-by: Fable 5 (Cursor) Co-authored-by: Cursor <[email protected]>
ggerganov
left a comment
There was a problem hiding this comment.
If there is a problem in this function, it should be first fixed in the CPU implementation and then all backends should follow that. cc @ggml-org/ggml-cuda
I could not reproduce the problem on the CPU-specific backend. I only had the problem with SYCL. However, in comparing the CPU implementation and my proposed fix, I did uncover one thing: the CPU implementation returns 0.0f for both 0x00 and 0x7F, and I was missing the 0x7F path. Fixed. |
The NVFP4 quantization format stores a scaling factor for every group of 16 weights, packed into a single UE4M3 byte. The SYCL GPU code was converting these scale values using the E4M3 path, but that's *signed*, and these are unsigned values.
7ec2e03 to
676129a
Compare
The CPU implementation matches gguf-py/gguf/quants.py:NVFP4.ue4m3_to_fp32. |
Overview
The NVFP4 quantization format stores a scaling factor for every group of 16 weights, packed into a single UE4M3 byte.
The SYCL GPU code was converting these scale values using the E4M3 path, but that's signed, and these are unsigned values. (The CPU backend converts these values correctly; this bug is in the SYCL backend alone.)
Additional information
The unsloth/Qwen3.6-27B-NVFP4 quant, after converting to GGUF, generates coherent output on my B70 instead of gibberish once this patch is applied.
Requirements
I used GLM-5.2 to diagnose this issue and to generate the patch, but I tested the output manually before and after applying the patch by hand to confirm that the fix works on my Intel Arc Pro B70 GPU.