fix: Merge branch 'main' into fix/preserve-bf16-gguf-v2 - #9064
Open
Ricardo-M-L wants to merge 10 commits into
Open
fix: Merge branch 'main' into fix/preserve-bf16-gguf-v2#9064Ricardo-M-L wants to merge 10 commits into
Ricardo-M-L wants to merge 10 commits into
Conversation
…n list When users request multiple quantization methods including the base format (e.g., ["q4_k_m", "bf16"]), the bf16 GGUF serves as both the intermediate conversion and a user-requested output. The cleanup step unconditionally deleted this file, losing the explicitly requested bf16 output. Only delete the intermediate base GGUF when the user did not request it. Fixes unslothai#4932 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…ring Address review feedback: the reverse() call must always execute when quants_created is True to maintain correct [text_model, mmproj] ordering for VLMs. Only the file deletion should be conditional on whether the user requested the base format. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Signed-off-by: Ricardo-M-L <[email protected]>
…xample commands Address review from @Datta0: when the base format (e.g. bf16) is kept in all_saved_locations, it could end up at [-1], causing the VLM example command to use bf16 as --mmproj instead of the actual projector file. Move the preserved base file to index 1 (after the primary quantized model, before mmproj) so [0] and [-1] remain correct for example commands. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> Signed-off-by: Ricardo-M-L <[email protected]>
for more information, see https://pre-commit.ci
weight.shape[X] // 8 != 0 is True for any non-zero dimension, causing incorrect fallback to dequantization for small weights. Using % 8 correctly checks non-divisibility: weights not divisible by 8 should dequantize, while those with % 8 == 0 stay on the fast kernel path.
convert_to_gguf can return multiple base text shards plus an mmproj entry when llama.cpp splits the output. The previous cleanup only removed/repositioned base_gguf=initial_files[0]: - when the base format is NOT in quantization_method, sibling shards were left both in all_saved_locations and on disk as orphans - when the base IS preserved, the reverse + insert(1, base_gguf) step left a sibling base shard at all_saved_locations[-1] for VLMs, so the example llama-mtmd-cli command ended up with --mmproj pointing at a text shard instead of the projector Treat every initial file whose basename does not contain "-mmproj" as part of the base set, then remove/unlink or reposition all of them together. Drop the redundant frozenset() construction at both call sites and the dead `base_gguf in all_saved_locations` clause in the reorder guard.
unsloth/kernels/fp8.py: FbgemmFp8Linear_matmul.forward had a dequant fallback that called torch_matmul without adding bias. The fast row-wise branch and the block FP8 branch both apply `output = output + bias if bias is not None else output` immediately after the matmul; the fallback now matches. This silently dropped bias for any FP8 layer routed to the fallback (Qwen 2.5 VL gate/up_proj 3420x1280, transposed-weight backward dispatch, and the small-shape cases newly routed here by the recent `% 8` divisibility fix). unsloth/save.py: preserved_base inside the cleanup block and want_full_precision below it computed the identical expression `first_conversion in quantization_method`. Hoist want_full_precision above the cleanup block, reuse it for the not-preserved deletion and the preserved reposition, and assign True directly in the GPT-OSS branch.
for more information, see https://pre-commit.ci
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eaf976465d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # convert_to_gguf may return multiple base shards plus an mmproj entry, | ||
| # so treat every initial file that is not an mmproj as part of the base set. | ||
| base_files = [ | ||
| f for f in initial_files if "-mmproj" not in os.path.basename(f).lower() |
There was a problem hiding this comment.
For VLM conversions whose projector uses the repository's supported mmproj-BF16.gguf/mmproj-Q8_0.gguf naming form, this predicate fails to recognize the projector because it only searches for -mmproj. The projector is therefore included in base_files, and whenever an additional quantization is created without preserving full precision, the cleanup loop deletes it; the later VLM example then treats the quantized text model at all_file_locations[-1] as the projector. Classify any basename containing mmproj as the auxiliary file instead.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR addresses: Merge branch 'main' into fix/preserve-bf16-gguf-v2