convert: support compressed-tensors mixed-precision NVFP4 checkpoints - #195
Draft
danielhanchen wants to merge 6 commits into
Draft
convert: support compressed-tensors mixed-precision NVFP4 checkpoints#195danielhanchen wants to merge 6 commits into
danielhanchen wants to merge 6 commits into
Conversation
Every NVFP4 checkpoint published by Unsloth (unsloth/Qwen3.6-35B-A3B-NVFP4, unsloth/Qwen3.6-35B-A3B-NVFP4-Fast, unsloth/Qwen3.6-27B-NVFP4) uses the compressed-tensors "mixed-precision" format with two config groups: one float-quantized FP8 group covering the attention projections and lm_head, and one nvfp4-pack-quantized group covering the MoE experts. The converter rejected all of them with NotImplementedError: Can't handle multiple config groups for compressed-tensors yet because the nvfp4_compressed_tensors gate required every group to be nvfp4-pack-quantized. Relax both copies of that gate to accept a checkpoint in which any group is NVFP4, and handle the rest of the checkpoint: - _generate_nvfp4_tensors now identifies NVFP4 tensors by dtype and block geometry rather than by scale rank alone. The FP8 group also carries a 2D weight_scale of shape [out, 1], so the existing "scale.ndim < 2" test let FP8 tensors fall into the NVFP4 repacking path. - The nvfp4 branch of dequant_model now dequantizes the leftover FP8 weights the same way the float-quantized branch does, and drops the unused input_scale, k_scale and v_scale sidecars. Previously it did nothing, so those tensors reached the writer still quantized. With this, converting unsloth/Qwen3.6-35B-A3B-NVFP4-Fast with --fp8-as-q8 produces a MOSTLY_NVFP4 GGUF whose 240 expert tensors are GGML_TYPE_NVFP4 and whose attention and lm_head tensors are Q8_0.
Member
Author
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…roup The residual branch dequantized every leftover weight_scale with block_size None, which is right for the per-channel FP8 group this path was written for and wrong for a block group, whose scales are a grid that block_structure has to expand first. Using them directly either raises on broadcasting or, when the dimensions happen to line up, applies the wrong scales silently. Resolve each residual group and reject anything but the supported channel pairing.
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Post-convergence pass. Four comment lines to three, and the two over-long ones brought under the line limit. Comments only, no code change.
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
No code change: every remaining line carries a fact the code does not state.
Member
Author
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Recognising mixed-precision means the NVFP4 branch of dequant_model now runs on real inputs instead of being a bare pass, and everything the NVFP4 group does not consume lands there. The only dequantizer that branch has is dequant_simple, which is correct for one shape of input: an unpacked weight with one scale per row. The guard only checked strategy, so three things reached it that it cannot handle. A "pack-quantized" residual group is the one that matters. Its weights are nibble-packed ints needing dequant_packed, but its strategy really is "channel", so a strategy-only guard waves it through. The weight_name-not-in-model_tensors check does not catch it either, because prepare_tensors renames every .weight_packed to .weight before dequant_model runs, so by the time the branch looks, the name it is testing for absence exists. The result was a clean exit 0 and a file with packed nibbles multiplied by a scale where weights should be. unsloth/gemma-4-E2B-it-NVFP4 and unsloth/gemma-4-E4B-it-NVFP4 both ship exactly that group_2 over embed_tokens_per_layer, so this is a published model family silently mis-converting rather than a hypothetical. Before this change master refused those checkpoints outright at the multiple-config-groups raise, so widening the gate turned a hard refusal into a wrong file, which is worse than the bug it replaced. Also fixed, all in the same branch: - A weight still uint8 after _generate_nvfp4_tensors is one whose dtype or block geometry that function skipped. For most widths the shapes then fail to broadcast and you get a loud error, but a [out, 1] scale broadcasts cleanly and dequant_simple returns scaled nibbles with no error at all. Refuse it instead. - weight_config = tuple(groups.values())[0]["weights"] ran unconditionally right after the gate and is dead code for the NVFP4 branch, so a group with no "weights" key raised KeyError. Moved into the two branches that use it. - The block_structure rejection reported "a 'channel' group is not supported", naming the field that was fine and hiding the one that was not. Split into three messages that each name what actually tripped. scripts/unsloth/test_convert_nvfp4_mixed.py covers all four against the real gemma-4 group_2 config, plus controls that the supported nvfp4-plus-fp8-channel shape still converts to the right numbers and that single-group formats are untouched. It fails in seven places on the parent commit and passes here. Byte-identity re-checked with this change in: 44 conversions across six architectures and eleven quantization types, 37 byte-identical to the merge base and zero changed by this commit. unsloth/Qwen3.6-27B-NVFP4 still converts with max-abs-error 0.000e+00 against a reference dequantization from the safetensors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every NVFP4 checkpoint Unsloth publishes is refused by
convert_hf_to_gguf.py:This reproduces on
unsloth/Qwen3.6-35B-A3B-NVFP4,unsloth/Qwen3.6-35B-A3B-NVFP4-Fastand
unsloth/Qwen3.6-27B-NVFP4.ggml-org/llama.cppmaster has the same twoall(...)gates (
conversion/base.py:492and:861) and the sameraiseat:496, so it isrefused there too.
Why
These are compressed-tensors
"format": "mixed-precision"checkpoints with two configgroups, not one:
group_0float-quantizedstrategy: channelself_attn.(q|k|v|o)_proj,linear_attn.(in_proj_qkv|in_proj_z|out_proj),lm_head(and on the non-Fast 35B also the layer 32 to 39 experts)group_1nvfp4-pack-quantizedgroup_size: 16,scale_dtype: torch.float8_e4m3fnmlp.experts.*.(gate|up|down)_proj,shared_expert.(gate|up|down)_projThe
nvfp4_compressed_tensorsgate demands that all groups benvfp4-pack-quantized, so a checkpoint that is NVFP4 for the experts and FP8 for theattention is classified as neither, and falls into the multi-group
raise.Fix
Four hunks in
conversion/base.py:all(...)becomesany(...)in both copies of the gate (dequant_modelandprepare_tensors), so a mixed-precision checkpoint with an NVFP4 group is recognisedas NVFP4.
_generate_nvfp4_tensorsidentifies NVFP4 tensors by dtype and block geometry(
weightuint8,scalefloat8_e4m3fn, one scale per 16 values) rather than byscale.ndim >= 2alone. This matters because the FP8 group's per-channelweight_scaleis shape[out, 1], which is also rank 2, so without the extra testthe FP8 tensors were fed to
_nvfp4_pack.nvfp4_compressed_tensorsbranch ofdequant_modelwas a barepass. It nowdequantizes whatever
weight_scaleentries survive_generate_nvfp4_tensors(whichin a mixed-precision checkpoint are exactly the non-NVFP4 group) using the same
dequant_simplecall thefloat-quantizedbranch uses, and drops the unusedinput_scale,k_scaleandv_scalesidecars.Verification
Converts in about six minutes on a DGX Spark and writes
general.file_type = 39(
MOSTLY_NVFP4), 1233 tensors, 21.32 GiB:blk.0.ffn_down_exps.weight [512, 2048, 256]token_embd.weight [2048, 248320]output.weight [2048, 248320]blk.0.ffn_down_exps.scale [256]480 of the F32 tensors are the
weight_scale_2andinput_scalesidecars written as.scaleand.input_scale.Measured on a DGX Spark (GB10, sm_121a),
nvidia-smi -lgc 300,2100, observedclocks.sm2093 MHz on every cell, against the same model asMXFP4_MOEand asunsloth/Qwen3.6-35B-A3B-GGUFUD-Q4_K_XL:nsyson the NVFP4 prefill cell confirms the Blackwell FP4 path is what runs:42.3 % of GPU time in
mul_mat_q<(ggml_type)40, 128, false>plus 2.5 % inquantize_mmq_nvfp4<...>, the FP4 activation quantiser, so it is genuine W4A4.KL divergence over 4 chunks with the NVFP4 GGUF as the base (no BF16 checkpoint was
available for an absolute reference):
-ctk q8_0 -ctv q8_0Two things this does not do
quantization_config.kv_cache_scheme= 8 bit float, per-tensor,static_minmax, andship
self_attn.k_scale/self_attn.v_scalefor the full-attention layers. There isno GGUF representation for them, so they are discarded and llama.cpp's KV cache stays
F16 unless the user passes
-ctk/-ctv. The FP8 KV the checkpoint was calibratedfor is not reproduced.
input_scalesidecars are written but unused.llama-model.cpploads them asTENSOR_NOT_REQUIRED, but the MMQ activation quantiserquantize_mmq_nvfp4computesits own dynamic per-row global scale (
row_amax / (6 * 448)) at runtime and then doesa five candidate local search over neighbouring UE4M3 codes per 16 element sub-block.
The checkpoint's statically calibrated activation scale is therefore ignored, which is
a deliberate difference from the vLLM CUTLASS path but worth being explicit about.
Draft because the size result deserves a follow-up: the convert-only path cannot touch
what the HF checkpoint left unquantized, so
token_embdand the ignore-listedlinear_attn.in_proj_a/in_proj_bstay BF16 and the NVFP4 GGUF ends up larger thanQ4_K_XL. An
--outtypestyle override for the unquantized remainder would fix that.Follow-up measured after this PR was opened
Use
--outtype q8_0, not--outtype auto. A convert-only path cannot touch what the HFcheckpoint never quantized, so with
--outtype autothe ignore-listedtoken_embdandlinear_attn.in_proj_a/in_proj_bstay BF16 (2.53 GiB) and the file lands at 21.32 GiB, largerthan Q4_K_XL. With
--outtype q8_0 --fp8-as-q8it is 20.14 GiB:The 240 NVFP4 tensors and all 480
.scale/.input_scalesidecars are byte-identical between thetwo builds (sha256 over the concatenated NVFP4 tensor bytes matches).
llama-quantizecannot dothis shrink instead:
--tensor-typeis only consulted insidellama_tensor_get_type, which isonly reached when
tensor_allows_quantizationreturns true, and that function begins withif (params->only_copy) return false;(src/llama-quant.cpp:289), while withoutCOPYthe NVFP4expert tensors are eligible for requantization and there is no way to exempt them.
The q8_0 build also measures slightly better: PPL 4.9027 +/- 0.1260 against 4.9348 +/- 0.1269, and
prefill 2405.0 t/s against 2344.9 t/s, decode at npl 32 338.1 t/s against 339.7 t/s.
The GGUF reproduces the source checkpoint. Feeding vLLM the exact token ids that
llama-perplexity --kl-divergence-basescored (read out of the.datheader, so there is notokenizer mismatch) and scoring the same second-half positions with
prompt_logprobs=20, over 4092positions:
FlashInferCutlassNvFp4LinearKernel+FLASHINFER_CUTLASSMoETop-20 KL(vLLM || GGUF), renormalised over vLLM's top-20 support: 0.042921 nats, top-1 agreement
90.69 %. The weights are bit-identical between the two, so that residual is the KV cache and the
activation quantiser described above.