keep_excluded_layers_as_original_model_state=True renames weight_scale_inv to weight_scale; vLLM cannot load the result

[stefanskiasan]

Summary

When keep_excluded_layers_as_original_model_state=True preserves a block-FP8 weight, Quark
renames its scale from <weight>_scale_inv to <weight>_scale. vLLM's own consumer registers the
parameter as weight_scale_inv, so loading the resulting checkpoint fails with a KeyError.

The result is a trap: the argument that keeps FP8-native weights at FP8 produces a checkpoint vLLM
cannot load, and the obvious workaround (False) silently dequantizes those weights to BF16.

Where

quark/torch/quantization/file2file_quantization.py, around line 946:

if keep_excluded_layers_as_original_model_state and weight_name in keep_input_model_format_tensor_name_set:
    ...
    recovered_tensors[weight_name] = f.get_tensor(weight_name)
    if scale_inv_name in all_keys:
        # DeepSeek-V3 / standard FP8: scale stored as "{weight}_scale_inv"
        quark_scale_name = f"{weight_name}_scale"        # <-- renamed
        recovered_tensors[quark_scale_name] = f.get_tensor(scale_inv_name)

vLLM expects the original name in two independent places:

# vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py
#   QuarkW8A8Fp8PerBlock.create_weights
layer.register_parameter("weight_scale_inv", weight_scale)

# vllm/models/glm5next/nvidia/model.py  (_try_load_fp8_attn_proj)
is_scale = "weight_scale_inv" in name

Reproduction

GLM-5.3-Flash (natively FP8, block-scaled 128×128), quantized with
quantize_model_per_safetensor(..., keep_excluded_layers_as_original_model_state=True), MoE
experts to MXFP4, attention and dense MLPs excluded:

KeyError: 'layers.0.mlp.down_proj.weight_scale'

The produced index confirms the rename — 36,414 weight_scale (the MXFP4 experts, correct for
that format) and 0 weight_scale_inv, although the source checkpoint stores the excluded
layers with weight_scale_inv.

Why the workaround is worse than it looks

Setting False loads fine, so it is the natural escape — and that is what I did for weeks. But for
an FP8-native source model False does not leave excluded layers alone, it dequantizes them
to BF16. dtype census of the produced checkpoint, all 75,184 tensors:

dtype	tensors	size
U8 (packed MXFP4)	72,828	162.3 GB
BF16	2,065	31.7 GB
F32 (scales)	291	0.0 GB

Those 31.7 GB should be 15.9 GB of FP8. On one MI350X that is 15.9 GB of VRAM not available to the
KV cache, plus BF16 GEMMs where FP8 would run at twice the rate. With a BF16 source model the issue
is invisible, because excluded layers are BF16 either way — it only bites on FP8-native sources
(GLM-5.x, DeepSeek-V3 family), which is exactly where True is the intended setting.

Suggested fix

Keep the checkpoint's scale name when preserving original model state:

    if scale_inv_name in all_keys:
        quark_scale_name = f"{weight_name}_scale_inv"
        recovered_tensors[quark_scale_name] = f.get_tensor(scale_inv_name)

With that one-line change the checkpoint loads in vLLM and the excluded layers stay FP8 (verified:
924 weight_scale_inv alongside 36,414 weight_scale for the MXFP4 experts, correct separation).

The MXFP4-source branch a few lines above renames the same way; it may need the same treatment, but
I have not exercised that path and cannot say.

Two notes on what came after, in case they save someone else time:

1. GLM-5.3-Flash then hits a second, unrelated wall — QuarkW8A8Fp8MoEMethod rejects block scales
   (For FP8 Fused MoE layers, only per-tensor and per-channel scales ... Found per_block) — so
   FP8 MoE layers are not usable in vLLM regardless of this fix. Non-MoE layers are.
2. Environment: Quark 0.12, vLLM glm-release, ROCm 7.2.3, MI350X (gfx950).