Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/design/per-component-quantization-loading.md
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,27 @@ preprocessor can still pack QMoE expert banks and tie floating-point tables.
A per-component `WeightBundle` adapter is a future migration target, not the
currently implemented interface.

Native QMoE additionally has a projection-specific adapter for Olive fused
K-last routed experts. The initial mixed layout is deliberately narrow:
model-wide INT4, FC1 `gate_up_proj` overridden to INT2, FC2 `down_proj`
remaining INT4, and one common power-of-two group size of at least 16. Exact
and `re:` Olive overrides are resolved against the original Hugging Face paths;
plain Olive exclusions retain their producer-defined substring matching.
GPTQ and AWQ instead use the generic plain-path subtree and `re:` full-match
rules. FC1 and FC2 buffers are then validated independently. The adapter
requires every routed layer and validates rank, dtype, expert count, packed byte
width, scale geometry, and optional zero-point geometry before binding.
Qwen3.5-VL keeps the decoder's full module plan
(``preserve_module_plan=True``) while constructing the split package: expert
overrides are resolved against authoritative ``model.language_model`` source
paths, while validation and binding use the decoder-prefixed checkpoint roots
created by the VL weight router.
It emits QMoE's FC-specific bit attributes (including FC3=FC1 for fused
SwiGLU). Uniform INT4 omits those attributes and retains the legacy payload
path. Mixed-width execution remains an ONNX Runtime `NOT_IMPLEMENTED`
boundary; Mobius tests this serialized graph contract without claiming
runtime numerical support.

Appropriate model-specific operations include:

- HuggingFace-to-ONNX name alignment;
Expand Down
2 changes: 2 additions & 0 deletions src/mobius/_configs/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -651,6 +651,8 @@ class ArchitectureConfig(BaseModelConfig):
num_experts_per_tok: int | None = None
moe_intermediate_size: int | None = None
shared_expert_intermediate_size: int | None = None
# Internal HF path aliases used to resolve routed-expert module overrides.
qmoe_source_paths: tuple[str, ...] = ()
norm_topk_prob: bool = True
# When True, the decoder layer uses post-norm style (FlexOLMo): norms are applied
# to sub-layer outputs instead of inputs, with an extra post_feedforward_layernorm.
Expand Down
5 changes: 3 additions & 2 deletions src/mobius/_configs/_quantization.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,9 @@ class QuantizationConfig:
# Full HuggingFace module paths (including descendants) or ``re:``-prefixed
# full-match regexes that remain floating point inside this component.
modules_to_not_convert: tuple[str, ...] = ()
# The same path/regex matching rules apply to per-module overrides.
# Insertion order is significant: the first matching override wins.
# Generic component collapse treats a plain path as a subtree. Producer-
# specific adapters may apply stricter matching (Olive uses literal
# equality); insertion order remains significant in either case.
overrides: dict[str, QuantizationOverride] = dataclasses.field(default_factory=dict)
# Keep this field last: QuantizationConfig has historically supported
# positional construction, so inserting a field earlier would silently
Expand Down
Loading
Loading