models: fix GDN normalization from max to rsqrt - #28068
Conversation
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
Correctness: all reference implementations including Qwen's own FlashQLA use x*rsqrt(sum(x^2)+eps); llama.cpp used the max-form. Affects every token through the 36 GDN layers.
|
I'm not sure the difference of ±sqrt(1e-6) is worth this? |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
@CISC You're not wrong, but the numerics do somewhat change a bit and transformers had to be patched 4 days after their impl for Qwen3.5 last year It's more just a reconciliation between the upstream impl and the llama.cpp impl |
|
FLA is being weird though, the eps is there to avoid division by zero, not to offset the value. As this may mess with fusion (not sure it actually does) is it worth enough to have its own op? |
|
https://godbolt.org/z/nWa7se5da for CPU: vsqrtss xmm1, xmm1, xmm1
vmovsd xmm0, QWORD PTR .LC3[rip]
vcvtss2sd xmm1, xmm1, xmm1
vmaxsd xmm1, xmm1, QWORD PTR .LC2[rip] **
vdivsd xmm0, xmm0, xmm1
vcvtsd2ss xmm0, xmm0, xmm0vs vsqrtss xmm1, xmm1, xmm1
vmovsd xmm0, QWORD PTR .LC3[rip]
vcvtss2sd xmm1, xmm1, xmm1
vaddsd xmm1, xmm1, QWORD PTR .LC2[rip] **
vdivsd xmm0, xmm0, xmm1
vcvtsd2ss xmm0, xmm0, xmm0So I guess identical haha except for the max and add op and https://uops.info/table.html - interestingly vmaxsd is actually slower in latency than vaddsd (same throughput though) |
|
I can ask the Qwen team if that helps to reconcile which impl is in fact correct if that helps? Maybe FLA itself is wrong haha |
This comment was marked as spam.
This comment was marked as spam.
Even if FLA is technically wrong, they used FLA during training, so it's still right. :) |
|
I noticed this discrepancy way back during the implementation of Qwen3-Next and we had a discussion back then. I did some measurements and the difference turned out to be completely insignificant for implementation faithfulness, so just dropping the note here :) |
|
@pwilkin nice work as well! Ye i doubt this will change things that much |
|
I'm leaning towards matching the reference implementation, regardless if the numerical impact is small. |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
(cherry picked from commit 5fdfa62)
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
(cherry picked from commit 5fdfa62)
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
Thank you for the fix, the GDN normalization change resolved fabrication on On IQ3_XXS (UD-IQ3_XXS), 3 false-premise probes all correctly produced empty On IQ1_S (UD-IQ1_S), 2 of 3 false-premise probes behaved correctly (one Prompt: "Ce stii despre lema lui Barbilian-Teodorescu privind grupurile Lie This combines two real Romanian mathematicians (Dan Barbilian, who has a Not a thinking-budget issue: finish_reason is "stop" (natural stop, not All 7 true-premise controls on IQ1_S, including the 2 obscure ones, So the fix reduces fabrication (confirmed clean on IQ3_XXS on the probes I |
|
Correction to my earlier comment My comment above, "the GDN normalization change resolved fabrication on several probes I tested", overstated what I had measured. I am retracting the causal claim. Two defects, and the second one does not depend on any new data. First, there was no control arm. The battery I ran used new probes, because my original probe text had been lost, and it ran only on master after this PR was merged. Changing the probes at the same time as the code makes the attribution unsupportable whatever the outcome. I have since run the missing control. Pair: this PR's branch (757abc9) against its own merge base (9723942), so the only difference is the two commits of this PR. Model Qwen3.8-Flash-Next UD-IQ1_S, identical flags on both arms ( Result: one probe out of five changes verdict. It goes from a 2531 token fabrication on the base to an empty output after consuming the full 16000 token budget on this branch. The other false premise probe fabricates on both arms, with different wording and 13985 against 7491 tokens. Both true premise controls are identical on both arms. So the change produced no refusal on any probe that had been fabricating. Second, independent of the control arm: I scored empty output from budget exhaustion as correct behavior. It is not a refusal. Running that same probe at reduced reasoning effort on this branch produces a confident fabrication in 1595 tokens, so the empty output at default effort was the same failure cut off before it could finish, not caution. My own text made that distinction one paragraph later and I did not apply it when scoring. At reduced effort the difference between the arms is 2 cells out of 6 against 4 out of 6, Fisher exact two sided p = 0.57. Nothing detectable at this sample size. What I am not claiming: that the change is wrong or harmful. It clearly changes the computation, and substantially, since every fingerprint differs and token counts move a lot even where the verdict does not. I have no evidence either way on whether the new semantics are the correct ones, and the argument in the commit message, that it matches the reference implementation, is untouched by anything above. I am withdrawing a benefit I reported and could not demonstrate, not asserting a harm. Sorry for the noise. Happy to share the harness, the full texts and the per probe numbers if any of it is useful. |
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* models: use flash-linear-attention's l2norm for gated delta net q/k
The GDN q/k normalization is defined by flash-linear-attention as
l2norm(x) = x * rsqrt(sum(x*x) + eps)
with eps inside the root. Every GDN call site in the tree uses ggml_l2_norm
instead, which is x / max(sqrt(sum(x*x)), eps), i.e.
torch.nn.functional.normalize - its CUDA kernel cites that page.
The clamp never engages at these magnitudes, so in practice llama.cpp
normalizes with no epsilon at all where the reference has one inside the
root.
transformers made the same substitution when it first added Qwen3-Next and
corrected it three days later in huggingface/transformers#40842, 'Fix the
misalignment between the l2norm in GDN of Qwen3-Next and the implementation
in the FLA library'. vLLM and SGLang vendor FLA rather than reimplementing
it, so neither ever had the clamp.
eps keeps coming from the checkpoint, exactly as every call site already
passed it. The references hardcode 1e-6 for this norm; that is a separate
question and the two agree on every GDN checkpoint in the wild.
ggml_l2_norm itself is correct and unchanged, as is rwkv7-base, its original
caller, which passes normalize's own default eps of 1e-12.
No new ggml op: rms_norm already carries eps inside the root, so
rms_norm(x, eps/n) * (1/sqrt(n)) is exactly x * rsqrt(sum(x*x) + eps).
* Update src/models/models.h
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
5 participants
GDN normalizes q and k with
x * rsqrt(sum(x^2) + eps).llama.cpp uses
x / max(sqrt(sum(x^2)), eps)All references uses the non max form:
sum_sq = (x_f32*x_f32).sum(-1) + eps; rstd = rsqrt(sum_sq), eps default 1e-6.F.normalizefor Qwen3-Next on 2025-09-09 and corrected it three days later in Fix the misalignment between the l2norm in GDN of Qwen3-Next and the implementation in the FLA library. huggingface/transformers#40842-ub 128control-ub 128controlAffects qwen35, qwen35moe, qwen3next, qwen4exp, kimi-linear, kimi-k3, bailingmoe3
ggml_l2_norm unchanged, rwkv7 untouched
AI Usage
Used Claude and Local Models for testing, iteration and code design - manual verification of model / PR usage