ggml-cpu: vectorize sum-of-squares reduction in RMS-norm F32 kernel

[codspeed-hq]

Summary

Replaces the scalar sum-of-squares reduction in the F32 RMS-norm kernel (ggml_compute_forward_rms_norm_f32 in ggml/src/ggml-cpu/ops.cpp) with the already-vectorized dot-product kernel ggml_vec_dot_f32.

Why this is the hot path

RMS-norm runs before every attention and feed-forward block. It is dispatched from the fused ggml_compute_forward_rms_norm_mul_fused path, which flamegraph analysis of the walltime macro benchmarks showed to be the single largest self-time function of prompt_layer[q4_k]. Its first pass computed the per-row sum of squares with a plain scalar loop that even carried the code's own // worth switching to explicit SIMD? comment. The ggml_float (double) accumulation forces scalar execution and leaves the vector units idle over the entire hidden dimension for every row.

Change

sum(x[i]*x[i]) is exactly dot(x, x), so the loop is replaced with the existing SIMD kernel (8-way multi-accumulator FMA; NEON/SVE on ARM, AVX/FMA on x86):

float sumf = 0.0f;
ggml_vec_dot_f32(ne00, &sumf, 0, x, 0, x, 0, 1);
const float mean = sumf/ne00;

This is the same idiom already used by ggml_vec_norm_f32 in vec.h, so it introduces no new pattern. The change is architecture-neutral and benefits both the simulation micro path and the walltime macro path. Net −3 lines, scoped strictly to the reduction.

Correctness

The scalar 1/sqrt(mean+eps) scale is unchanged; only the reduction's accumulation strategy differs (float multi-accumulator vs. double scalar). I verified this with a standalone check over 100 random n=2048 rows: the maximum relative difference in the resulting scale factor was 1.4e-7 (float-epsilon level), far within the relative-error tolerance the RMS_NORM backend test uses.

Validation

Measured via CodSpeed simulation mode (micro benchmark suite), baseline vs. head:

Benchmark	Base	Head	Change
rms_norm	562.8 µs	466.1 µs	+20.75%

The other simulation benchmarks were unchanged — no regressions.

[codspeed-hq]

Merging this PR will improve performance by 20.85%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	rms_norm	561.2 µs	464.4 µs	+20.85%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-vectorize-the-sum-of-squares-reduction-in-the-fuse-1785230901711 (d8d6aa6) with master (46819c9)}