ggml-cpu: vectorize the fused RMS-norm kernel (ggml_compute_forward_rms_norm_f32)

[codspeed-hq]

Summary

Vectorizes both hot loops of the fused RMS-norm path (ggml_compute_forward_rms_norm_f32 with GGML_RMS_NORM_FUSE_OP_MUL) in ggml/src/ggml-cpu/ops.cpp. This is the norm × weight op applied before every attention and feed-forward block, so it is a genuine inference hot path.

Analysis

The CI walltime flamegraph for prompt_layer[f32] shows ggml_compute_forward_rms_norm_mul_fused at ~26.8% self time — the second-hottest function after the matmul dot kernel. Both of its per-row loops were plain scalar loops that GCC does not auto-vectorize at -O2:

• the sum-of-squares reduction (sum += x[i]*x[i]), and
• the fused output multiply (y[i] = x[i]*scale*w[i]).

Change

• ops.cpp: replaced the scalar sum-of-squares reduction with a call to the already-vectorized ggml_vec_dot_f32(n, &sumf, …, x, …, x, …) (i.e. sum(x·x)), and replaced the scalar output loop with a new ggml_vec_scale_mul_f32 helper.
• vec.h: added ggml_vec_scale_mul_f32(n, y, x, w, s) computing y[i] = (x[i]*s)*w[i] with the existing GGML_SIMD macros. Paths are provided for the generic SIMD build (AVX2/NEON, 4-accumulator idiom matching ggml_vec_scale_f32), SVE (2× unroll + predicated tail), and RISC-V, plus a scalar fallback. The two multiplies are kept separate (no FMA) so the result matches the original scalar expression.

Validation

• Correctness: the benchmark suite builds and runs cleanly (the fused path asserts scale > 0.0f internally). Kept as two separate multiplies so the result is numerically equivalent to the original scalar expression.
• Performance (CodSpeed simulation, matched scope, both runs produced in this sandbox):

Benchmark	Base	Head	Change
rms_norm	563.2 µs	325.5 µs	+73.05%

The other 14 benchmarks are unchanged; no regressions.

[codspeed-hq]

Merging this PR will improve performance by 73.53%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	rms_norm	561.2 µs	323.4 µs	+73.53%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-vectorize-the-fused-rms-norm-kernel-ggml-compute-f-1785230254392 (c1db5d3) with master (46819c9)}