ggml-cpu: vectorize the fused RMS-norm × weight kernel

[codspeed-hq]

Summary

Vectorizes both scalar loops of the fused RMS-norm path (ggml_compute_forward_rms_norm_f32<GGML_RMS_NORM_FUSE_OP_MUL> in ggml/src/ggml-cpu/ops.cpp). This computes rms_norm(x) * w, the kernel applied before every attention and feed-forward block, so it is a genuine inference hot path.

Why this path

The walltime macro benchmarks are dominated by this kernel: on the CodSpeed macro (aarch64) runner, ggml_compute_forward_rms_norm_mul_fused accounts for ~52% of self time in prompt_layer[q4_k]. The rms_norm micro-benchmark exercises the exact same fused template, and its self-time breakdown was ~56% instructions / ~42% memory — a classic unvectorized scalar loop with the vector units idle.

Both loops in the fused path were plain scalar (the reduction even carried the code's own // worth switching to explicit SIMD? comment):

• the per-row sum-of-squares reduction, and
• the output loop y[i] = x[i] * scale * w[i].

ggml-cpu is built at -O2 where the compiler does not auto-vectorize these, so they ran scalar over the whole hidden dimension for every row.

Change

• ops.cpp: replaced the scalar sum-of-squares with ggml_vec_dot_f32(ne00, &sumf, 0, x, 0, x, 0, 1) since sum(x*x) == dot(x, x) (the same idiom already used by ggml_vec_norm_f32), and replaced the scalar output loop with a new helper.
• vec.h: added ggml_vec_scale_mul_f32(n, y, x, w, s) computing y[i] = (x[i]*s)*w[i] with the existing GGML_SIMD 2-accumulator idiom (SVE / RISC-V / generic paths + scalar fallback), matching the structure of ggml_vec_scale_f32. The two multiplies are kept separate (no FMA) so each product is rounded exactly as in the scalar expression.

Validation

• Correctness: test-backend-ops builds cleanly; a standalone check of ggml_vec_scale_mul_f32 against a double-precision scalar reference over a 2048×37 (odd-row, leftover-exercising) input was bit-identical to the scalar expression (max_abs = 0, max_rel = 0) on the compiled SIMD path.

• Performance (CodSpeed simulation, this repo's benchmark suite, matched base vs head):

  Benchmark	Base	Head	Change
  rms_norm	561.2 µs	325.5 µs	+72.45%

  The other 14 benchmarks are unchanged — no regressions.

[codspeed-hq]

Merging this PR will improve performance by 73.53%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	rms_norm	561.2 µs	323.4 µs	+73.53%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-vectorize-the-fused-rms-norm-weight-kernel-ggml-co-1785215968422 (8f1eb12) with master (46819c9)}