ggml-cpu: keep the generic SIMD vec kernel accumulators in registers

[codspeed-hq]

Problem

The generic GGML_SIMD kernels in vec.h / vec.cpp hold their vector operands and accumulators in small arrays indexed by the inner loop counter:

GGML_F32_VEC sum[GGML_F32_ARR] = { GGML_F32_VEC_ZERO };
for (int i = 0; i < np; i += GGML_F32_STEP) {
    for (int j = 0; j < GGML_F32_ARR; j++) {   // constant trip count (4)
        ...
        sum[j] = GGML_F32_VEC_FMA(sum[j], ax[j], ay[j]);
    }
}

Those arrays only end up in registers if the inner loop is unrolled. GCC unrolls it at -O3, but not at -O2 - which is what RelWithDebInfo builds use, including the benchmark jobs. At -O2 every FMA round-trips its accumulator through the stack, and ggml_vec_dot_f32 additionally re-zeroes its 128-byte accumulator array with a rep stos on every call (i.e. once per matmul output element).

Change

A new GGML_UNROLL(n) helper (#pragma GCC unroll n, no-op on compilers that do not support it) applied to the constant-trip inner loops of the generic SIMD kernels: ggml_vec_dot_f32, ggml_vec_dot_f16, ggml_vec_dot_f16_unroll, ggml_vec_mad_f32/_f16/_f32_unroll, ggml_vec_mad1_f32, ggml_vec_scale_f32/_f16.

+28 lines, 0 deletions. No arithmetic is touched: the operand partitioning, the number of accumulator groups and the reduction order are all unchanged - the accumulators simply stop living in memory. The hand-written SVE and RISC-V paths are untouched.

Verified on the built objects (x86-64, AVX2, -O2), ggml_vec_dot_f32:

	before	after
instructions in the function	58	51
rep stos (accumulator re-zeroing per call)	1	0
vfmadd in the vector loop	2	5

Measured impact

CodSpeed CPU simulation, base vs head of this branch, same build configuration as the codspeed.yml micro job:

Benchmark	Base	Head	Change
mul_mat[f32]	347.0 ms	232.5 ms	+49.2%
ffn_swiglu	1,069.7 ms	727.8 ms	+47.0%

The other 13 micro benchmarks are unchanged (the quantized kernels live in arch/*/quants.c and do not go through these code paths). The macro walltime suite exercises the same kernels - prompt_layer[f32] is ~99% ggml_vec_dot_f32, and lm_head[f16] goes through ggml_vec_dot_f16 - and is measured by CI on this PR.

Correctness

• Bit-identical results. F32 matmul, F16 matmul, rms_norm+mul, soft_max, silu, scale and 600 dot-product sizes (K = 1 ... 600, so the odd/leftover tails are exercised) were dumped from a build of the base commit and from the patched build; the two dumps compare byte-for-byte equal.
• test-quantize-fns, test-rope and test-barrier pass. test-backend-ops reports OK (it skips the CPU backend when no accelerator device is present).
• -O3 builds are unaffected: objdump of vec.cpp compiled at -O3 before and after the change is identical apart from the object file name, so builds that already use -O3 keep exactly the code they have today.
• No new compiler warnings with the full CI flag set (-Wall -Wextra -Wpedantic -Wcast-qual -Wextra-semi ...), and it also cross-compiles clean for aarch64 with both -march=armv8.2-a+fp16+dotprod and +sve.

Trade-off worth knowing

Because -O3 already produces this code, the win applies to builds compiled at -O2 (RelWithDebInfo), which is what the benchmark jobs use on both the simulation and the walltime macro runners. An alternative fix would be to build the benchmark targets with -O3 -g; this change instead makes the kernels insensitive to the optimization level.

[codspeed-hq]

Merging this PR will improve performance by 40.8%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
✅ 24 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[f32]	346.1 ms	232.6 ms	+48.81%
⚡	Simulation	ffn_swiglu	1,067.7 ms	727.9 ms	+46.69%
⚡	WallTime	lm_head[f16]	60.6 ms	42.6 ms	+42.19%
⚡	WallTime	prompt_layer[f32]	247.1 ms	195.1 ms	+26.62%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-keep-the-simd-vec-kernel-accumulators-in-registers-1785335630636 (0ebdd46) with master (46819c9)}