ggml-cpu: 2x2 register-blocked kernel for ggml_vec_dot_f32 (F32 matmul)

[codspeed-hq]

Summary

F32 matrix multiplication in ggml_compute_forward_mul_mat() computes one output element per ggml_vec_dot_f32() call. Each call streams a full src0 row and a full src1 column, so evaluating a 2x2 output tile re-reads both operands twice.

This PR adds a 2x2 register-blocked kernel (nrc == 2): two src0 rows against two src1 columns in a single pass. ggml_compute_forward_mul_mat() already supports this through ggml_type_traits_cpu::nrows (it is the path used by the ARM i8mm kernels), so only the kernel and the F32 trait entry change.

Why it helps

Every loaded vector feeds two FMAs instead of one, so the four dot products of a tile issue half the loads and touch half the cache lines of four independent calls. Per 4 FMA-vectors the loop costs 2.25 instructions instead of 2.5, and the L1/L2 traffic of the weight matrix is halved.

• Four accumulator pairs x two groups (12 live vectors) keep enough independent FMA chains in flight without spilling on AVX2 (16 regs).
• Written entirely with the generic GGML_SIMD mappings, so AVX/AVX2/AVX512, NEON, VSX, wasm and LoongArch all benefit - not just x86.
• SVE / RISC-V V (which have their own hand-written paths) and non-SIMD targets fall back to four ordinary dot products and keep nrows = 1, so they are strictly unaffected.

Measured impact (CodSpeed, CPU simulation)

Base = master (46819c9), head = this branch, same machine, same build flags (-DCODSPEED_MODE=simulation, AVX2/FMA/F16C):

Benchmark	Base	Head	Change
ffn_swiglu	1,069.7 ms	857.8 ms	-19.8%
mul_mat[f32]	347.0 ms	282.2 ms	-18.7%

The 13 other micro benchmarks are unchanged (mul_mat[q4_0], mul_mat[q4_k], quantize_chunk[*], rope, rms_norm, soft_max, ... all bit-identical), which is expected: only the F32 trait entry changed.

Walltime could not be measured locally (the sandbox has no perf), so the macro suite result comes from this PR's codspeed-macro job. decode_*[f32] are expected to be unchanged: with a single token ne11 == 1, so the 2x2 path is not selected.

Correctness

test-backend-ops skips the CPU backend when it is the only device, so the kernel was validated against a double-precision reference over 1440 matmul shapes (K in 1...5632 including odd values, M in 1...64, N in 1...32, 1 and 4 threads), on both the AVX-512 (-march=native) and AVX2 builds. Maximum absolute deviation is 3.6e-5 on outputs of magnitude ~90 (~4e-7 relative) - pure float32 re-association noise, the same class of reordering the existing 4-accumulator SIMD reduction already performs.

The nr0 % 2, ne11 % 2 and chunk-parity guards in ggml_compute_forward_mul_mat() already prevent the 2-row path from crossing tile or dim-1 boundaries, and the tmp[32] staging buffer is already sized for two columns. ggml_compute_forward_mul_mat_id() always passes nrc == 1 and is unaffected.

Cross-target compile checks of vec.cpp passed for SSE3, AVX, AVX2, AVX-512, scalar (no GGML_SIMD), aarch64 (NEON, SVE, SVE2), s390x VXE and riscv64 rv64gcv.

[codspeed-hq]

Merging this PR will improve performance by 20.88%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
✅ 25 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	ffn_swiglu	1,067.7 ms	858.1 ms	+24.42%
⚡	Simulation	mul_mat[f32]	346.1 ms	282.1 ms	+22.7%
⚡	WallTime	prompt_layer[f32]	247.1 ms	213.6 ms	+15.69%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-2-2-register-blocked-kernel-for-ggml-vec-dot-f32-f-1785295416304 (5a7c570) with master (46819c9)}