ggml-cpu: 2x2 register-blocked kernel for ggml_vec_dot_f16

[codspeed-hq]

What

ggml_compute_forward_mul_mat() calls ggml_vec_dot_f16() once per (src0 row, src1 column) pair, so for an F16 matmul every weight vector is loaded — and, on targets without native f16 arithmetic, converted to f32 — once per column of the tile.

This adds an nrc == 2 path to ggml_vec_dot_f16() that computes a 2×2 output tile (two src0 rows × two src1 columns) in one pass, and enables nrows = 2 for GGML_TYPE_F16 on the targets that use the generic GGML_F16_VEC macros. The SVE and RVV paths have their own hand-written loops and keep nrows = 1.

Each loaded vector now feeds two of the four accumulators: 4 loads + 4 FMAs per vector position instead of 8 loads + 4 FMAs. The tile also keeps its operands and accumulators in named register variables instead of the GGML_F16_VEC ax[GGML_F16_ARR] stack arrays the 1×1 loop uses, which at -O2 (the RelWithDebInfo benchmark build) GCC does not unroll — so the accumulator is reloaded from and stored back to the stack around every FMA there. The generated tile loop is pure loads + FMAs, with no spills, on both x86-64 (AVX2/F16C) and aarch64.

Measured effect

CodSpeed CPU simulation on the macro benchmarks, base (46819c9) vs head, same sandbox:

Benchmark	Base	Head	Change
lm_head[f16]	476.2 ms	151.9 ms	×3.1
lm_head[q8_0]	170.9 ms	171.5 ms	unchanged
lm_head[q4_k]	131.2 ms	133.8 ms	unchanged

The blast radius is limited to F16 matmuls: no other benchmark in the suite uses an F16 src0, and the two quantized lm_head variants are reported unchanged. Wall-time numbers on the macro runner will come from this PR's macro-benchmarks job — the walltime instrument could not be run in the sandbox (it needs linux-tools/perf, unavailable there).

Correctness

Results are no longer bit-identical to the 1×1 path (the tile accumulates in two groups per output instead of GGML_F16_ARR), so the change was validated numerically rather than by comparison:

• F16×F32 matmuls checked against a double-precision reference computed from the same f16-rounded inputs, over 1404 combinations (K ∈ 1…2048, M ∈ 1…100, N ∈ 1…9 — including odd sizes that must fall back to nrc = 1 — with 1/2/3/8 threads): max relative error 7.3e-7, NMSE 9.0e-14, no failures.
• The test was confirmed to actually exercise the new path: injecting a deliberate error into the tile output makes 297 of the 1404 cases fail.
• Compile-checked for SSE3, AVX2 (with and without F16C), AVX-512 (with and without FP16), aarch64 (armv8-a, armv8.2-a+fp16, +sve), s390x/VXE (z16) and RISC-V RVV. POWER9 does not compile vec.cpp with a plain cross-compiler on this branch or on the base commit (pre-existing vector float vs std::vector clash in ggml_vec_dot_f16_unroll), so it could not be checked.

Note on the POWER9 macro: GGML_F16_VEC_LOAD(p, i) derives from the parity of i which half of the loaded 16-byte block to convert, so the tile passes the index of the vector within the step (0 for the first half, 1 for the second) rather than an operand id, matching the existing 1×1 loop's usage.

One behavioural note for targets with native f16 vector arithmetic (aarch64 +fp16): as before, accumulation happens in f16, but each output now has 2 accumulator chains instead of GGML_F16_ARR (4), i.e. ~√2 more accumulation error on an already-f16 accumulation.

[codspeed-hq]

Merging this PR will improve performance by ×2.3

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	WallTime	lm_head[f16]	60.6 ms	26.4 ms	×2.3

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-2-2-register-blocked-kernel-for-ggml-vec-dot-f16-f-1785342079645 (7de386d) with master (46819c9)}