Optimize AVX2 ggml_vec_dot_q4_0_q8_0: two FMA accumulator chains

[codspeed-hq]

Summary

Optimizes the AVX2 implementation of ggml_vec_dot_q4_0_q8_0 (ggml/src/ggml-cpu/arch/x86/quants.c), the per-row dot-product kernel used by every Q4_0 matrix multiplication. ggml_compute_forward_mul_mat calls it once per output element, so it dominates every Q4_0 linear/projection layer of transformer inference (prompt processing, decode and the LM head).

Analysis

On the CodSpeed micro suite, mul_mat[q4_0] (~102 ms in CPU simulation) spends essentially all of its time in this kernel. The AVX2 inner loop processed one 32-element block per iteration and accumulated every block into a single __m256 acc via _mm256_fmadd_ps. That forms a serial dependency chain — each block's FMA must wait for the previous block's FMA to retire — so the loop is FMA-latency bound, and the per-iteration loop-control work is paid on every block.

Change

Process two blocks per iteration into two independent accumulators (acc0, acc1), each with its own scale and nibble/maddubs chain, then combine them with a single _mm256_add_ps before the existing hsum_float_8 reduction. A trailing scalar-style loop handles an odd final block. The per-block computation (bytes_from_nibbles_32, the -8 offset, mul_sum_i8_pairs_float, the FMA) is byte-for-byte the same as before — only the accumulator that receives it changes. This keeps two FMA chains in flight to hide latency and halves the loop-control overhead. The change is confined to the #if defined(__AVX2__) branch; the AVX/SSSE3/scalar paths are untouched.

Correctness

The result is mathematically equivalent: the two accumulators sum the same per-block products, just partitioned across two lanes, and are added before the horizontal reduce — the same class of float-reassociation the kernel already performs across its 8 SIMD lanes.

Performance (CodSpeed, CPU simulation)

Measured locally through the CodSpeed CLI on the mul_mat micro benchmarks:

Benchmark	Metric	Base	Head	Change
mul_mat[q4_0]	cpuTotal	0.101908 s	0.098354 s	-3.5%
mul_mat[q4_0]	instructions	0.093849 s	0.090295 s	-3.8%
mul_mat[q4_0]	cache	0.0047673 s	0.0047674 s	identical
mul_mat[q4_0]	memory	0.0032914 s	0.0032915 s	identical

The cache and memory components are byte-for-byte identical — the win is a pure, deterministic instruction-count reduction. mul_mat[f32] and mul_mat[q4_k] are unchanged (no regressions). The kernel also feeds the walltime macro benchmarks' Q4_0 layers (prompt_layer[q4_0], decode_layer[q4_0]), which exercise the same hot path.

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-unroll-the-avx2-ggml-vec-dot-q4-0-q8-0-loop-into-t-1785229421638 (fa9c3be) with master (46819c9)}