ggml-cpu: hoist loop-invariant constant and 2-block unroll in AVX2 ggml_vec_dot_q4_0_q8_0

[codspeed-hq]

Summary

Optimizes the AVX2 implementation of ggml_vec_dot_q4_0_q8_0 in ggml/src/ggml-cpu/arch/x86/quants.c, the dot-product kernel that dominates Q4_0 matrix multiplication (mul_mat[q4_0]) — the bulk of every linear/projection layer when running Q4_0-quantized weights.

Analysis

Flamegraph analysis of the mul_mat[q4_0] benchmark (baseline ~101.9 ms, 92% instruction-bound) showed nearly all time inside ggml_vec_dot_q4_0_q8_0. Two cheap-to-fix instruction overheads were visible in the per-block loop:

1. The nibble-offset vector off = _mm256_set1_epi8(8) was rebuilt inside the block loop on every iteration even though it is loop-invariant.
2. The loop ran one block per iteration with a single FMA accumulator, so the loop-control arithmetic (index increment, bound check, branch) was paid once per block.

Change

• Hoist the loop-invariant off vector out of the block loop (built once).
• Process two blocks per iteration using two independent FMA accumulator chains (acc0, acc1), with a small tail loop for an odd block count. The two accumulators are summed before the existing horizontal reduce.

This roughly halves the loop-control overhead per block and removes the redundant per-block constant materialization. The change is confined to the __AVX2__ path; the __AVX__/__SSSE3__/scalar paths are untouched, and the block math itself is unchanged.

Correctness

The result is mathematically the original sum, only regrouped across two accumulators before the reduce (same precedent as the existing f32 dot-product accumulator grouping). Verified locally with test-quantize-fns (AVX2 build): all types pass, including q4_0, so the dot-product error stays within tolerance.

Performance (CodSpeed, CPU simulation)

Measured with the CodSpeed CLI in an identical local environment (base = parent commit 46819c9, head = this change):

Benchmark	Metric	Base	Head	Δ
mul_mat[q4_0]	cpuTotal	101.91 ms	98.35 ms	−3.5%
mul_mat[q4_0]	instructions	93.85 ms	90.30 ms	−3.8%

Cache and memory components are unchanged (deterministic instruction-count reduction), and the other 14 benchmarks are unchanged — no regressions.

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-hoist-loop-invariant-constant-and-2-block-unroll-i-1785156502235 (c739572) with master (46819c9)}