ggml-cpu: 2x2 register-blocked AVX2 kernel for ggml_vec_dot_q4_0_q8_0

[codspeed-hq]

What

Prompt processing multiplies each Q4_0 weight row against many activation columns, but the x86 kernel handled one (row, column) pair per call, so for every output element it re-unpacked the 4-bit weights, re-converted the two FP16 block scales and re-loaded the Q8_0 activation block.

Two changes to the AVX2 path:

1. An nrc == 2 path in ggml_vec_dot_q4_0_q8_0 computing a 2x2 output tile (two src0 rows x two src1 columns). Each row's nibbles are unpacked once and each column's Q8_0 block is loaded once, then reused by all four dot products. Q4_0 gets nrows = 2 under AVX2, which is what ggml_compute_forward_mul_mat already does on aarch64 with i8mm.
2. A cheaper nibble unpack in the shared bytes_from_nibbles_32() helper: vbroadcasti128 + a per-qword vpsrlvq replaces load + vpsrlw + vinserti128, removing one instruction per 32 weights in every AVX2 4-bit kernel.

The inner loop drops from ~21 instructions per 32 weights per dot product to ~13, and the weight/activation traffic through L1 is halved.

Measured impact

CodSpeed CPU simulation, micro benchmark suite (ggml-benchmarks), base 46819c9 vs this commit:

Benchmark	Base	Head	Change
mul_mat[q4_0]	101.9 ms	65.7 ms	-35.5% (+55% faster)

The other 14 benchmarks are unchanged, no regressions (mul_mat[f32], mul_mat[q4_k], quantize_chunk[*], ffn_swiglu, rope, rms_norm, soft_max, fp16/fp32 conversions).

Walltime could not be measured locally (the sandbox has no perf), so the macro job in CI is the first walltime data point. The aarch64 macro runners already select a 2-row kernel for Q4_0 via i8mm, so the walltime job is expected to be neutral there.

Correctness

Results are bit-identical, not just close:

• The rewritten bytes_from_nibbles_32() was compared against the old sequence exhaustively (every byte value in every one of the 16 positions) plus 2M random inputs - output vectors identical, and the low/high-nibble semantics verified directly.
• End-to-end through ggml_mul_mat: a Q4_0 matmul computed with the 2x2 kernel is bitwise equal to dedicated single-column matmuls (which take the 1x1 kernel), across several shapes at 1, 2 and 4 threads, including odd shapes that fall back to 1x1. The same matmul built against the base commit produces byte-identical output. The per-block accumulation order is deliberately kept identical to the 1x1 path.
• arch/x86/quants.c still compiles for SSSE3-only and AVX-only targets (the new path is inside the existing __AVX2__ guard).

Scope

x86/AVX2 only; the non-AVX2 paths and the assert(nrc == 1) they rely on are untouched.

[codspeed-hq]

Merging this PR will improve performance by 55.13%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[q4_0]	102 ms	65.7 ms	+55.13%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-2-2-register-blocked-avx2-kernel-for-ggml-vec-dot-1785326413536 (418d842) with master (46819c9)}