ggml-cpu: 2x2 register-blocked AVX2 kernel for ggml_vec_dot_q4_K_q8_K

[codspeed-hq]

Summary

Q4_K matrix multiplication on x86 computes one output element per ggml_vec_dot_q4_K_q8_K() call. Every call re-does the full per-block weight decoding - the 6-bit scale/min unpacking, the _mm256_cvtepu8_epi16 + shuffle broadcast of the scales, and the 4-bit nibble extraction - even though those depend only on the src0 row. Computing a 2x2 output tile therefore decodes the same weights twice.

This adds an nrc == 2 2x2 register-blocked path: two src0 rows against two src1 columns in one pass. ggml_compute_forward_mul_mat() already drives this through ggml_type_traits_cpu::nrows - it is the path the ARM i8mm kernels use, and GGML_TYPE_Q4_K already sets nrows = 2 there. x86 was simply missing the kernel, so this closes an existing gap rather than adding new machinery.

Files changed (2, +125/-1)

• ggml/src/ggml-cpu/arch/x86/quants.c - the blocked kernel plus a small hsum_float_4() helper.
• ggml/src/ggml-cpu/ggml-cpu.c - GGML_TYPE_Q4_K trait nrows = 2 also under __AVX2__.

Why it helps

Per 64-weight sub-block, the blocked kernel decodes each row's nibbles and scales once and feeds them into four maddubs/madd chains instead of four independent decodes. All four accumulators, both scale vectors and the nibble masks stay in registers.

Measured impact (CodSpeed, CPU simulation)

Benchmark	Base	Head	Change
benchmarks/bench_ggml.cpp::mul_mat[q4_k]	71.7 ms	52.9 ms	+35.7%
benchmarks/bench_macro.cpp::prompt_layer[q4_k]	789.7 ms	627.2 ms	+25.9%

Everything else is unchanged: mul_mat[f32], mul_mat[q4_0], prompt_layer[f32], prompt_layer[q8_0] and prompt_layer[q4_0] all report as unchanged (byte-identical values for the pure-f32/q4_0 micro benchmarks).

Correctness

The per-block accumulation order is identical to the existing 1x1 path, so results are bit-identical, not merely close. Verified locally by building the same ggml_mul_mat Q4_K test program against an unpatched and a patched libggml, dumping every output float, and comparing the raw bytes:

• 8 shapes (K = 256...5632, M = 16...512, N = 1...64, including odd M/N which fall back to the 1x1 path and N = 1 decode) x 1/2/4 threads - all outputs bit-for-bit identical (cmp clean over 529 KB of results).
• A deliberately poisoned nrc == 2 result confirms the new path is genuinely taken for the even shapes and not for the odd-M/N or N = 1 cases, so the bit-exactness is not a false negative.
• test-quantize-fns passes; test-backend-ops -o MUL_MAT is OK (a CPU-only run has no second backend to compare against).
• arch/x86/quants.c compiles warning-free with -msse4.2, -mavx -mf16c, -mavx2 -mfma -mf16c and -mavx512f/bw/vl/dq.

Scope

The 2x2 path activates only when nr0 % 2 == 0, ne11 % 2 == 0 and both chunk extents are even - i.e. prompt processing and the LM head. Single-token decode (ne11 == 1) keeps the existing 1x1 path and is untouched, as are all non-AVX2 targets and every other quantization type.

[codspeed-hq]

Merging this PR will improve performance by 36.42%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[q4_k]	72.4 ms	53.1 ms	+36.42%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-2-2-register-blocked-avx2-kernel-for-ggml-vec-dot-1785301877014 (471ee09) with master (46819c9)}