Hoist scale-shuffle masks out of the AVX2 ggml_vec_dot_q4_K_q8_K hot loop

[codspeed-hq]

Summary

Optimizes the AVX2 implementation of ggml_vec_dot_q4_K_q8_K (ggml/src/ggml-cpu/arch/x86/quants.c), the Q4_K × Q8_K dot product that dominates mul_mat for Q4_K-quantized weights. The flamegraph of mul_mat[q4_k] shows ~95% of the benchmark time inside this function.

Analysis

The inner quarter-block loop applies the per-sub-block scales via _mm256_shuffle_epi8(scales, get_scale_shuffle_k4(2*j+…)). get_scale_shuffle_k4() returns an _mm256_loadu_si256 from a constant 256-byte table. Because the loop indexes it with the run-time variable j, the compiler could not hoist those loads — so the 8 shuffle-mask loads were re-issued from the k_shuffle table on every superblock, even though the masks are data-independent constants.

Change

• Materialize the 8 shuffle masks once, before the superblock loop.
• Fully unroll the QK_K/64 == 4 quarter-block loop (via a local macro) so each mask is used as a compile-time-constant operand.

Everything else — the load/and/srli unpack, maddubs, madd_epi16 scaling and the integer/float accumulation — is byte-for-byte the same sequence of operations, just with the redundant table reloads removed and the tiny 4-iteration loop control eliminated. This is a pure unroll + constant-hoist, so the result is mathematically identical.

Correctness

Verified locally with test-backend-ops:

./build-test/bin/test-backend-ops test -o MUL_MAT -b CPU
1177/1177 tests passed
Backend CPU: OK

All Q4_K MUL_MAT cases (type_a=q4_K) pass, confirming the change is bit-consistent with the reference implementation.

Performance (CodSpeed, CPU simulation)

Benchmark	Base	Head	Change
mul_mat[q4_k]	71.7 ms	63.6 ms	+12.85%

No regressions in the other benchmarks. The win is an instruction-count reduction (fewer table loads in the hot loop), which is exactly what the simulation instrument measures.

[codspeed-hq]

Merging this PR will improve performance by 12.4%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[q4_k]	72.4 ms	64.4 ms	+12.4%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-hoist-scale-shuffle-masks-out-of-the-avx2-ggml-vec-1785163186472 (67db298) with master (46819c9)}