ggml-cpu: unpack the Q4_K scales/mins with SIMD in the AVX2 ggml_vec_dot_q4_K_q8_K

[codspeed-hq]

Summary

ggml_vec_dot_q4_K_q8_K dominates the quantized transformer path in the CodSpeed profiles (~95% of mul_mat[q4_k], ~83% of lm_head[q4_k], ~76% of prompt_layer[q4_k]). A large share of its cost is the per-superblock prologue rather than the multiply-accumulate loop: the scalar utmp[] unpack of the 12 packed 6-bit scales/mins costs a 12-byte memcpy, ~20 general-purpose shift/and/or operations, and then a store of utmp[] followed by a reload through _mm_set_epi32 to get the bytes back into a vector register (a store-to-load forwarding round trip on every superblock).

Change

ggml/src/ggml-cpu/arch/x86/quants.c, AVX2 branch only: the same 12 -> 16 byte expansion is now done entirely in vector registers with two pshufb byte shuffles plus three masked shifts/ors, with the 5 shuffle/mask constants hoisted out of the loop. The dot-product loop, the mins/bsums handling and the accumulation are untouched. The prologue drops from 45 to 28 instructions per 256-weight superblock and the utmp[] store/reload disappears.

Scope: x86 AVX2 only (the branch used by the simulation CI job and by most x86 CPUs without VNNI). ARM, plain AVX and the generic paths are untouched.

Correctness

Verified locally in this sandbox (AVX2 host):

• The unpack is bit-exact, not an approximation. A standalone harness compared the new SIMD sequence against the original scalar utmp[] transform over 2,000,000 random 12-byte inputs, all 12x256 single-byte-position cases, and the all-0x00/0xff/0xaa/0x55/0x0f/0xf0/0xc0/0x3f patterns: 0 mismatches.
• End-to-end A/B: 512 Q4_K x Q8_K dot products over 8-superblock rows of pseudo-random data produced a bit-identical FNV hash of all results before and after the change (0x6d5df0ec45020523).
• tests/test-quantize-fns.cpp passes against the patched build (no failures, including q4_K).
• The 16-byte load starts at scales[0]; block_q4_K is {d, dmin, scales[12], qs[128]}, so the 4 trailing bytes belong to qs[] of the same 144-byte superblock and the load can never read out of bounds.

Measured impact (CodSpeed, CPU simulation, this sandbox)

Benchmark	Base	Head	Change
mul_mat[q4_k]	71.7 ms	62.9 ms	+14.07%
mul_mat[q4_0]	101.9 ms	101.9 ms	unchanged
mul_mat[f32]	347.0 ms	347.0 ms	unchanged

No regressions. The macro (walltime) benchmarks are expected to improve on lm_head[q4_k] and prompt_layer[q4_k]; decode_layer[q4_k] is bandwidth-bound (a single token streaming the full weight matrices), so removing instructions does not move it.

[codspeed-hq]

Merging this PR will improve performance by 14.85%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[q4_k]	72.4 ms	63 ms	+14.85%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-unpack-the-q4-k-scales-mins-with-simd-in-avx2-ggml-1785298752860 (3db79c4) with master (46819c9)}