ggml-cpu: dequantize each weight row once per matmul tile (4-column Q4_0/Q4_K AVX2 kernels)

[codspeed-hq]

What

ggml_compute_forward_mul_mat() computes one output element per vec_dot call. For quantized weights this means the whole src0 side of the work — nibble extraction, the -8 offset, the |x| needed by the unsigned multiply, the 6-bit Q4_K scale/min unpacking and the FP16 block-scale conversions — is redone for every column of the tile, even though it only depends on the row.

Flamegraph analysis of mul_mat[q4_0] confirmed it: ggml_lookup_fp16_to_fp32 12.6%, the nibble/offset/mask intrinsics ~13%, _mm_loadu_si128 12%. For mul_mat[q4_k], srli + shuffle + and + memcpy + set_epi32 account for ~37%. All of it row-only work, repeated once per column.

This PR adds 4-column kernels (ggml_vec_dot_q4_0_q8_0_4cols, ggml_vec_dot_q4_K_q8_K_4cols, AVX2) that unpack the row once and reuse it for 4 src1 columns, plus a fast path in ggml_compute_forward_mul_mat_one_chunk() that walks the tile 4 columns at a time. Leftover columns and every other case keep the existing path.

Why it is safe

• The per-column arithmetic and the order of the accumulation are unchanged, so results are bit-identical.
• The fast path is restricted to plain 2-D matmuls (ne12 == ne13 == 1, no src0 broadcast) with nrows == 1; batched attention matmuls, GEMV decode steps (1 column) and all other types fall back unchanged. The tmp staging that limits false sharing between threads is preserved.

Verification performed on this PR

• Bit-identity harness: a standalone program computes ggml_mul_mat over 6,480 configurations — types q4_0, q4_K, f32, q8_0, q6_K × K ∈ {256, 512, 1024, 2560} × M ∈ {1, 3, 7, 16, 17, 33, 64, 129, 512} × N ∈ {1, 2, 3, 4, 5, 7, 8, 15, 16, 32, 33, 64} × {1, 3, 8} threads — and hashes the exact bit pattern of every output element. The hashes are identical between the base build and this build for all 6,480 cases (diff is empty).
• Fast path is actually exercised: verified with temporary instrumentation that both the Q4_0 and Q4_K 4-column kernels are entered by mul_mat[q4_0] / mul_mat[q4_k].
• Valgrind memcheck: a reduced version of the harness (odd M/N shapes, N ∈ {1, 4, 5, 9, 33} to cover the 4-column remainder) runs with 0 errors from 0 contexts.
• Repository tests: test-quantize-fns, test-opt and test-barrier pass. test-backend-ops -o MUL_MAT reports OK but skips the CPU backend (it is the reference), so it provides no additional coverage for a CPU-only change — hence the bit-identity harness above.

Measured impact — CodSpeed CPU simulation

Benchmark	Base	Head	Change
mul_mat[q4_0]	101.9 ms	58.7 ms	+73.7%
mul_mat[q4_k]	71.7 ms	46.5 ms	+54.3%

The other 13 benchmarks of the suite are unchanged (e.g. mul_mat[f32] 346.97 ms → 347.00 ms), no regressions.

Notes

• The kernels are AVX2-only for now. On other architectures ggml_get_vec_dot_4cols() returns NULL and the code path is unchanged; the NEON equivalents are the natural follow-up.
• Token generation (single-column GEMV) is unaffected by design — the win is on prompt processing / batched decoding, where the redundant dequantization is real.
• Local walltime validation could not be run in the sandbox (the perf tooling required by walltime mode is not installable there). The walltime macro job on this PR is the ground-truth validation for that mode; note that if the macro runners are aarch64 they will not exercise these AVX2 kernels.

[codspeed-hq]

Merging this PR will improve performance by 62.77%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 26 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	mul_mat[q4_0]	102 ms	59.6 ms	+71.03%
⚡	Simulation	mul_mat[q4_k]	72.4 ms	46.7 ms	+54.91%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-dequantize-each-weight-row-once-per-matmul-tile-4-1785338847703 (7baa65d) with master (46819c9)}