ggml-quants: evaluate the k-quant scale search error in O(1) in make_qkx2_quants

[codspeed-hq]

Summary

Rewrote the sum-of-squares scale/min search in make_qkx2_quants (ggml/src/ggml-quants.c), the hot inner routine of k-quant quantization used by q4_K and q5_K.

Bottleneck

Flamegraphs of the quantize_chunk[q4_k] simulation benchmark show ~90% of time in make_qkx2_quants, ~99% instruction-bound. For each of the nstep+1 (21 for q4_K) candidate scales, the original code did two full passes over the block:

1. a pass computing nearest_int, storing the candidate into a Laux[] buffer, and accumulating sum_l/sum_l2/sum_xl;
2. a second pass re-reading Laux[] to accumulate the reconstruction error;

plus a Laux[]→L[] copy every time a candidate improved the best error.

Change (for the use_mad == false objective only)

• The sum-of-squares error Σ w·(a·L + b − x)² is a quadratic form in the fitted (scale, min) whose coefficients (sum_l, sum_l2, sum_xl, sum_w, sum_x, sum_wx2) are already accumulated during the first pass. It is now evaluated in O(1), removing the entire second scan.
• Because the error no longer depends on a materialized candidate, the Laux[] staging store is removed from the sweep and the Laux→L copy is removed from the improvement branch. Only the winning (iscale, min) is remembered, and the final L[] is recomputed once after the sweep.
• The use_mad == true path (q2_K) is left byte-for-byte unchanged.

The winning-candidate reproduction uses the min value as it stood at the start of the winning iteration (min is only updated after a candidate is accepted), so the emitted L[] matches the original algorithm exactly.

Correctness

tests/test-quantize-fns.cpp built against the modified ggml: all quant types pass, and for the affected types the fast path is bit-identical to the reference:

• q4_K reference implementation error: ok (0.000000), dot-product error 0.002318 (unchanged from baseline)
• q5_K reference implementation error: ok (0.000000), dot-product error 0.000957 (unchanged)
• q2_K (unchanged code path) still ok

Measured impact (CodSpeed, CPU simulation)

Benchmark	Base	Head	Change
quantize_chunk[q4_k]	10.3 ms	3.9 ms	×2.7

quantize_chunk[q6_k] is unchanged (it uses make_qx_quants, a different routine), and all other benchmarks are unchanged — confirming no collateral regression.

[codspeed-hq]

Merging this PR will improve performance by ×2.7

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 27 untouched benchmarks

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	quantize_chunk[q4_k]	10.4 ms	3.9 ms	×2.7

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codspeed-optim-evaluate-the-k-quant-scale-search-error-in-o-1-and-1785213170675 (8e3fe21) with master (46819c9)}