ggml-cpu: cache-aware tiling in ggml_compute_forward_mul_mat

[codspeed-hq]

Summary

Two tiling fixes in the chunked CPU matmul (ggml_compute_forward_mul_mat). Neither touches the arithmetic: every output element is still one vec_dot over the full row, so results are bit-identical.

1. Do not split dim1 when the src0 rows already provide enough chunks.
Each chunk streams all the src0 rows it covers, so splitting dim1 makes the weight matrix be read from memory once per dim1 chunk. With chunk_size = 16, a 64-token batch produced nchunk1 = 4, i.e. the weights were streamed 4 times per matmul. When nchunk0 alone already gives at least 4 chunks per thread (the same criterion the existing "poor chunking" fallback uses), all of dim1 now stays in one chunk and the weights are read once. Dynamic chunk stealing is preserved: with nchunk0 >= nth*4 every thread still gets at least four chunks to balance on.

2. Size the row block so its rows stay in L1.
Inside a chunk, the rows of a block are re-read for every column of that block. blck_0 was hardcoded to 16, and 16 rows of F32 weights (K = 2048 -> 128 KiB) is several times L1, so every column re-read all of its rows from L2. blck_0 is now halved until the block's rows fit a 24 KiB budget (F32 K=2048 -> 2 rows, Q8_0 -> 8, Q4_K/Q4_0 -> unchanged at 16). If a single row already exceeds the budget, nothing is reusable either way and 16 is kept.

Measured impact

Micro suite (CPU simulation, single-threaded, deterministic), full 15-benchmark runs of benchmarks/bench_ggml.cpp, base = this branch's merge base, head = this change:

Benchmark	Base	Head	Change
mul_mat[f32]	346.97 ms	316.33 ms	-8.8%
ffn_swiglu	1.070 s	991.6 ms	-7.3%
mul_mat[q4_0]	101.91 ms	101.71 ms	-0.2%
mul_mat[q4_k]	71.72 ms	71.42 ms	-0.4%

The gain is exactly where the model predicts it: on mul_mat[f32] the memory-access penalty drops 40.5 ms -> 20.5 ms (2 passes over the weights -> 1) and the cache-miss penalty drops 34.2 ms -> 20.0 ms (row block now L1-resident), against a +1.3% increase in executed instructions from the extra block iterations. The 11 non-matmul benchmarks are unchanged, so there are no regressions elsewhere.

Earlier macro-suite measurements (walltime, 8 threads) showed prompt_layer[f32] 3.31 s -> 2.98 s (-10.0%) and prompt_layer[q8_0] 1.01 s -> 944 ms (-6.5%); the walltime job on this PR re-measures that on the macro runners.

Correctness

Bit-exact checksums of ggml_mul_mat outputs were compared between an unmodified build and this build: 135 cases (F32/F16/Q8_0/Q4_0/Q4_K x 8 shapes with K/M/N including odd values and partial chunks x 1/3/8 threads, plus batched 3-D cases up to 32 heads). All checksums are identical.

Caveats

The 24 KiB row budget is a heuristic tuned against the simulation cache model (a 32 KiB budget measured worse, +2.3% on mul_mat[f32]). The dim1 change is a pure memory-traffic reduction and is architecture independent.

[codspeed-hq]

Merging this PR will not alter performance

✅ 28 untouched benchmarks

---

_{Comparing codspeed-optim-cache-aware-tiling-in-ggml-compute-forward-mul-mat-1785319430773 (5bf0bc3) with master (46819c9)}