ggml-cuda: GEMM weight row padding + one-time K-padded f16 dequant for prefill

[roberteg16]

Summary

Ports two prefill GEMM optimizations onto the refactored gfx11 CUDA matmul path (the old ggml_cuda_op_mul_mat / ggml_cuda_op_mul_mat_cublas were replaced by the templated ggml_cuda_mul_mat_cublas_impl<> + ggml_cuda_mul_mat), plus a follow-up that moves the quantized-weight dequant from inference time to load time.

• Pad GEMM weight rows to avoid cache-set aliasing (default ON, disable with GGML_CUDA_NO_PAD_WEIGHTS). When a float weight's packed row size is a multiple of 2048 bytes, consecutive rows collide in the same LLC sets. The loader tags such weights with GGML_TENSOR_FLAG_PAD_ROWS; the CUDA buffer reserves and applies a one-cache-line (128 B) padded nb[1]. The three kernel paths (mmf, mmvf, cuBLAS impl) all derive their leading dimension from nb01, so the padded stride is honored natively. Quantized weights are excluded (128 B is not a multiple of their block size).
• Load-time K-padded f16 shadow for prefill GEMMs (opt-in GGML_PREFILL_DEQUANT=<min_tokens>, off by default). For large-batch quantized dense GEMMs (ne11 >= min_tokens) the prefill path skips MMQ/MMVQ and runs an f16 hipBLAS GEMM. Each eligible quantized 2D dense weight is dequantized once at load time (ggml_backend_cuda_buffer_set_tensor) into a persistent f16 shadow owned by the buffer context (freed with the model), stored in tensor->extra. The shadow is K-padded by one cache line only when its f16 row size aliases at 2048 B, so weights with K % 1024 != 0 (e.g. a down_proj with K=3584) also get a shadow instead of being re-dequantized per call. The dispatch only bypasses MMQ for weights that actually have a shadow, so mul_mat_id expert slices stay on MMQ (MoE out of scope in v1). Decode (small batch) is untouched.

This supersedes an earlier runtime, pointer-keyed dequant cache: it removes the per-call dequantize_block_* kernels from the timed graph entirely (verified via roofline) and covers the K % 1024 != 0 weights the runtime cache missed.

Measured impact (gfx1151, Radeon 8060S 32 GB, Q4_K_M, -p 128,1024,4096 -n 128 -d 128)

Padding alone (default): no regression on quantized models (within run-to-run noise), as expected since quantized weights are excluded — it targets F16/BF16/F32 weights.

Prefill with GGML_PREFILL_DEQUANT=32 ROCBLAS_USE_HIPBLASLT=1 (load-time shadow) vs baseline (t/s). pp1024/pp4096 at default reps; pp128 averaged over -r 40 (matched baseline) since it is small-batch and jittery:

Model	test	baseline	deq+hipBLASLt	Δ
Qwen3.5 0.8B	pp128	7638	7174	-6.1%*
	pp1024	9503	9665	+1.7%
	pp4096	9318	9234	-0.9%
Qwen2.5 3B	pp128	2817	2766	-1.8%
	pp1024	3061	3734	+22.0%
	pp4096	3049	3396	+11.4%
gemma-3 4B	pp128	2520	2268	-10.0%
	pp1024	2570	3172	+23.4%
	pp4096	2358	2809	+19.1%
Llama-3.1 8B	pp128	1293	1232	-4.7%
	pp1024	1384	1829	+32.2%
	pp4096	1284	1693	+31.9%

* = Qwen3.5 0.8B pp128 stays noise-dominated even at -r 40 (±~1130 t/s); the tiny GEMMs are bound by launch/scheduling overhead, so treat it as inconclusive. Decode (tg128) is flat across all configs, as expected (the shadow only feeds prefill).

Moving the dequant to load time notably improves the models whose down_proj has K=3584 (missed by the earlier runtime cache): gemma-3 4B pp1024 went from +10% to +23%, Qwen2.5 3B pp4096 from +6% to +11%. Roofline confirms the per-call dequantize_block_q4_K/q6_K kernels no longer appear in the timed prefill graph.

Batch-size dependence: the f16 path is a large-batch win (pp1024/pp4096) but is a small, consistent regression at pp128 (-2% to -10% with -r 40), where the activation-convert + GEMM overhead is not amortized and MMQ is already efficient. Note min_tokens is compared against the per-u-batch token count (ne11, capped at n_ubatch, default 512), not the full prompt, so pp1024 and pp4096 both run M=512 GEMMs and only differ in u-batch count. The crossover is therefore M=128 (loses) vs M=512 (wins): min_tokens should be tuned between 128 and 512 (e.g. ~256); 32 is used here only to exercise the path, and any value > n_ubatch disables the path entirely (verified: GGML_PREFILL_DEQUANT=2048 at pp1024 stays on MMQ with zero runtime dequantize_block_*).

u-batch sensitivity: raising -ub to 2048 (so pp1024/pp4096 run M=1024/2048 GEMMs) does not improve throughput — the f16 advantage over MMQ persists (Llama-3.1 8B +21-22%, Qwen2.5 3B +8-11%) but absolute t/s is flat-to-slightly-worse vs the default u-batch (Llama 8B pp4096 1693 -> 1625). The M=512 GEMMs already saturate compute, so the default n_ubatch=512 is the sweet spot for this path.

Known limitation

The f16 shadows are kept resident alongside the quantized originals (decode still uses MMVQ on the quantized weight), so enabling GGML_PREFILL_DEQUANT roughly quadruples weight VRAM. On 32 GB, models >=12B OOM at load time (fails early and clearly, instead of mid-inference). Off by default, so unaffected unless opted in.

Test plan

• test-backend-ops test -o MUL_MAT (F16/BF16/F32 + quantized)
• llama-bench across 8 GGUF models, default flags: no crashes / no NaN, no regression
• Default (flag unset): no shadows built; Llama-8B/gemma-12B unchanged, no OOM
• llama-bench (4 models <=8B) with GGML_PREFILL_DEQUANT=32 ROCBLAS_USE_HIPBLASLT=1, -p 128,1024,4096: results above
• Roofline (GGML_ROOFLINE_OUT) with the flag: per-call dequantize_block_* gone from timed prefill (incl. K=3584 down_proj)