ggml-cuda: add dequant-float matvec (mmvdq) for Q4_K/Q5_K/Q6_K on RDN…

[Annieren]

MMVDQ change summary

What it adds: a dequantize-to-float matvec path (mmvdq) for K-quant weights
during decode (n=1), as an alternative to the existing mul_mat_vec_q (mmvq)
path.

Why: mmvq quantizes the activation vector to q8_1 first (a quantize_q8_1
pass) so it can do int8 dot products. On RDNA3.5 (gfx1151) these matvecs
are DRAM-bandwidth-bound on the weights, so the q8_1 quantize pass is
pure overhead. mmvdq dequantizes the weight blocks straight to float and
does the dot in float, skipping the quantize pass entirely.

Kernel design (mmvdq.cu): one warp per output block, 16 threads process one
super-block, activations loaded once (as float4/float2) and reused across the
block. Covers Q4_K, Q5_K, Q6_K. Includes a fused gate+up SwiGLU variant
(mul_mat_vec_dq_glu) that computes silu(gate)*up in one dispatch sharing the
activation load.

Routing / gating (ggml-cuda.cu):

• Non-fused path in ggml_cuda_mul_mat: triggers when ne11==1, type is
  Q4_K/Q5_K (or Q6_K if enabled), all tensors contiguous, ne02/03/12/13==1, and
  ne[0] % QK_K == 0.
• Fused SwiGLU path in ggml_cuda_try_fuse: requires ids==nullptr,
  GGML_GLU_OP_SWIGLU, matching gate/up shapes+type, single row.

Enablement (env overrides):

• GGML_CUDA_DQ_MMV — master switch: unset = arch default (on for RDNA3.5), 0
  = force off, non-zero = force on
• GGML_CUDA_DQ_Q6K — Q6_K gate, same semantics (Q6_K is bandwidth-bound and
  ~neutral vs mmvq)

Default is on only for RDNA3.5; every other arch keeps mmvq.

Full decode comparison (38 models)

Mean +3.09%, median +2.78% (n=38). * = controlled DQ-toggle run (DQ-off/mmvq
vs DQ-on/mmvdq) substituted for the three thermally-throttled full-suite
rows.

Model	Base t/s	MMVDQ t/s	Δ%
Gemma-4-E2B-IT_Q4_K_M_GGUF_128	95.15	105.09	+10.4%
Gemma-4-E2B-IT_Q4_K_M_GGUF_3968	89.86	98.22	+9.3%
Gemma-2B_Q4_K_M_GGUF_128	106.68	115.00	+7.8%
Janus-Pro-1B_VLM	148.35	158.28	+6.7%
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_128	143.74	153.18	+6.6%
Qwen3-1.7B_Q4_K_M_GGUF_128	139.69	147.69	+5.7%
Gemma-4-31B-IT_VLM	10.43	10.95	+4.9%
Qwen3-4B_Q4_K_M_GGUF_128	71.07	74.46	+4.8%
Qwen3-VL-4B-Instruct_VLM	66.35	69.31	+4.5%
Gemma-3-4B-IT_VLM	67.67	70.45	+4.1%
Qwen3.5-4B_Q4_K_M_GGUF_128	60.36	62.84	+4.1%
Qwen3-1.7B_Q4_K_M_GGUF_3968	106.89	111.27	+4.1%
Qwen3-4B_Q4_K_M_GGUF_3968	59.31	61.70	+4.0%
Llama-2-7B_Q4_K_M_GGUF_128	46.14	47.84	+3.7%
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_128	44.86	46.41	+3.5%
Janus-Pro-7B_VLM	43.50	44.81	+3.0%
Gemma-3-12B-IT_Q4_K_M_GGUF_VLM	25.73	26.49	+3.0%
Llama-2-7B_Q4_K_M_GGUF_1920	33.86	34.84	+2.9%
Gemma-3-12B-IT_Q4_K_M_GGUF_4096 *	23.71	24.37	+2.8%
Qwen3-8B_Q4_K_M_GGUF_3968	36.27	37.28	+2.8%
Qwen3-30B-A3B-Instruct-2507_Q4_K_M_GGUF_128	74.51	76.35	+2.5%
Gemma-2B_Q4_K_M_GGUF_8000	104.26	106.59	+2.2%
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_128	87.44	89.27	+2.1%
Qwen2.5-3B-Instruct_Q4_K_M_GGUF_3968	82.55	84.09	+1.9%
Qwen3.5-9B_Q4_K_M_GGUF_128	36.51	37.11	+1.6%
SmolLM2-1.7B-Instruct_Q4_K_M_GGUF_8000	36.19	36.77	+1.6%
Qwen3-8B_Q4_K_M_GGUF_128	41.46	42.12	+1.6%
Llama-3.1-8B-Instruct_Q4_K_M_GGUF_128	42.48	43.13	+1.5%
Qwen2.5-VL-3B-Instruct_VLM	84.45	85.56	+1.3%
Qwen2.5-7B-Instruct_Q4_K_M_GGUF_3968	43.26	43.64	+0.9%
MiniCPM-V-2_6_VLM	45.17	45.47	+0.7%
Qwen2.5-VL-7B-Instruct_VLM	44.63	44.90	+0.6%
MiniCPM-o-2_6_VLM	45.21	45.47	+0.6%
Qwen3.5-35B-A3B_Q4_K_M_GGUF_128 *	54.58	54.69	+0.2%
Qwen3.6-27B_Q4_K_M_GGUF_4096 *	12.22	12.24	+0.2%
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_128	318.81	318.94	+0.0%
Qwen2.5-0.5B-Instruct_Q4_K_M_GGUF_3968	291.28	290.39	-0.3%
Gemma-4-26B-A4B-IT_VLM	47.30	47.07	-0.5%

[roberteg16]

Correct me if my understanding is wrong. We go from:

fp32
|
v
quant  q_weights
|     /
v    v
gemmv (int8 compute)

To:

fp32   q_weights
|     /
v    v
gemmv  (fp32 compute)

Right?

Getting rid of one kernel dispatch in decode is usually good idea. The same applies to the fusion you are proposing. Usually the activation quantization kernels hit caches since activations are fairly small. I am wondering what roofline you are seeing for quant+gemmv prior to these changes. Are they at their peak and this is the only way to make it faster, or are the kernels poorly implemented/optimized?

[jimw567]

Suggestions for Improvement. non blocking

• mmvdq.cuh has no include guard / #pragma once. It's included by two TUs and only declares functions, so it's fine today, but a guard is cheap insurance
  and matches convention.
• dq_num_rows() re-reads the env on every dispatch, while ggml_cuda_dq_mmv_enabled/q6k_enabled cache via static const. Minor, but caching it too would be
  consistent and avoids a getenv per matvec.
• DQ_DISPATCH_ROWS default is 2 for any unrecognized GGML_CUDA_DQ_ROWS value (e.g. a typo like =3 silently runs <2>). Consider clamping/logging unknown
  values rather than silently falling to 2.
• The Q4_K/Q5_K/Q6_K dot helpers are close paraphrases of the vecdotq/convert.cu reference math (get_scale_min_k4, dall/dmin, the qh bit-plane merges).
  Since they now live in a second place, a one-line comment cross-referencing the canonical source (you already did this for dq_get_scale_min_k4) on the
  Q5_K/Q6_K bit-manipulation blocks would help future maintainers keep them in sync.

Overall this is a clean, well-organized kernel addition with a conservative default (RDNA3.5-only, env-overridable) and a solid perf story. The two
functional items to address before merge are the fused-path Q6_K gate (easy, correctness-of-intent) and either fixing or documenting the multi-row OOB
read (default-safe, but a real footgun on the tuning knob).