CUDA: cuBLAS path for lightning_indexer on Volta

[CoREse]

Summary

GGML_OP_LIGHTNING_INDEXER (DeepSeek-V4) computes score[b,kv] = sum_h w[b,h] * relu(q[b,h] . k[kv]) + mask[b,kv]. The CUDA backend has a wmma kernel for Turing and newer and a generic vector kernel for everything else. On V100 the vector kernel reaches ~4.3 TFLOPS, while cuBLAS on the same card does the same multiply-accumulate at ~20 TFLOPS (tensor cores via CUBLAS_COMPUTE_32F_FAST_16F).

This PR adds a cuBLAS path for pre-Turing NVIDIA GPUs: for each chunk of query rows, one GEMM computes C = K^T Q (n_kv x n_head*nb, column-major) and one fused kernel does relu, the per-head weighting, the sum and the mask add. Scratch memory is bounded (128 MB by default) and does not grow with context length. The existing kernels are untouched and still used everywhere else.

The layout of C matters a lot: with the products of one (b, kv) pair contiguous (n_head*nb x n_kv) the reduce kernel has neighbouring lanes gathering across a n_head*nb stride and takes 60-78% of the total time; with n_kv x n_head*nb each head is one contiguous stream and the same GEMM ends up more than twice as fast.

Test

test-backend-ops -o LIGHTNING_INDEXER, V100 (sm_70), CUDA 12.9: 144/144 pass (the op's max_nmse_err is 1e-6).

test-backend-ops perf -o LIGHTNING_INDEXER, hsk=128, nh=64, type_K=f16:

kv	batch	master	this PR
4096	512	4.31 TFLOPS	20.73 TFLOPS
65536	512	4.32 TFLOPS	19.93 TFLOPS
65536	2048	4.32 TFLOPS	19.91 TFLOPS
65536	1	263 us	171 us

End to end, DeepSeek-V4-Flash (UD-Q8_K_XL, 4x V100-16GB, experts on CPU), 54,564-token prompt: the indexer was 24% of GPU time at 23K context; combined with #28887 prompt processing goes 147.1 -> 178.4 t/s.

Quality, llama-perplexity --kl-divergence, wikitext-2, -c 32768 --chunks 2: mean KLD 0.0045, top-1 agreement 97.5%, PPL 3.9701 vs 3.9689 on master (±0.048). Note that a mathematically equivalent variant of this path (plain cublasSgemm, exact f32) gives the same 0.0043: the deviation comes from the changed summation order flipping near-ties in the model's own top-512 routing, not from the reduced-precision accumulate. Running the same binary twice gives KLD 0.000000 / 100%, so this is measured against a deterministic baseline.

Notes

• Enabled only for cc == GGML_CUDA_CC_VOLTA; I only have V100s to test on. Pascal/Maxwell would very likely benefit too (they run the same vector kernel), but I did not want to enable something I cannot measure.
• An f16 GemmEx variant is ~1.8x faster still (36 TFLOPS) but one of the 144 correctness cases lands at 1.015e-6 vs the 1e-6 threshold, so it is not enabled here.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01FBqdxhtLYCXWCqbGx6LJuh

[ggml-gh-bot]

Hi @CoREse, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[taronaeo]

https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md#prohibited-ai-usage-results-in-immediate-pr-closure