CUDA: gated_delta_net - share per-token k/q via shared memory for long prefills#54
Draft
roberteg16 wants to merge 1 commit into
Draft
CUDA: gated_delta_net - share per-token k/q via shared memory for long prefills#54roberteg16 wants to merge 1 commit into
roberteg16 wants to merge 1 commit into
Conversation
RDNA3.5-only optimisation. The serial GATED_DELTA_NET kernel reloaded the per-token k/q from global once per warp, even though k/q are identical for every state column in a block. On RDNA3.5 (gfx115x) stage them in shared once per token (one block-wide load) for long, memory-bound sequences. Gated at compile time (#ifdef RDNA3_5) and at runtime (GGML_CUDA_CC_IS_RDNA3_5(cc) && n_tokens >= 1536) so no other arch pays the extra LDS or the barrier. Short sequences stay latency-bound (hidden by inter-warp overlap) and use the direct global load. Measured on gfx1151 (d=128, 32 heads, median of 3): ~1.05x at 1536, ~1.20x at 2048, ~1.60x at 4096; neutral below the threshold. Also removes the super-linear slowdown of the serial scan at long sequences.
roberteg16
force-pushed
the
rogarcia.gdn-shared-kq
branch
from
July 16, 2026 13:51
e332179 to
5bb1972
Compare
Author
|
This seems to be a wash. I'll keep it open until it is 100% clear. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The serial
GATED_DELTA_NETkernel reloaded the per-tokenk/qfrom global memory once per warp, even thoughk/qare identical for every state column in a block. This stages them in shared memory once per token (a single block-wide load) when the kernel is memory-bound.n_tokens >= 1536(measured crossover on gfx1151, d=128). Short sequences stay latency-bound and are hidden by inter-warp overlap, which the extra__syncthreadswould break, so they keep the direct global load.Measured (gfx1151, d=128, 32 heads,
test-backend-ops perf, median of 3)Test plan
test-backend-ops -o GATED_DELTA_NET— 36/36 pass (CPU reference parity; scalar + KDA + K>1 + decode paths).