Conversation
added 7 commits
September 10, 2026 00:39
…tests The gated_delta_net -> cpy fusion itself already exists on CUDA (upstream ggml-org#23940, 5a460de); what was missing vs the PR #112 arm spec is a fusion-specific toggle and the test coverage that landed upstream only with the later Metal-side fusion commit (d011a21), which is not part of this baseline. - GGML_CUDA_FUSE_GDN_CACHE=0 disables only the gdn cache-cpy fusion (default on), so one binary A/Bs fused vs unfused without GGML_CUDA_DISABLE_FUSION killing every other CUDA fusion. - Port test_gated_delta_net_cache_fusion into test-backend-ops (5 cases): the whole-graph CUDA run takes the fused path and the cache view (the elided cpy's dst) is compared against the CPU reference.
…ndmine E (UM + imbalanced -ts collapse)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the codeable delta of the PR #112 arm spec on a fresh branch off
fork/pr103-gdn-fusion-arm.Important correction to the #112 premise
The GDN -> cache-cpy fusion already exists on CUDA in this baseline: upstream ggml-org#23940 (5a460de, "Remove redundant CUDA copies after gated_delta_net") added
ggml_cuda_try_gdn_cache_fusion(ggml-cuda.cu) + the fused kernel path, months before the Metal-sideGGML_METAL_FUSE_GDN_CACHEcommit (d011a21) that #112 was modeled on. #112's claim that "CUDA still runs the unfused GDN + cpy" was wrong — the Metal commit is not even in this baseline's history. What was actually missing, and what this PR adds:GGML_CUDA_FUSE_GDN_CACHEtoggle (ggml-cuda.cu, inggml_cuda_try_gdn_cache_fusion):=0disables only this fusion, default on. The pre-existingGGML_CUDA_DISABLE_FUSIONkills every CUDA fusion, which would contaminate the PR103.0 A/B (moe weighted reduction, topk-moe, etc. would also turn off). One binary now serves fused and unfused arms.test_gated_delta_net_cache_fusionported intotests/test-backend-ops.cpp(5 cases) — this test landed upstream only with the Metal commit, outside this baseline. Whole-graph CUDA run takes the fused path; the elided cpy's dst (cache view) is compared against the CPU reference.Kernel and fusion matcher are untouched — no new kernels, no graph changes, no ggml-core changes, per #112 non-goals.
Validation (RTX 3060 sm_86, CUDA 13, local rig — 5060 Ti was fully occupied)
GATED_DELTA_NET_CACHE_FUSIONcases OK; debug log confirms the fused path fires per case (fused gated_delta_net snapshot copies ... skipped 3 nodes)GGML_CUDA_FUSE_GDN_CACHE=0: 5/5 OK via the unfused path, zero fusion hits — toggle verifiedGATED_DELTA_NETcases OKcmp'd): fused cache buffer == unfused cache buffer == unfused gdn tail, bit-for-bit; attention output identical across runs. The fused run's gdn-output tail region is intentionally unwritten (kernel writes state straight to the cache; the matcher guarantees the only tail consumer — the elided cpy — and that the gdn output is not a graph output), matching the Metal-side design.Remaining for the arm (rig, later)
Test plan
AI usage disclosure: YES — implemented by AI (opencode) as a handoff task; the human reviewer is expected to verify the toggle placement and test port against ggml-cuda.cu:2759 and the upstream ggml-org#23940/#d011a214b commits. All work stays on the ddvnguyen fork; nothing pushed to upstream origin.