Conversation
Contributor
CI recapDashboard: View test results in Grafana
|
Member
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On AMD Strix Halo (Ryzen AI Max,
gfx1151, RDNA 3.5) theflaandcausal_conv1dfast paths have no ROCm build, so Qwen3.5-family models (Qwen3.5, Qwen3.6-27B, Qwen3.8-27B) fall back to the slow pure-torch Gated DeltaNet (is_fast_path_available=False). This adds arocmdevice entry, gated to that GPU, to the existingQwen3_5GatedDeltaNetmapping from #46423, pointing at the ROCm build of the same Hub kernel (Atlas-Inference/gdn). Strix Halo users get the fast path automatically; every other GPU, including other ROCm GPUs, keeps its existing path.Device(type="rocm", properties=ROCMProperties(min_capability=115, max_capability=115))entry next to the GB10 one inintegrations/hub_kernels.py, same layer name and repo, revision pinned.docs/source/en/model_doc/qwen3_5.md.The kernel is the same stateless
kernelslayer as the GB10 build. it reuses the host module's projections, gated RMSNorm andout_proj, and replaces only the conv1d, q/k L2-norm and delta-rule cores. On gfx1151 it runs the kernels the Atlas engine serves Qwen3.6/3.8-27B with on Strix Halo.causal_conv1d_update_prefill→l2_norm_bf16→ FLA-chunked delta rule (recompute_wu→chunk_delta_h→chunk_fwd_o)causal_conv1d_update_l2norm_f32→gated_delta_rule_decode_f32What does this PR do?
The gated
rocmentry inintegrations/hub_kernels.pyplus the docs note.@use_kernel_forward_from_hub("Qwen3_5GatedDeltaNet")decorators from #46423 already cover the dense and MoE classes.Verified on Strix Halo (gfx1151,
torch.cuda.get_device_capability() == (11, 5)), bf16, torch 2.14.0 ROCm 7.2 and ROCm 7.14 wheels, transformersmain+kernels0.17:kernelize()swaps the forward of all 24/24 GDN layers ofQwen/Qwen3.5-9B, and the profiler shows the kernel ops running.main+ this PR,from_pretrained(..., use_kernels=True)with an empty kernel cache pullsAtlas-Inference/gdnat the pinned revision. All 24/24 GDN layers ofQwen/Qwen3.5-9Brun it, and greedygenerate()produces identical token IDsMeasured through the end-user path (
from_pretrained(..., use_kernels=True), kernel pulled from the Hub) onQwen/Qwen3.5-9B(bf16, gfx1151, torch 2.14 + ROCm 7.2, 1024-token prompt, greedy decode of 256 tokens; mean of two runs):use_kernelsFalse(PyTorch fallback)True(Atlas-Inference/gdn)About 74% of this prefill is the dense projection GEMMs, which the kernel doesn't touch. Per Gated DeltaNet layer of
Qwen/Qwen3.8-27B(real weights), the kernel is 1.65x faster on a 1024-token prefill and 1.26x faster per decode step. The published build targets torch 2.14 with ROCm 7.2.Follow-up to #46423 (the GB10/SM121 entry). The ROCm build needs kernel-builder with gfx1151 support, which landed in huggingface/kernels#839 (tracked in huggingface/kernels#834). No overlapping PR exists.
The same trust note applies as for the GB10 entry: the mapped kernel needs
trust_remote_code=TrueuntilAtlas-Inferenceis on thekernelstrusted-publisher allowlist, would love to enable this for continued future contributions without any blocks!Code Agent Policy
Before submitting
Who can review?
@vasqu original PR merger, @danieldk first issue
Potentially @drbh (kernels integration) and @Cyrilvallez (text models / Qwen3.5) as well :)
If you could enable myself on the
kernelstrusted-publisher allowlist that would help for the future!