Sharing a performance finding from profiling the offloaded-MoE path (--n-cpu-moe / -ncmoe) — the setup where a big MoE model runs on a small GPU with the experts kept in system RAM and streamed in per layer.
My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04. Model: Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048. Everything below is single-box and gen4-specific, so numbers will move on other hardware.
On this setup prefill is bottlenecked by PCIe transfers, not GPU compute. Profiling with Nsight, the GPU sits idle a large fraction of each prefill pass waiting on expert-weight H2D copies, and those copies run serially with the matmul that consumes them — layer N's weights finish uploading, then layer N computes, then N+1's weights start uploading. The upload and the compute don't overlap on the offload path.
Two changes helped a lot, both prefill-only and both producing token-identical output at temp 0:
- Page-locking the mmap-backed CPU expert weights so the H2D transfers DMA directly instead of going through a pageable bounce buffer. ~+21% on its own.
- Overlapping the expert uploads with compute on a second stream with a few staging slots, so layer N+1's weights upload while layer N is still computing. This is where most of the win is.
Together that took prefill from ~1143 to ~1880 tok/s at identical settings — roughly +64%, pure code, no config change. GPU idle during prefill dropped from ~42% to a few percent.
I prototyped both behind env flags on a fork so I could A/B them on the same binary and confirm the output stayed identical: https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts (there's a benchmark writeup in the branch with the exact commands and the full before/after log). I'm raising this as a finding rather than a PR — partly because AGENTS.md asks for that, and partly because the right way to expose something like this upstream is probably whoever owns the CUDA scheduler deciding on the shape, not me dropping a patch on you.
Two honest caveats: this was measured a few weeks back at around the b6xxx era, and the CUDA backend has moved since, so the exact idle percentage may be stale — I haven't re-run against current master. And I know pipeline copies are already a thing for the multi-device case; the gap I'm describing is specifically the single-GPU offload path, which always gets n_copies = 1.
Main question for maintainers: is reducing GPU idle on the offload path something you'd want addressed, and if so, is overlap-on-a-second-stream the right approach — or is there a reason the offload copies are serial today that I'm not seeing? Happy to share nsys timelines or run other configs if that's useful.
here is nsys trace
Sharing a performance finding from profiling the offloaded-MoE path (
--n-cpu-moe/-ncmoe) — the setup where a big MoE model runs on a small GPU with the experts kept in system RAM and streamed in per layer.My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04. Model: Qwen3.6-35B-A3B (Q4_K_M),
-ncmoe 26, prefill at pp2048 / ub2048. Everything below is single-box and gen4-specific, so numbers will move on other hardware.On this setup prefill is bottlenecked by PCIe transfers, not GPU compute. Profiling with Nsight, the GPU sits idle a large fraction of each prefill pass waiting on expert-weight H2D copies, and those copies run serially with the matmul that consumes them — layer N's weights finish uploading, then layer N computes, then N+1's weights start uploading. The upload and the compute don't overlap on the offload path.
Two changes helped a lot, both prefill-only and both producing token-identical output at temp 0:
Together that took prefill from ~1143 to ~1880 tok/s at identical settings — roughly +64%, pure code, no config change. GPU idle during prefill dropped from ~42% to a few percent.
I prototyped both behind env flags on a fork so I could A/B them on the same binary and confirm the output stayed identical: https://github.com/thecodacus/llama.cpp/tree/fable5/prefetch-experts (there's a benchmark writeup in the branch with the exact commands and the full before/after log). I'm raising this as a finding rather than a PR — partly because AGENTS.md asks for that, and partly because the right way to expose something like this upstream is probably whoever owns the CUDA scheduler deciding on the shape, not me dropping a patch on you.
Two honest caveats: this was measured a few weeks back at around the b6xxx era, and the CUDA backend has moved since, so the exact idle percentage may be stale — I haven't re-run against current master. And I know pipeline copies are already a thing for the multi-device case; the gap I'm describing is specifically the single-GPU offload path, which always gets
n_copies = 1.Main question for maintainers: is reducing GPU idle on the offload path something you'd want addressed, and if so, is overlap-on-a-second-stream the right approach — or is there a reason the offload copies are serial today that I'm not seeing? Happy to share nsys timelines or run other configs if that's useful.
here is nsys trace