cuda : add experimental bounded MoE host pinning

[GenerelSchwerz]

Overview

Draft for testing bounded CUDA host pinning for the MoE expert cache. This is a capacity feature, not a claim of full-pinning throughput parity or certified Windows support.

• Add --moe-expert-cache-host-pinned-mb as a model-wide cap on cache-owned source registrations and pinned staging/control storage. Omitting it retains the existing default path. It supports mmap and non-mmap backing without implicit CPU expert placement.
• Keep one reusable staging batch per group and a fixed registered source prefix. Size the batch using the model's maximum selected-expert count, capped at 16 and reduced further by the budget. The existing reservation API keeps its 16-miss behavior.
• Retain optional x86-64 streaming stores with bounded source read-ahead and an optional single CPU copy helper per model owner. Both build options default OFF. The helper uses serial fallback when busy or unavailable and is disabled with multiple visible CUDA devices.
• Exclude the overlap pipeline, shifted pin-window control and temporary profiling instrumentation. The pipeline regressed Flash; those experiments were archived separately, not included for users to enable.

The cap is not a total-RAM or VRAM limit. Insufficient staging fails explicitly; a captured callback failure stops execution before incomplete slots can be consumed. No affinity policy, predictive expert prefetch or PLE change is included.

Trying it

Build this PR branch using the repository's CUDA build instructions. To try the optional host-copy optimizations, add both CMake options:

-DGGML_CUDA_MOE_STREAMING_COPY=ON
-DGGML_CUDA_MOE_PARALLEL_COPY=ON

Run test-moe-cache --host-pinning-only, then compare the same model, GPU expert slots, prompt, context and KV settings with the host-pin budget omitted versus explicitly configured. Use -fit off -lv 4 and choose a budget that leaves room for staging and other allocations. For a staging-only check, set GGML_CUDA_MOE_HOST_REGISTER=0, then remove it before testing source registration again.

Native Windows testers: report Windows/driver/toolkit versions, exact build options and command, allocation logs, RAM/VRAM measurements, repeated-request timings and whether outputs match. A successful model load alone is not an inference pass. There is no assumed universal Windows pinning limit. CUDA 12.8 is the Linux-tested target; use a toolkit and architecture appropriate for your GPU.

Validation and limitations

The cleaned candidate passed the full MoE suite with both copy optimizations ON and with both OFF. Optimized-build checks also passed LRU, forced staging, CUDA memcheck, the expected captured failure path, argument parsing and 880/880 CUDA MUL_MAT_ID cases. Earlier feature validation covered mmap/non-mmap servers and FLASH_ATTN_EXT. The retained CPU worker matches the exact extraction previously checked with ThreadSanitizer and MSVC/Wine; these are not native Windows CUDA tests.

The final cleaned-build Flash server completed four 1024-token requests and a different 512-token request, with all five output hashes matching the reference and no job swap, memory-limit/OOM events or watchdog stops. Warm median generation was 34.97 tokens/s. This is a post-cleanup correctness check, not another controlled speed comparison.

Model-sized staging reduced Flash copy traffic by 2.8% under the same pin cap, but its measured throughput gain was only 0.5%, below run variation. The CPU helper reduced callback body time in a matched profile, but the whole speed gain cannot be attributed solely to it. A fresh same-build partial/full comparison produced server medians of 39.61/35.78 versus 42.57/45.99 tokens/s, with all corresponding outputs matching. File-backed page activity varied, and one full-pinning load triggered memory-limit reclaim without OOM or job swap. These results do not promise steady 40/47 tokens/s rates.

See implementation notes and measurements for usage, accounting, failure behavior and evidence limitations. Still draft pending native Windows, physical multi-GPU and CUDA 11 runtime validation.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: YES. Codex assisted with implementation, tests, profiling, documentation and PR updates under repository-owner direction. The contributor remains responsible for understanding, reviewing and maintaining the changes.

[luisantich]

I have been testing bounded MoE host pinning on native Windows with an RTX 5080 and can provide native Windows and physical mixed-GPU validation results.

System:

• Windows
• Ryzen 9 9900X3D
• 32 GB DDR5
• RTX 5080 16 GB
• RX 6800 16 GB
• CUDA 13.3
• Vulkan SDK 1.4.357.0
• MSVC 19.51.36256
• Qwen3.8-Flash-Next UD-IQ3_XXS (~76.3 GiB GGUF)
• --moe-expert-cache-size 120
• --moe-expert-cache-host-pinned-mb 4096
• --load-mode mmap
• -fit off

On the clean #76 branch (1287e2af4), the native Windows static build of:

test-moe-cache.exe --host-pinning-only

passes.

I also tested real CUDA inference with both bounded-pinning modes.

With:

GGML_CUDA_MOE_HOST_REGISTER=0

(staging-only), inference completed successfully with source registration disabled. Staging was approximately 906 MiB.

With normal source registration enabled, the main-model host pin accounting reached:

source=3254255616
staging=904003584
peak_reserved=4158259200
cap=4294967296

so the peak remained within the configured 4 GiB cap, and inference completed without a Compute error.

I then tested a physical mixed-vendor configuration:

--device CUDA0,Vulkan2 --tensor-split 1,1 -sm layer

where CUDA0 is the RTX 5080 and Vulkan2 is the RX 6800.

There is one important interaction: #76 alone exposed the layer-split ROUTE failure addressed by #75. After cherry-picking #75, the unsupported grouped route correctly falls back to cached mul_mat_id and mixed CUDA + Vulkan inference completes.

For the final clean validation I rebuilt the combined #76 + #75 branch (7d3e8efad) without the Qwen MTP compatibility patch, ran one 1024-token warmup request, and then three repeated 1024-token requests.

Warm generation results:

• 12.15 t/s
• 11.90 t/s
• 12.17 t/s
• median: 12.15 t/s

The corresponding MoE cache hit rates were:

• 97.41%
• 97.42%
• 97.42%

All requests completed successfully.

Observed memory use during generation was approximately:

• RTX 5080: 14.9 GB
• RX 6800: 3.1 GB
• system RAM: 30.7 GB

I also separately tested Qwen3.8 MTP drafting on the combined branch. It was stable, including the same cached mul_mat_id fallback in the target/draft paths. With my current MTP configuration (n_max=1, ~70.2% draft acceptance), it did not improve performance, so I consider that a separate configuration/performance observation rather than an issue with bounded host pinning.

So far I have not seen host-pin budget overruns or Compute errors caused by the bounded-pinning path on native Windows.

The 4096 MiB cap is quite tight for this model/configuration: the measured main-model peak was 4,158,259,200 bytes versus the 4,294,967,296-byte cap.