cuda/moe-cache: pinned-alloc fallback puts experts on the CPU backend

[LokenSI]

Thanks for the moe-cache work, and for writing up Notable Runs the way you did — exact commit, exact request, output hashes, and an honest list of reasons someone else's machine might differ. That is what made it possible to find this rather than just conclude my box was slow.

The fallback does not do what its comment says

In ggml_backend_cuda_moe_cached_buffer_type_alloc_buffer, current tip:

if (ptr == nullptr) {
    // Pinned alloc failed -- fall back to a regular CPU buffer. This costs
    // PCIe bandwidth on cache miss but keeps the model loadable.
    return ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size);
}

The intent in that comment is a slower path: experts still executing on the GPU, paying extra PCIe traffic on a miss. But returning ggml_backend_cpu_buffer_type() changes the buffer's backend, so ggml's scheduler assigns those ops to the CPU. The experts stop running on the GPU entirely and the expert cache never engages.

So the actual cost is not the documented one. Measured here: 17-20 tok/s instead of 47, moe-cache-experts: tensors=0, GPU ~19% busy, ~7 GB VRAM instead of ~14 GB.

It is also quiet. Every symptom reads as host overhead, and the line that explains it is at GGML_LOG_DEBUG — below the -lv 3 your guide recommends, so at the recommended verbosity there is nothing to see.

The fix

On pinned-alloc failure, allocate pageable host memory but keep the buffer under this buffer type (buffer->buft = buft). The CUDA dispatch hook keeps ownership of mul_mat_id, so the experts stay on the cache path and only the H2D copies become pageable — which is what the comment intended. The alloc-failure log is promoted DEBUG -> WARN.

This part is independent of any platform. It matters wherever the fallback fires at all.

Two additions you may not want

The branch also carries two opt-in knobs, because on my host the fallback fires for the whole expert source at once and partial pinning is worth a lot:

• GGML_CUDA_MOE_SOURCE_CHUNK_MIB — a get_max_size hook so ggml splits the expert source into several buffers, which is what allows part of it to be pinned when all of it cannot be.
• GGML_CUDA_MOE_PINNED_BUDGET_MIB — pinned-byte accounting against a budget, so the spill is a decision rather than an allocation failure late in the load that would also take the cache's staging buffers down with it.

Both unset preserves current behaviour exactly: one buffer, pin all of it, no budget check. Happy to drop these and keep only the fallback fix if you would rather not carry the env knobs.

Where I hit it

Windows 11 -> WSL2 -> Docker Desktop, same GPU and RAM as your entry. WSL2 caps pinned host memory at roughly 80% of the VM's RAM — 47.0 GiB measured on a 60 GiB VM, same through cudaHostRegister, and it scales with VM size. UD-Q3_K_XL's ~52 GiB source therefore cannot be pinned at any VM size that fits in 64 GB, so the fallback fires every time.

I have only measured WSL2 and am not going to claim which other hosts reach this.

Why it is a throughput issue, not just a loadability one

Grouped decode is all-or-nothing across layers, as you document. One 2 GiB chunk deliberately left pageable takes covered from 48/48 to 0, and decode from 47.6 to 25.9 tok/s on UD-IQ3_XXS.

Results

RTX 5070 Ti 16 GB (also driving the display), i7-14700F, 64 GB DDR5-6000. Your request.json, your measure.py, cold server, first request.

	this box (WSL2)	your entry (CachyOS)
Quant	UD-IQ3_XXS	UD-Q3_K_XL
Decode	47.63 tok/s	47.00 tok/s
Prefill	73.91 tok/s	95.31 tok/s
GPU loaded / peak	14,774 / 15,126 MiB	13,908 / 14,384 MiB
Min available RAM	10,172 MiB	2,290 MiB
Cache slots / threads	96 / 16	80 / 12
Grouped decode	registered=48 covered=48, fallback=0	same

Warm, 20 back-to-back requests with cache_prompt: false: 52.6-53.2 tok/s, identical content hash all 20 times.

On your quant, UD-Q3_K_XL, this box gets 25.1 tok/s with the cache engaged and an output hash matching yours (b6f509be) — same computation, legacy path, because 52 GiB cannot pin under the cap. That is why the quant differs above: a constraint, not a preference.

Full setup, all 173 raw logs and the measurement table are under wsl2/ on my fork.

[GenerelSchwerz]

The pageable-fallback fix and the warning look worth integrating. Your WSL2 results are especially useful: you have a working configuration that exercises the pinning limit and retained evidence, so your machine is the best candidate we currently have for checking the revised version against the working branch.

Could you take the split you offered and narrow this PR to preserving the cached buffer type on pageable fallback plus the allocation-failure warning, leaving the two environment knobs on your experimental branch?

The reasons for separating the knobs, reviewing 667d0aa58:

• The source-budget check reads the counter before allocation and increments it afterward. Concurrent allocations can both pass and exceed the budget. Retaining this would need an atomic reservation with rollback on allocation failure.
• More seriously, source chunking does not safely compose with updated cuda : add experimental bounded MoE host pinning #76. Its bounded buffer type inherits the new get_max_size hook, so non-mmap allocations can return a multi-buffer wrapper. The loader then passes that wrapper to ggml_backend_cuda_moe_pin_sources, which casts its context to moe_bounded_buffer. The wrapper actually holds ggml_backend_multi_buffer_context, so the registered access reads beyond that context. A clean textual merge does not catch this. Supporting chunking there would require handling the child buffers explicitly or preventing the bounded type from inheriting that hook.
• Both environment parsers use atoll followed by an unchecked MiB shift; malformed and overflowing values need validation if those knobs are retained.

#76 already provides the model-wide source/staging budget. The narrower fallback fix still has value for the default allocation path when no explicit budget is supplied. We would also like a focused case in the existing MoE tests that forces pinning failure/pageable fallback and checks cached CUDA execution, output correctness and cleanup. Repeating your matched WSL2 run on that revision would give us much stronger integration evidence than extrapolating from another machine. Your existing results support the current branch; the extracted revision and its composition with #76 still need checking.

If you would rather keep this branch intact or do not have time to make the split, that is fine. We can extract the fallback and warning changes into a separate integration branch, retain your authorship/credit, add the focused test, and validate against current moe-cache and #76 before merging. We would link that work here and still welcome a WSL2 comparison from you when convenient. The optional knobs can remain available on your fork for the configuration you have already validated.

AI disclosure: Codex assisted with this review and comment under repository-owner direction. Findings are from source and integration review; no new WSL2 runtime validation was performed here.