cuda: fail closed grouped decode ROUTE to cached mmid

[Lathly]

Overview

Layer-split + MoE expert cache would load and prefill, then abort on the first decode token with:

moe-cache: grouped decode certificate failed: semantic_group=25 ... tensor=blk.25.ffn_gate_exps.weight graph=route(8)
llama_decode: failed to decode, ret = -3
Compute error.

The grouped planner requires a local argsort to certify the expert-id route. With --split-mode layer, that argsort often lives on one GPU while later layers run on the other, so GPU1 groups fail GROUP_REASON_ROUTE. Prefill already used cached mul_mat_id. Decode treated ROUTE as OUTCOME_ERROR and aborted the whole graph instead of failing closed.

This treats ROUTE as a legacy-safe decode reason (same as materialization / execution / consumer-equivalence) and skips binding those groups so cached mul_mat_id can run.

Reproduced on dual RTX 5070 Ti with Qwen3.8-Flash-Next UD-IQ3_XXS, --moe-expert-cache-size 80, --split-mode layer. After the change, decode returns tokens. test-moe-cache passes.

Additional information

The README already says unsupported grouped-decode layouts fail closed to cached mul_mat_id. This closes that hole for layer-split.

This is not tensor-split. Tensor-split of cached expert weights is still unsupported.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Grok 4.6 (DeepSeek Harness) reproduced the crash, identified the missing fail-closed reason, wrote the patch, and opened this PR. I reviewed the diagnosis and the change.

[GenerelSchwerz]

Thanks for reproducing this and keeping the fix narrow. I reviewed the control flow and tested the PR head on a single RTX 5070 Ti: it builds, test-moe-cache --registry-only, --grouped-decode-only, and the full test-moe-cache suite all pass.

The fallback looks correct: ROUTE is a failure of the stricter grouped-route certificate, while DECODE_LEGACY leaves the cache-aware mul_mat_id path available. Skipping the grouped binding therefore avoids the abort without dropping the expert cache.

I cannot independently reproduce the layer-split topology with one GPU. Could you add the exact before/after command and a short log showing both CUDA devices/layer placement, the original GROUP_REASON_ROUTE abort, and successful token generation with this patch? That would close the remaining hardware-validation gap.

I also added a normal-level fallback notice directly to moe-cache in 6f1254e and scoped it per model generation/device in 6dd4c62. Once this PR lands, the route fallback will be visible without --experimental-logs, while repeated tokens stay quiet. Git reports the PR still merges cleanly with those base changes.

[GenerelSchwerz]

^^ AI btw. Thank you for contributing to my fork! I appreciate it.

[luisantich]

I was able to reproduce and validate this fix on a physical mixed-vendor Windows setup.

Environment:

• Windows
• Ryzen 9 9900X3D
• 32 GB DDR5
• RTX 5080 16 GB via CUDA 13.3 (CUDA0)
• RX 6800 16 GB via Vulkan (Vulkan2)
• MSVC 19.51.36256
• Qwen3.8-Flash-Next UD-IQ3_XXS
• layer split

Runtime command used for both before/after tests:

llama-server.exe ^
-m "C:\llama-moe-cache-drafting\build-cuda\bin\Release\models\Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf" ^
--host 127.0.0.1 ^
--port 8080 ^
--device CUDA0,Vulkan2 ^
-ngl 999 ^
--tensor-split 1,1 ^
-sm layer ^
--moe-expert-cache-size 120 ^
--moe-expert-cache-host-pinned-mb 4096 ^
--load-mode mmap ^
-fit off ^
-c 4000 ^
-np 1 ^
--flash-attn on ^
--jinja ^
-lv 4

With PR #76 alone (1287e2af4), model load/prefill succeeded, but the first inference request failed with the same ROUTE case described here:

moe-cache: grouped decode certificate failed: semantic_group=25 layout=1 domain=1 tensor=blk.25.ffn_gate_exps.weight graph=route(8)
graph_compute: ggml_backend_sched_graph_compute_async failed with error -1
llama_decode: failed to decode, ret = -3
srv decode: Compute error. off = 0, n_batch = 2048, ret = -3

After cherry-picking this PR (cafbebe094e722e6c6bf78ee211b186e7c6a927e) on top of #76, the same configuration instead reports:

moe-cache: device 0 grouped decode unavailable for 1 group(s); using cached mul_mat_id fallback

and inference completes successfully.

I also rebuilt the combined branch cleanly and ran one 1024-token warmup plus three repeated 1024-token requests. All completed without decode/compute errors.

Warm generation results:

• 12.15 t/s
• 11.90 t/s
• 12.17 t/s
• median: 12.15 t/s

MoE cache hit rates for those three warm requests were 97.41%, 97.42%, and 97.42%.

Observed memory use during generation was approximately:

• RTX 5080: 14.9 GB
• RX 6800: 3.1 GB
• system RAM: 30.7 GB

The host-pin accounting at the fallback point was:

cap=4294967296 source=3254255616 staging=904003584 pending=0 peak_reserved=4158259200 bytes

So this closes the hardware-validation gap for the same layer-split ROUTE failure on a real CUDA + Vulkan two-GPU system as well.

[GenerelSchwerz]

Merged as a compatibility fix for the layer-split ROUTE failure. When the grouped route cannot be certified, this keeps inference working through the existing cache-aware legacy mul_mat_id path. The GPU expert cache remains active, but the affected graph gives up grouped-decode execution.

Grouped-decode fast-path support for layer-split multi-GPU is planned as follow-up work. That needs route validation across device transfers with correct synchronization and lifetime handling; this fallback remains useful for layouts that cannot use the fast path. No timeline is committed yet, and tensor-split expert weights are a separate scope.

Thanks for the reproduction and the additional Windows CUDA + Vulkan validation with #76.

AI disclosure: Codex assisted with review, merge and this comment under repository-owner direction.