cuda : MoE look-ahead expert prefetch (PR-A consumer) + LFRU eviction guard

[ddvnguyen]

What

PR-A of the MoE look-ahead expert prefetch: the consumer half plus the --moe-lookahead
gate, on top of the design doc. The producer (GGML_OP_MOE_PREFETCH / llama-graph.cpp,
"PR-P1" in the doc) is out of scope here.

Three commits over feat/763-reconcile-qwen4exp-mtp:

• 137fa17bb docs: look-ahead design doc (folds measured evidence from upstream RFC RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution ggml-org/llama.cpp#24528)
• cbb19d802 feat(moe-cache): the consumer behind --moe-lookahead
• ec23a599f fix(moe-cache): the LFRU eviction guard (see below)

Consumer

ggml_backend_cuda_moe_prefetch_experts() (plus a tensor variant) asks the CUDA MoE cache to
prefetch the next layer's expert rows during the current layer's compute. Gate: --moe-lookahead N
(env LLAMA_ARG_MOE_LOOKAHEAD), default 0.

Invariants kept: is_prefetch=true, wait_for_compute=false, pin=false; no stream synchronize
and no compute-stream wait on the prefetch path; pinned slots are never chosen; expert ids outside
[0, n_experts) are ignored; H2D copies are batched through cudaMemcpyBatchAsync (CUDART >=
12.80) instead of one copy per expert id; a target whose buffer is not MoE-cached is skipped with a
one-shot warning; the exported ggml_backend_cuda_moe_prefetch_experts signature is unchanged;
width 0 leaves the decode step untouched.

Eviction guard (the review finding)

The first consumer revision picked its victim by plain LRU on the prefetch path, so a prediction
could displace an expert that demand had already proven hot. Only the pin guard (never evict a slot
a running GEMM is reading) existed, on both the demand and speculative paths.

ec23a599f ports colibri's PILOT_EVICT_GUARD: a speculative fill may displace a resident only
when that resident is not genuinely warm - protected when it has at least 2 demand accesses AND
is clearly hotter than the prediction, by the 25% + 4-frequency hysteresis in LFRU score units
(score ported from colibri c/tier.h:40-43: frequency in the high bits, recency saturating in the
low byte). A blocked prediction is dropped, never forced in. Heat counts demand accesses only;
counting predictions would let a speculation inflate the score that decides whether it may displace
a resident (the failure colibri hit in ggml-org#490).

The eviction core is now shared (select_victim_locked / install_fill_locked) so the demand and
speculative paths cannot drift; the prefetch path's copy batching is unchanged.

Verification (RTX 5060 Ti CUDA0 + RTX 3060 CUDA1, CUDA 13.2.1)

• tests/test-moe-cache: 19/19 cases pass, test-moe-cache: OK, exit 0 (was 18 before the new case).
• New case lookahead prefetch eviction guard: 3-slot pool. A warm resident (3 demand accesses)
  survives a prediction for a never-seen expert - dropped, no copy, no miss, no eviction, still a
  demand hit. A once-demanded resident is evicted for the prediction.
• --lookahead-prefetch-only: lookahead prefetch legacy layer OK + the guard case, exit 0.
• CLI: --help lists --moe-lookahead N; default and --moe-lookahead 0 load stock;
  --moe-lookahead 8 --moe-expert-cache-size 84 loads; --moe-lookahead 8 with cache-size 0 fails
  loud at load (--moe-lookahead requires --moe-expert-cache-size > 0).
• Build note: /opt/software/cuda/13.2 does not exist on this box (only 12.9, 13.2.1, 13.2.2,
  13.3.1), so the build used -DCUDAToolkit_ROOT=/opt/software/cuda/13.2.1.

Base

This branch sits on the gs qwen4exp-mtp lineage, so it targets feat/763-reconcile-qwen4exp-mtp
(PR #120) - the same stacking PR #123 uses. baseline-flash-next is currently cdd11021b, a parent
of that lineage, so targeting it directly would re-show PR #120's whole diff. GitHub retargets this
to baseline-flash-next automatically once #120 merges.

Open for PR-P1 (producer), not addressed here

1. prefetch_legacy_layer calls acquire_legacy_cache(experts) with compute_stream=nullptr; if
   layer L+1's pool is not installed yet the lease is empty and the prefetch silently no-ops. The
   producer must guarantee the target pool is installed first (spec Q1). Preferred fix: install all
   MoE-cached pools at model load when --moe-lookahead > 0 and fail at load if the budget does not
   fit, so the no-op is unreachable rather than silent.
2. A failed cudaMemcpyBatchAsync clears the new booking but does not restore the resident it
   displaced; the miss falls to the demand path. Safe now that the guard keeps the victim cold.
3. phase_prefetch_dropped is exposed only through the test accessor; the runtime telemetry line
   does not report it yet.