MoE look-ahead PR-P1: producer (NOT LANDABLE, see body) by ddvnguyen · Pull Request #127 · ddvnguyen/llama.cpp

ddvnguyen · 2026-09-13T20:17:10Z

PR-P1: look-ahead prefetch producer (NOT LANDABLE)

Base is feat/763-reconcile-qwen4exp-mtp (PR #120 head), the same deviation PR #126
took for the llama.cpp-side consumer, as accepted by the reviewer.

This branch contains the complete producer implementation plus the measured blockers.
It must not merge in this state. The three blockers below are all reproducible,
deterministic and measured on the rig; the first one is a correctness blocker.

What is implemented

GGML_OP_MOE_PREFETCH end to end: enum, constructor, op name, symbol, CPU dispatch
as a documented no-op, ggml_get_n_tasks, RPC op list, backend-ops skip list.
The graph-level producer: build_moe_lookahead() picks layer il+1's experts from
layer il's post-attention state (build_lora_mm on ffn_gate_inp, then
ggml_argsort_top_k) and ends in ggml_moe_prefetch(), gated on --moe-lookahead > 0,
ctx_type == DEFAULT and decode rows only.
Call sites in src/models/qwen4exp.cpp and src/models/qwen35moe.cpp.
Reserve-time pool preinstall (preinstall_legacy_pools()), the -1 "snapshot not
published" contract, and a bounded self-healing retry in the consumer.
A debug-only recall instrument behind GGML_CUDA_MOE_LOOKAHEAD_DEBUG.
The producer must use ffn_up_exps, not ffn_gate_up_exps: ffn_gate_up_exps is
null for every layer of this model. The consumer resolves the layer by name, so any
of the layer's cached expert tensors works.

Blocker 1: the producer changes the model output

Logits are not preserved. Same prompt, same flags, only --moe-lookahead differs:

configuration	PPL	all-logits md5
look-ahead 0 (baseline)	3.0613	`fc1da5b9e41d249e7da51791a934d561`
look-ahead 8, full producer	2.9434	`8e0a9d10e9095ce7806f645e7edbba0e`
look-ahead 0, CUDA graphs disabled	3.0613	`fc1da5b9e41d249e7da51791a934d561`
look-ahead 8, prefetch op removed, prediction only	2.9434	`8e0a9d10e9095ce7806f645e7edbba0e`
look-ahead 8, prediction reads a trunk tensor, no extra chain	3.1849	`4ba9f8e7a60ca9830b0dcbed073ffc3e`
look-ahead 8, gate MUL_MAT only, no argsort, no op	3.1377	`9bbf44eb3b98b5503ed2d1f9b028dda3`
look-ahead 8, neutral COPY node only	3.0613	`fc1da5b9e41d249e7da51791a934d561`

Readings, all deterministic across reruns:

The extra MUL_MAT that reads ffn_gate_inp is the trigger. Removing the argsort and
the prefetch op does not remove the divergence.
A neutral extra node (a COPY of an existing tensor marked as an output) leaves the
logits byte-identical, so the scheduler is not merely shape-sensitive; the platform is
shape-neutral for a node that does not touch the router.
Disabling CUDA graphs on the baseline reproduces the baseline exactly, so losing graph
capture is not the cause of the numeric change.
A count-based gate is the likely mechanism: the same use_counts proof that rejects
the graph for the prefetch node (graph=unproven(14)) also covers the router tensors,
and an extra reader of ffn_gate_inp changes those counts. This is unconfirmed, and it
is the one open question left.

An earlier handoff recorded these logits as byte-identical. That reading came from a
build in which the producer never ran. It is refuted here.

Blocker 2: every prediction is dropped, so the feature is inert

acquire_legacy_cache() installs a new record only while the target group's authority is
GGML_CUDA_MOE_GROUP_AUTHORITY_LEGACY with admission open. On this rig the groups are
not under legacy authority, so the pool install fails for every target:

E moe-cache: look-ahead could not install 144 of 144 MoE expert cache pools;
             the target layers are not under legacy cache authority, so look-ahead prefetch is inert
W moe-cache: look-ahead prefetch found no installed pool for the target layer, prediction dropped

The producer runs and the consumer resolves the right layer, but no copy is ever
enqueued. This answers the design's open question Q1 negatively: the cross-layer pool
install is not reachable in this configuration.

Blocker 3: -13% decode throughput

arm	decode t/s
look-ahead 0	43.23
look-ahead 8	37.71 (also 37.57, 37.61, 37.68 across runs)

The ids readback synchronizes the stream, which forces use_cuda_graph = false for every
graph containing the op. The precedent is ggml-cuda.cu ggml_cuda_mul_mat_id_needs_sync
at the same decision point, and the demand path's own ids readback has the same cost class.
The feature stays default-off.

Correctness work that is independent of the blockers

ggml.c: the per-node src loop must not count GGML_OP_MOE_PREFETCH sources. The op
reads no tensor values, and counting them made the MoE grouped-decode certificate reject
the graph:
grouped decode certificate failed: ... graph=unproven(14) followed by a failed graph
compute. With the exclusion, zero certificate failures.
preinstall_legacy_pools() returned 0 when the candidate snapshot was not published,
which is indistinguishable from success, so the caller latched "pools ready" forever.
It now returns -1 and the caller only latches on a real result.
The consumer retries the install behind a cooldown, bounded at 8 attempts.

Verification performed

test-moe-cache on the final code: OK, exit 0, including
test_lookahead_prefetch_eviction_guard OK.
All builds are ninja -C build llama-server test-moe-cache llama-perplexity, CUDA arch
86;120.

Recommended decision

Do not merge.
Decide whether the look-ahead producer is worth pursuing at all on this rig. Even a
perfect producer has nothing to page into while the target layers are not under legacy
cache authority (blocker 2), and the readback cost is structural (blocker 3).
If it is pursued, blocker 1 has to be understood first: an extra reader of
ffn_gate_inp must not be able to change the model's output. That is a property of the
fork's router identification or of its count-based proofs, not of the prediction math,
and it needs to be fixed where it lives.

Diagnostics kept on the branch (env-gated, inert when unset)

Added after the reopen-transport ruling commit, at the owner's request that the
instruments stay in the tree for troubleshooting rather than be reverted:

9d24345de - phase attribution probe (GGML_CUDA_MOE_PHASE_PROBE=1) and off-lease
recall scoring (GGML_CUDA_MOE_LOOKAHEAD_DEBUG=1), plus the two cache populations that
were counted but never printed (prefetch_dropped, evicted_prefetched_unused).
570370183 - measured results in docs/moe-lookahead-design.md.

Both probes are inert with the env vars unset (one cached getenv behind a static bool)
and both turn themselves off with a single stderr note if an event or a memcpy cannot be
issued, which is what happens under CUDA graph capture, so they cannot break a capture
run. An attribution needs GGML_CUDA_DISABLE_GRAPHS=1, because replay steps never visit
the host dispatch loop where the events are recorded. This does not change the three
blockers below: the producer is still not landable.

Measured on the RTX 3060, ctx 81920, cache 42, single 13,946-token prompt, 256 decode
steps at 96.90 ms/step (10.43 t/s). Phase rows sum to 95.7 ms, so the split covers the
step:

bucket	ops	ms/step	share
MOE	`MUL_MAT_ID`, `MOE_PREFETCH`, `ARGSORT`, `TOP_K`	67.18	69.3%
DENSE	every other matmul and elementwise op	23.80	24.6%
ATTN	flash-attn, indexer, SSM/GDN, rope, softmax	1.41	1.5%
OTHER	copy/cont/reshape/view/permute/transpose	3.30	3.4%
PLE	`GET_ROWS` on `per_layer_token_embd`	0.00	0.0%

Dispatch state mode_legacy=3 mode_direct=253; prefill (28 ubatches, 4555 ms each) is
87.9% MoE. Recall scored in the dispatch loop instead of the legacy blk.47 lease: 64.5%
of the used experts predicted, 80.6% of the predictions used; the other two populations
are structural zeros because nothing is offered to the installer (decode phase line
ops=0, legacy cache authority).

The bytes/token this implies (GGUF tensor inventory, 10 of 512 experts per layer):
1031 MiB of expert weights, 3.34 GiB of dense/attention/head/shared, 4.35 GiB total, and
the MoE phase streams its 1.01 GiB at 16 GB/s, i.e. 5.4% of the card's bandwidth, while
the dense phase reaches 126 GB/s. The decode step is MoE-kernel-bound, not transport- or
capacity-bound: prefetching ahead layers cannot make the kernels faster, and the expert
H2D (337.6 MiB/token, 3.71 GB/s of the gen4 x4 link's 7.88 GB/s) is already fully hidden
(calls=ready=12240, ready_min=255).

…ok-ahead design

…kahead Wire the consumer half of docs/moe-lookahead-design.md: a new --moe-lookahead N gate (default 0, env LLAMA_ARG_MOE_LOOKAHEAD) that asks the CUDA MoE cache to prefetch the next layer's expert weights during the current layer's compute. The prefetch path mirrors acquire with is_prefetch=true, wait_for_compute=false and pin=false: no stream synchronize, no compute-stream wait, pinned slots are skipped, expert ids outside [0, n_experts) are ignored, and prefetch telemetry is accounted separately from demand. Host-to-device copies are batched through cudaMemcpyBatchAsync when CUDART >= 12.80, with a per-copy fallback otherwise. Targets whose buffer is not MoE-cached are skipped with a one-shot warning. --moe-lookahead 0 leaves the decode step unchanged. A non-zero width without --moe-expert-cache-size > 0 fails loudly at model load. Adds test-moe-cache --lookahead-prefetch-only and the lookahead prefetch legacy layer case. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

The speculative prefetch path picked its victim with plain LRU, so a prediction could displace an expert that demand had already proven hot. The eviction guard the design requires (docs/moe-lookahead-design.md, "Invariants to preserve" #2 and the colibri safety invariant at c:1462-1478) was missing: only the pin guard - never evict a slot a running GEMM is reading - was implemented, on both the demand and speculative paths. Port colibri's PILOT_EVICT_GUARD. A speculative fill may displace a resident only when that resident is not genuinely warm; a resident is protected when it has at least 2 demand accesses AND is clearly hotter than the prediction, by the 25% + 4-frequency hysteresis in LFRU score units (score ported from colibri c/tier.h:40-43: frequency in the high bits, recency saturating in the low byte). A blocked prediction is dropped rather than forced in, so it can never thrash a demand-loaded expert. Heat is recorded on demand accesses only. Counting predictions would let a speculation inflate the very score that decides whether it may displace a resident, which is the failure colibri hit in ggml-org#490. Refactor the eviction core so the demand and speculative paths cannot drift: ggml_cuda_moe_cache_select_victim_locked picks the LRU unpinned slot and applies the guard for speculative fills, and ggml_cuda_moe_cache_install_fill_locked records the eviction telemetry and publishes the new entry. The copy-batching split in the prefetch path is unchanged, and the free-slot, pin and unknown-eid behaviour is identical. Add phase_prefetch_dropped so a guard that drops everything (colibri ggml-org#490) is visible rather than silent, and extend the test prefetch accessor with it. Add the lookahead prefetch eviction guard case: a 3-slot pool, a warm resident survives a prediction for a never-seen expert (dropped, no copy, no eviction, still a demand hit), while a once-demanded resident is evicted for it. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…kahead Predict layer il+1's top-k experts from layer il's post-attention state and page them while layer il still computes, so the H2D copy overlaps compute instead of stalling the next layer. Adds GGML_OP_MOE_PREFETCH (an explicit no-op outside CUDA), the graph-level producer, the per-context gate and a reserve-time pool preinstall, plus a debug-only recall instrument. Measured on the rig (Qwen3.8-Flash-Next-APEX-I-Mini, -b 1 -ub 1 so every ubatch is a decode row, --moe-expert-cache-size 84, look-ahead width 8): - decode 37.6 t/s against 43.2 t/s with the feature off, about -13%, because the ids readback synchronizes the stream and forces use_cuda_graph = false. - the MoE grouped-decode certificate rejects the graph unless GGML_OP_MOE_PREFETCH is excluded from the backend use counts, otherwise the run dies with "graph=unproven(14)" and a failed graph compute. - logits are not preserved: PPL 2.9434 against 3.0613, and byte-different dumped logits. Isolation shows the extra MUL_MAT that reads ffn_gate_inp is the trigger; a neutral extra node does not reproduce it, disabling CUDA graphs does not reproduce it, and the prefetch op itself contributes nothing. - every prediction is dropped in this configuration. A pool can only be installed while its group is under legacy cache authority with admission open, and the groups here are not, so preinstall_legacy_pools() fails for all 144 targets and the look-ahead has nothing to page into. Not landable as is. The review PR carries the full isolation matrix. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…roducer Adds a measured-outcome section to the design note: the three blockers with their numbers, the isolation matrix that pins the trigger to the extra MUL_MAT reading ffn_gate_inp, Q1 answered negatively (the authority regime refuses the cross-layer pool install), the throughput cost of the ids readback, the already-recorded byte-identical claim refuted, and the future direction (early-router copy worker plus device-to-host mailbox) if the track is ever re-approached. Also corrects ffn_gate_up_exps being null, the preinstall point, Q3 and Q4. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

Recall was the one unmeasured number in the park decision. Measured it properly: the horizon is one MoE layer (never a token), and the demand path cannot score it because during decode only blk.47 reaches the host-visible ids read - the rest run in the certified grouped path. Ground truth was therefore built in-graph from each layer's own router matmul on its real FFN input. Width 2/4/6/8/10 gives 97/95/91/86/79% recall of the predicted set against 19/38/55/69/79% coverage of the 10 experts the model actually uses; width 8 is about 22x chance. RTX 3060 reproduces width 8 within noise (85.85% / 68.68%). Also records the two traps that had kept this unmeasured: the committed instrument's GGML_LOG_INFO never reaches the server log at default verbosity, and cached-path scoring only ever sees the one layer holding a legacy lease. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…rence The design cites colibri's 71.6% PILOT recall on GLM-5.2 as the expectation for this mechanism. Records that the producer measures 91.50% at width 6 and 86.21% at width 8, with the explicit caveat that the models and expert counts differ so the comparison is not like-for-like. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…design invariant preinstall_legacy_pools returned -1 for all 144 targets and every prediction was dropped. Recording the ruling so it is not re-litigated: the acquire_legacy_cache new-record latch (authority == LEGACY && !admission_closed) binds pool creation to a certified execution, and an unauthenticated install would let a caller claim VRAM pools certification never proved - the overcommit class README.md:25 warns about. L+1 is never the authoritative group while L runs, so cross-layer install is unreachable by design, not broken. If the track reopens the seam is authority publication at graph reserve/certification time, which is a certification-contract change needing its own review. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…evice-side) Reviewer ruling on how blocker 3 would be fixed if the track reopens. Mailbox approved as the transport (it reuses the shipped early-router copy worker plus the async paging half); the FreeToken-style device-side gather is rejected because it needs device-resident slot management and would be a second cache implementation, reintroducing the demand/speculative drift that select_victim_locked and install_fill_locked eliminated. Deleting the use_cuda_graph=false rule alone does not suffice: the op is capture-incompatible as written (pageable D2H memcpy, stream sync, host-side slot booking), and a captured cuStreamWaitValue32 bakes an expected value, so it must be refreshed per step or replaced by a polling kernel. Minimal change set, required guards, and the unchanged sequencing (census exclusion #128 -> authority publication -> transport) are recorded. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

Two env-gated diagnostics, off by default, kept for troubleshooting the look-ahead track: - GGML_CUDA_MOE_PHASE_PROBE=1 attributes each step's GPU time to op classes (attention ops, the MoE expert path, the per-layer embedding read, dense compute, plumbing), separates decode from prefill by inter-step wall time, reports the dispatch mode that produced each step, and prints a per-layer table. One CUDA event pair per maximal run of same-(phase, layer) nodes, read back once per step after a single stream sync (~2.6% of decode throughput on the RTX 3060). - GGML_CUDA_MOE_LOOKAHEAD_DEBUG=1 scores the recorded prediction against the ids the router actually selected, for every MUL_MAT_ID node (the ids are in src[2]), i.e. off the legacy blk.47 lease, and reports recall and precision plus the used-not-predicted population. The other two populations are cache counters that were counted but never printed; they now appear on the moe-cache-phase line as prefetch_dropped and evicted_prefetched_unused. Both probes turn themselves off with a single stderr note if an event or a memcpy cannot be issued, which is what happens under CUDA graph capture, so a diagnostic cannot break a capture run; during replay the host never visits the dispatch loop either, so an attribution needs GGML_CUDA_DISABLE_GRAPHS=1. With the env vars unset the only added work is a cached getenv lookup behind a static bool. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

…sults Instrument A on the RTX 3060 at ctx 81920 / cache 42: a decode step is 69% MoE execution, 25% dense compute, 1.5% attention ops and 0% per-layer embedding, with the phase rows summing to the step time; prefill is 88% MoE. In the same window SM sits at 95-99% and the DRAM controller at 19-26%, the PCIe link is gen4 x4 carrying 337.6 MiB of expert traffic per token at ~47% of its ceiling, and the grouped telemetry reports every staged copy already complete, so the limiter is MoE kernel execution rather than PCIe or VRAM bandwidth. Instrument B, scoring predictions in the dispatch loop instead of the legacy blk.47 lease: 64.5% of the used experts were predicted and 80.6% of the predictions were used, the same order as the earlier in-graph measurement. The other two populations are structurally zero in this configuration because the consumer is inert (decode phase line ops=0, legacy cache authority printed once), so every prediction is simply unused. Also records the independent finding that the rig's -ot per_layer_token_embd=CPU is redundant and inert. Assisted-by: Oh My Pi (deepseek-v4.1-flash)

ddvnguyen · 2026-09-14T14:44:16Z

Measured addendum from the same 3060 run, because it changes what the consumer is worth:

moe-cache-phase: phase=prefill ops=4512 l1_hits=127479 l1_misses=128285 l1_hit_rate=49.84%
  h2d_copies=128285 h2d_mib=75928.77 prefetch_hits=122059 prefetch_misses=122327
  prefetch_used=4720 prefetch_h2d_mib=73356.04 evicted_prefetched_unused=114496
  prefetch_dropped=0 ids_d2h_mib=3833.06 ids_d2h_ms=464.692 op_cpu_ms=1398.584
  copy_wait_events=423 copy_wait_event_ms=0.292

During prefill the legacy lease path installs 122,327 speculative slabs, uses 4,720 of
them (3.9%) and evicts 114,496 without a single hit - 73.4 GiB of prefetch H2D plus
3.8 GiB of ids D2H per request for a 50% L1 hit rate, with no queue wait at all
(copy_wait 0.29 ms over 423 events). The copies are fully hidden; the install decisions
are ~96% wasted. Decode does not use this path at all (ops=9, the blk.47 lease) and runs
on the grouped path instead.

Two consequences:

The counters added in 9d24345 (prefetch_dropped, evicted_prefetched_unused) are
what make this visible; the 114,496 wasted installs were counted but never printed.
The LFRU victim ranking behaves, but the install side over-installs by roughly 25x in
prefill. Any reopening of this track should fix prediction/install precision before
adding a wider look-ahead, otherwise extra prediction depth lands in
evicted_prefetched_unused.

Blocker status is unchanged: the producer is still not landable.

ddvnguyen and others added 5 commits September 13, 2026 23:03

docs: fold measured evidence from upstream RFC ggml-org#24528 into lo…

137fa17

…ok-ahead design

ddvnguyen mentioned this pull request Sep 14, 2026

MoE decode: an extra reader of a router weight (ffn_gate_inp) changes model output #128

Open

ddvnguyen added 6 commits September 14, 2026 09:17

ddvnguyen mentioned this pull request Sep 14, 2026

MoE decode optimization: tiered residency plan, measured attribution, and what is already implemented #129

Open

Navigation Menu

MoE look-ahead PR-P1: producer (NOT LANDABLE, see body) - #127

MoE look-ahead PR-P1: producer (NOT LANDABLE, see body)#127
ddvnguyen wants to merge 11 commits into
feat/763-reconcile-qwen4exp-mtpfrom
feat/moe-lookahead-p1

ddvnguyen commented Sep 13, 2026 •

edited

Loading

Uh oh!

ddvnguyen commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ddvnguyen commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR-P1: look-ahead prefetch producer (NOT LANDABLE)

What is implemented

Blocker 1: the producer changes the model output

Blocker 2: every prediction is dropped, so the feature is inert

Blocker 3: -13% decode throughput

Correctness work that is independent of the blockers

Verification performed

Recommended decision

Diagnostics kept on the branch (env-gated, inert when unset)

Uh oh!

ddvnguyen commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ddvnguyen commented Sep 13, 2026 •

edited

Loading