example : add MoE route-trace for offline routing analysis

[ddvnguyen]

What

Adds examples/route-trace, a measurement-only tool that dumps the per-position MoE expert selection of every routed layer in colibri's ROUTE_TRACE format:

<call> <pos> <layer> <expert>:<gate> ...

It reuses the existing ggml_backend_sched eval callback and the graph's existing ffn_moe_topk-<il> tensor name (llama_context::graph_get_cb), so no core code changes are required. Enabled with LLAMA_ROUTE_TRACE=<path>.

Why

We want to decide whether a cross-layer expert prefetch ("next-token-layers") is worth introducing into the MoE expert cache. Colibri's offline calculators turn a routing trace into a concrete answer:

• c/tools/route_coupling_report.py — cross-layer co-activation lift + simulated prefetch recall at a fixed experts-per-layer budget.
• c/tools/route_pairs.py — builds a .coli_pairs coupling table.

How the trace is captured

• The graph names the selected-expert tensor ffn_moe_topk-<il>, so the eval callback can read it back with ggml_backend_tensor_get.
• Prompt is evaluated one token at a time and an optional greedy continuation is appended, so each row maps to exactly one position.
• Prompt/continuation routing is captured on the decode path, which is the path where CPU/disk expert misses matter.

Results (2026-09-13, RTX 5060 Ti + RTX 3060, greedy decode, top-K = 8)

Qwen3.8-Flash-Next apex (qwen4exp, 48 layers, 512 experts, 437 positions, 305/132 train/test):

predictor	budget 8/layer	budget 16/layer	budget 32/layer
marginal frequency	28.9%	38.3%	49.9%
coupled (L -> L+1)	37.2% (+8.3pp)	51.2% (+13.0pp)	64.7% (+14.8pp)

L->L+1 lift vs independence: median 2.77x, p90 16.94x, p99 101.67x.

Qwopus3.6-35B-A3B (40 layers, 256 experts, 837 positions, 585/252):

predictor	budget 8/layer	budget 16/layer	budget 32/layer
marginal frequency	25.0%	36.1%	51.9%
coupled (L -> L+1)	41.3% (+16.3pp)	57.2% (+21.1pp)	72.4% (+20.5pp)

L->L+1 lift vs independence: median 2.17x, p90 10.83x, p99 58.50x.

L->L+2 is close behind on both models (36.4% / 40.6% at budget 8).

Interpretation

• Cross-layer routing structure is real and model-wide on both checkpoints: a coupling-table prefetch gets 37-41% of the true top-8 at only 8 experts/layer, versus 25-29% for a marginal predictor.
• This is the short-lookahead case: the coupled predictor uses layer L's observed routing (available once L's router has run, before L's expert matmul completes). It is not the longer stale-state prediction colibri calls PILOT (71.6-75.8% recall on GLM-5.2), which needs the L+1 router run early.
• The number alone does not prove an end-to-end win: a wrong prefetch costs bandwidth, and predicted experts may already be resident. Colibri's own decode-failure ledger records PREFETCH=1 at -2.5% on fully resident decode for exactly this reason. The win case is a genuine cold-miss workload (CPU/disk experts), which is the deep-context qwen4exp configuration.
• Relevant fork context: the feat/763-* lineage already carries an experimental, env-gated GGML_CUDA_MOE_EARLY_ROUTER cross-layer lookahead in ggml-cuda/moe-cache.cu. This tool provides the model-specific recall numbers that feature was missing.

Related finding (recorded, no code)

ivanfioravanti/Qwen3.8-Flash-Next-DS4-IQ2 (44.8 GB DS4-targeted GGUF) ships an MTP block (token-level speculation) plus metadata/results. It contains no coupling table, hotlist, or any expert-routing prediction asset, and it is a DS4-format GGUF that colibri cannot read.

Testing

• cmake --build build --target llama-route-trace (CUDA 13.2, arch 86;120).
• Smoke: 40 layers x 15 positions on Qwopus3.6-35B-A3B; both colibri tools parse the file.
• Full traces above run on the two checkpoints; analysis reproducible with the two colibri commands.

Notes

• Default off; no effect unless LLAMA_ROUTE_TRACE is set.
• -n 0 traces the prompt only.
• Gate field is a placeholder (1); the offline consumers only use expert ids.