perf(speculative): gate B=1 MTP on GPU generation, not Neural Accelerator (#1217)

[inureyes]

What this does

Re-measures the batch-capable B=1 MTP pairing on M3 Ultra and moves mtp_b1_default off the has_neural_accelerator proxy onto the mechanism the round-cost model already attributed the host ordering to: the use_qmv_wide split at Apple GPU generation 15.

Why the old gate was wrong

The gate ran batch-capable targets only where has_neural_accelerator held, which is M5 and nothing else. That rested on two numbers from #165: about 1.2 to 1.4x on M5 Max against a 0.75 to 0.96x regression on M1 Ultra. Both predate #1194, #1199, #1203, #1208 and #1215, and M3 Ultra had never been run on this pairing at all, so a generation-15 part was grouped with generation 13 by a binary proxy.

Neither bench script could reach the pairing the gate governs. bench_speculative.sh covered the 12B and Qwen pairings and bench_block_width.sh the same two, so the founding numbers were not reproducible through the #1215 protocol. That is most of why they went stale without anyone noticing. Both scripts gain a gemma31b case here.

Measured

Gemma 4 31B + bf16 assistant, M3 Ultra (512 GB), 2026-08-20, current main, block 4, greedy, under with_indexers_paused.sh. Round cost is emitted per verify / speedup, in this host's own classic decode steps.

Output	acceptance	emitted/verify	classic	MTP	speedup	round cost
enumeration	1.000	3.990	31.5	83.6	2.65x	1.51
source code	0.882	3.646	31.8	76.6	2.41x	1.51
prose	0.656	2.956	31.7	61.9	1.95x	1.52

Spreads were 0.5% to 1.0% on both arms of every row against the harness limit of 4%. Three prompts with nothing in common agree on the round cost to within 0.7%, and the pairing clears its break-even by roughly double on all of them. Single-stream acceptance is 0.66 to 1.00, not the too-low value the old code comment asserted.

Width sweep on the code row, 8 interleaved rounds: peak at width 5 with width 4 tied inside its 1.7% spread, everything from 3 to 8 gaining more than 2x. Fit round cost = 0.83 + 0.170 K classic steps, largest residual 0.06.

Why generation 13 still declines

This is the part the sweep changed my mind on, so it is worth stating plainly. The tempting move is to take M1 Ultra's published block-4 round cost of 2.71, note this pairing emits 2.96 to 3.99 tokens per verify, and conclude post-#1203 M1 Ultra would now clear break-even. That is not sound: 2.71 belongs to the 12B pairing with a 4-bit drafter, and the sweep shows this pairing's slope is 1.9x steeper (0.83 + 0.170 K against the 12B pairing's 1.14 + 0.090 K on the same host). The two lines merely cross at K = 4, which is the only reason their block-4 costs match.

Carrying that slope ratio onto generation 13's 1.35 + 0.346 K puts a block-4 round near 3.6 classic steps there, which the emitted tokens would only just cover, consistent with the founding 0.75 to 0.96x. So the generation-13 decline reads as sound rather than merely stale and it stays. The estimate extrapolates across a pairing and a generation at once, so it is a reason to leave M1/M2 alone, not a result; it is written down in falsifiable form so an M1 Ultra run can settle it.

The new predicate is strictly more permissive than the one it replaced, so no host loses a path it previously had. A unit test pins that property so the safety argument cannot rot.

Verified through the real dispatch path

Unit tests cover mtp_b1_default as a pure function, which is not the same as showing the running scheduler consults it. Same host, same checkpoints, server rather than the offline CLI, MLXCEL_MTP_ADAPTIVE=0 so the static gate decides:

Binary	MLXCEL_ENABLE_MTP_B1	Scheduler outcome
main at 9e2c6675	unset	declined, classic decode
this branch	unset	ran the burst (block 4, 80 tokens over 21 rounds, acceptance 0.921)
this branch	0	declined, classic decode

Worth recording separately: the offline mlxcel generate path never consults this gate. mtp_b1_default has exactly one caller, Scheduler::mtp_b1_should_run, and MtpPolicy is built only in the server worker, so the bench harness runs the burst unconditionally on every host. That is what makes it the right instrument for deciding the gate, and it is also why MLXCEL_ENABLE_MTP_B1=1 and MLXCEL_MTP_ADAPTIVE=0 are inert there.

Checks

cargo fmt --check clean, cargo clippy --all-targets clean, 361 server::batch tests pass, plus the new hardware and mtp_b1_default unit tests.

Not done here

Issue #1217 deliberately stays open, because two of its acceptance criteria need hardware this host is not. No closing keyword is used anywhere in this PR or its commit.

• M1 Ultra rows for this pairing. No generation-13 host was available. Nothing here changes M1 or M2 behaviour, and the prediction above is left falsifiable for that run.
• M5 Max re-measurement. Its ~1.2 to 1.4x is the founding, pre-perf(speculative): quantize the MTP drafter's projections at load #1203 figure. M5 keeps its path either way so the gate does not depend on it, but the M3 Ultra rows now exceed it, which is itself a reason to re-run it.
• M3 Ultra Qwen rows were already measured on current main under this protocol in feat(bench): measure speculative decoding properly, and record what it measures #1215 and are in docs/benchmarks.md (1.67x at block 3, full width sweep). Not re-run.

Full record and method: docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md.

Refs #1217.