perf(speculative): close the early-exit verify walk question as a measured no

docs/benchmark_results/mtp-verify-early-exit-decision-m5max-2026-08-22.md

@@ -0,0 +1,146 @@
+# MTP verify early-exit walk: decided by measurement, not built (issue #1179)
+
+Issue #1179 asked whether a device-side early-exit verify walk beats the
+batched full-logits verifier, with a stated prior: the LM head is
+weight-read-bound, so skipping tail positions should save almost nothing.
+The measurement below answers no, the walk should not be built, but for a
+sharper reason than the prior: the projection cost is *not* flat in the
+projected width, and the very effect that makes tail positions expensive
+also makes a sequential walk pay the full weight read once per walked
+position, which loses to the batched projection outright at every
+production block size.
+
+## Environment
+
+| Field | Value |
+|---|---|
+| Host | Apple M5 Max, 128 GB unified memory, macOS 26.6.1 |
+| Build | `cargo build --release --features metal,accelerate`, branch `perf/issue-1179-verify-early-exit-decision` (main at `77c71402` plus this branch's cleanup) |
+| Harness (a) | `examples/mtp_projection_width_bench` 16 5: T=16 projections per batch folded into one eval, best of 5 rounds, weights round-robined over 2 copies (566 to 715 MB each) so every read hits DRAM |
+| Harness (b) | offline CLI, 300 tokens, temperature 0, `MLXCEL_MTP_BLOCK_CONTROLLER=requested` (block width pinned, #1207), `MLXCEL_MTP_ALLOW_INEXACT=1` (widths above the narrow batch limit cannot pass the exactness probe; this sweep measures cost, not output), 15 s cooldowns, Time Machine stopped and verified stopped at both ends, indexers paused |
+| Guard | each width row asserts and prints its logits shape `[1, W, vocab]`; the two arms of the question differ by that shape and nothing else |
+
+## (a) LM-head projection cost against projected width
+
+Projection + argmax at the real head shapes, affine 4-bit group 64.
+
+Default kernel selection (`qmv_wide` on, what a non-MTP process runs):
+
+| W | gemma4-12b tied head (262144 x 3840) | GB/s | qwen3.8-27b lm_head (248320 x 5120) | GB/s |
+|---:|---:|---:|---:|---:|
+| 1 | 0.985 ms | 575 | 1.234 ms | 580 |
+| 2 | 1.020 ms | 555 | 1.301 ms | 550 |
+| 3 | 1.149 ms | 493 | 1.419 ms | 504 |
+| 4 | 1.352 ms | 419 | 1.903 ms | 376 |
+| 5 | 1.846 ms | 307 | 2.246 ms | 318 |
+| 8 | 3.406 ms | 166 | 4.299 ms | 166 |
+| 16 | 2.561 ms | 221 | 3.266 ms | 219 |
+| 32 | 2.452 ms | 231 | 3.111 ms | 230 |
+
+Narrow kernel selection (`MLXCEL_QMV_WIDE=0`, what the MTP verify path
+actually runs on generation 15+ after the #1199 exactness retry):
+
+| W | gemma4-12b tied head | GB/s | qwen3.8-27b lm_head | GB/s |
+|---:|---:|---:|---:|---:|
+| 1 | 0.985 ms | 575 | 1.225 ms | 584 |
+| 2 | 1.243 ms | 456 | 1.453 ms | 492 |
+| 3 | 1.811 ms | 313 | 1.960 ms | 365 |
+| 4 | 2.331 ms | 243 | 2.849 ms | 251 |
+| 5 | 3.141 ms | 180 | 3.353 ms | 213 |
+| 8 | 4.510 ms | 126 | 5.030 ms | 142 |
+| 16 | 2.559 ms | 221 | 3.256 ms | 220 |
+| 32 | 2.450 ms | 231 | 3.110 ms | 230 |
+
+W = 1 is identical in both selections (`M = 1` takes `qmv` regardless),
+and so are W = 16 and 32 (the matrix-matrix kernel ignores the flag). The
+`M` in between is where the narrow pin costs, which is the #1261/#1278
+collateral measured at the head shape.
+
+Two structural facts, visible in both selections:
+
+- The curve is not flat. W = 1 runs at the memory-bandwidth roof
+  (575 to 580 GB/s); the `M >= 2` kernels degrade steadily to about
+  166 GB/s at W = 8; the matrix-matrix kernel takes over at the batch
+  limit and W = 16 and 32 cost *less* than W = 8. The issue's prior
+  (projection cost invariant in W, ceiling near zero) was wrong in the
+  letter but right in the verdict.
+- The early-exit ceiling `t(K) - t(1)` at the widths anyone runs is
+  0.74 to 0.83 ms per round at K = 3 and 1.35 to 1.62 ms at K = 4 under
+  the production (narrow) selection, against verify forwards of 31 to
+  40 ms per round: about 2 to 4% of the verify forward, before paying
+  anything for the walk itself. And the walk cannot collect it, per the
+  next section.
+
+## Why the walk loses even that ceiling
+
+A device-side early-exit walk projects position by position and stops at
+the first mismatch, so it pays `E[A] x t(1)` where `E[A]` is the expected
+number of positions walked, and every one of those positions re-reads the
+full weight matrix. At the measured production operating points:
+
+`E[A]` below is `E[accepted] + 1` under a per-position independence
+approximation of the measured acceptance rate, capped at K. `t(K)` is the
+narrow-selection column, because that is what the MTP verify path runs.
+
+| pairing | K | acceptance | E[A] | walk cost `E[A] x t(1)` | batched `t(K)` | walk wins? |
+|---|---:|---:|---:|---:|---:|---|
+| gemma4-12b | 3 | 0.876 | 2.64 | 2.60 ms | 1.81 ms | no, loses 1.4x |
+| gemma4-12b | 4 | 0.799 | 2.95 | 2.91 ms | 2.33 ms | no, loses 1.2x |
+| gemma4-12b | 8 | 0.636 | 2.67 | 2.63 ms | 4.51 ms | ~1.9 ms, see below |
+| qwen3.8-27b | 3 | 0.827 | 2.51 | 3.10 ms | 1.96 ms | no, loses 1.6x |
+
+The one width where the walk gets ahead, K = 8, is dominated by a far
+simpler lever the width curve exposes: padding the projection to the
+matrix-matrix width costs 2.56 ms there, which collects the same ~1.9 ms
+with a reshape instead of a sequential dispatch-compare-branch walk, and
+K = 8 is not an operating point on either pairing anyway (see below). The
+walk estimate also charges nothing for its own K sequential
+dispatch/compare steps.
+
+The one cell where the walk is marginally ahead (K = 8, ~0.7 ms against a
+57 ms verify forward, ~1.2%) is not an operating point: the production
+sweep below puts Gemma 4's throughput optimum at K = 4 and the #1207
+controller now finds that optimum by measurement, and the estimate charges
+the walk nothing for its K sequential dispatch/compare steps, which a real
+implementation would pay.
+
+## (b) Production verify cost against block size
+
+Same protocol as (a)'s harness row. `verify_forward_ms / rounds`:
+
+| K | gemma verify ms/round | gemma tok/s | qwen verify ms/round (relative) | qwen tok/s |
+|---:|---:|---:|---:|---:|
+| 3 | 31.2 | 83.1 | 107.9 | 22.6 |
+| 4 | 35.5 | 89.1 | 136.6 | 22.6 |
+| 8 | 57.0 | 85.9 | 255.5 | 15.7 |
+| 16 | 61.1 | 75.2 | 301.4 | 12.5 |
+| 32 | 155.4 | 28.6 | 377.5 | 8.9 |
+
+The Gemma column is consistent with this host's clean baselines (89.15
+tok/s pinned at K = 4 in the #1207 record). The Qwen block ran late in the
+sweep and its absolute values carry sustained-load throttling (this host's
+known failure mode; its clean K = 3 verify round is ~40 ms), so read only
+its ordering: K = 3 and 4 are equivalent, everything wider loses. On both
+pairings the operating range stays at K <= 8, where (a) says early exit
+has nothing to win.
+
+An observation for whoever revisits wide blocks: the W = 8 anomaly (more
+expensive than W = 16) means a K = 8 verify would project cheaper padded
+to the matrix-matrix width than early-exited. If wide blocks ever become
+an operating point, pad-to-qmm is the lever to measure first, not the
+walk.
+
+## Decision
+
+Not built, closed as a measured no (the outcome #1179 names as success):
+
+- At the production widths (K = 3 to 5) the batched projection beats a
+  sequential early-exit walk by 1.2 to 1.6x on the walk's own best case,
+  because the walk re-reads the weight per walked position.
+- The ceiling the walk chases there is 2 to 4% of the verify forward,
+  and where a wide block would make it larger, pad-to-qmm collects it
+  cheaper.
+- The stale gate comment and the `MLXCEL_ENABLE_MTP_DEFERRED` flag, which
+  gated a path that split the verify forward from the projection without
+  deferring any work (identical compute, one extra bridge crossing), were
+  removed with their orphaned helper chain.

docs/environment-variables.md

@@ -183,7 +183,6 @@ The OpenAI audio endpoints (`/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1
 | `MLXCEL_MTP_ADAPTIVE` | `0`/`false`/`no`/`off` to disable, any other value (or unset) to enable | on | Adaptive B=1 MTP policy (issue #333). When on, the server profiles the first few B=1 MTP bursts of each (target, drafter, hardware, block_size) pairing (acceptance length, verify latency, drafter latency, batch size, prompt shape) and settles to a data-driven enable/decline verdict that overrides the static per-hardware gate when the measured profile is clearly favorable or unfavorable, falling back to the static default otherwise. The verdict (enable/decline plus the coarse acceptance rate, no prompt data) is persisted at `${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/mtp-policy/<key-hash>.json` (hint format v3; v2 hints were settled against the pre-#725 verify kernel and the pre-#736 estimator, so they are ignored and the pairing re-profiles once), so profiling runs once per pairing and a restart reuses the verdict. Changing `MLXCEL_DRAFT_BLOCK_SIZE` changes the block_size dimension of the key, so the old hint is discarded and profiling restarts for the new K. Set to an off value to disable profiling and use the pre-#333 static per-hardware gates. `MLXCEL_ENABLE_MTP_B1` still pins the decision and, when set, suppresses profiling. The experimental batched (B>1) path is unaffected and stays behind `MLXCEL_ENABLE_MTP_BATCH`. The speedup estimate is measured, not modeled (issue #736): while profiling, each burst runs a couple of classic-step probe rounds (drafterless rounds whose `[1, 1]` verify forward is shape-identical to a classic decode step; each emits one real greedy token, so nothing is wasted and temperature-0 output stays byte-identical), and the estimator compares the measured speculative round cost (verify + drafter + walk/finalize overhead) against the measured classic step time, which is taken as the median (not the mean) of the per-burst probe means so the first burst's one-time CUDA kernel/graph compilation for the `[1, 1]` verify shape cannot skew the estimate toward a falsely slow classic step. This makes the verdict correct across backends and kernel eras without hardware heuristics: on GB10 with the multirow qmv verify (`MLXCEL_QMV_MULTIROW`, issue #725) the Gemma 4 12B pairing profiles to about 1.5× and enables, while the same pairing on the pre-#725 per-row verify profiles to about 0.5× and declines with margin. When a window collects no probe signal the pre-#736 shape heuristic (issue #638: 1.0 on Apple Silicon, `sqrt(K)` elsewhere) remains as the fallback. The resulting state is readable at `GET /v1/internal/mtp-policy`; see [Adaptive MTP policy API](mtp-policy-api.md), and read it there rather than parsing the hint files, whose format is private. |
 | `MLXCEL_ENABLE_MTP_B1` | `0`/`false`/`no`/`off` to disable, any other value to force on | adaptive (per hardware) | Manual override for the singleton (B=1) MTP burst, in both directions. When set it pins the decision and disables adaptive profiling (issue #333). When unset, the adaptive policy decides (see `MLXCEL_MTP_ADAPTIVE`); with `MLXCEL_MTP_ADAPTIVE=0` the decision is the static per-hardware default (issue #165): non-batchable targets (`gemma4_unified` 12B pairs, whose only decode path is B=1) default **on** everywhere (measured across three prompts: 1.90x to 3.14x on M5 Max, 1.74x to 2.61x on M3 Ultra, 0.95x to 1.48x on M1 Ultra, where the prose end is a 5% loss rather than a gain); batch-capable targets (the 31B + bf16 assistant) default **on from Apple GPU generation 15** (M3, M4, M5), and fall back to classic decode on generation 13 (M1, M2). This was `has_neural_accelerator` (M5 only) until issue #1217, on the strength of ~1.2 to 1.4× on M5 Max against a ~0.75 to 0.96× regression on M1 Ultra, both measured before #1194/#1199/#1203/#1208/#1215 and neither re-measured since. M3 Ultra, which the old predicate lumped with M1 Ultra, was never measured on this pairing at all until #1217 did it: 1.95× (prose), 2.41× (source code) and 2.65× (enumeration) on 2026-08-20 under the #1215 protocol, at a verify-round cost of 1.51 classic steps against 2.71 on M1 Ultra. The discriminator is the `use_qmv_wide` split (an affine-quantized projection at `M >= 2` runs as one wide pass from generation 15 and as `K` narrow passes below it), not the Neural Accelerator. M4 is grouped by that shared dispatch rather than measured. Generation 13 keeps declining and the width sweep supports that: round cost fits `0.83 + 0.170 K` classic steps for this pairing on M3 Ultra against `1.14 + 0.090 K` for the 12B pairing there, so the bf16 drafter costs about 1.9x as much per block position, and carrying that ratio onto generation 13's `1.35 + 0.346 K` puts a block-4 round near 3.6 classic steps, which 2.96 to 3.99 emitted tokens would only just cover. See `docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md`. |
 | `MLXCEL_ENABLE_MTP_BATCH` | truthy value | off | **Advanced.** Forces the batched Gemma 4 MTP burst path for parity/debug testing. Not governed by the adaptive policy (issue #333), which scopes to the validated, byte-identical B=1 path. |
-| `MLXCEL_ENABLE_MTP_DEFERRED` | `1` | off | **Advanced.** Enables the deferred greedy verifier path for Gemma 4 MTP when sampling settings allow it. |
 | `MLXCEL_METAL4_ATTENTION` | `0`/`false`/`no`/`off` to disable; unset or any other value to enable | on (where the hardware has it) | **Advanced, diagnostic kill switch.** Forces the M5 neural-accelerator fused attention route off on hardware that has it, so `layers::metal4_causal_attention` is skipped and the ordinary SDPA path runs instead. Off-switch only: it cannot turn the route on where `has_neural_accelerator && macos_supports_na` is false, so setting it on an M1 is inert. This route is the first suspect whenever an M5 disagrees numerically with an earlier Apple GPU generation (issue #1065), and before this switch the only way to A/B the hypothesis was to patch `should_use_metal4_attention` and rebuild, which is what the #1182 M5 investigation had to do. Read once per process, so set it before starting `mlxcel` or `mlxcel-server`. Inert on non-Metal builds. |
 | `MLXCEL_GDN_CHAIN_PARITY` | `0` to disable, any other value (or unset) to enable | on | **Advanced, diagnostic escape hatch.** Gates the chain-parity gated-delta Metal kernel used by Qwen 3.5 MTP's speculative verify and rollback-replay paths (issue #1165). The standard gated-delta kernel carries float32 recurrent state across a `T = K` verify block and rounds it to the storage dtype only once at the end, while the classic single-token decode chain rounds after every token; a `T = K` verify block is therefore NOT bit-identical to `K` consecutive single-token decode steps unless the state is rounded per in-block step. The chain-parity kernel (`gated_delta_step_seqpar`) does that rounding, which is what makes Qwen 3.5 MTP's temperature-0 output byte-identical to classic decode. **Setting this to `0` forfeits that exactness contract**, restoring the pre-#1165 block numerics for A/B attribution of the parity kernel's own cost and acceptance effect; do not set it to `0` in a deployment that needs byte-identical speculative output. Metal-only: the non-Metal ops fallback ignores the flag (the parity guarantee does not exist off Metal today). **The kernel is necessary but not sufficient**: byte-identity also requires every quantized projection to dispatch to the same MLX kernel at `M = block_size` as at `M = 1`, which is not true on every GPU generation or at every block width, so the runtime probe behind `MLXCEL_MTP_ALLOW_INEXACT` is what actually decides whether MTP engages. See `docs/benchmark_results/qwen38-mtp-m1ultra-2026-08-16.md` for the measured kernel cost (inside the dispatch-noise band). |
 | `MLXCEL_MTP_ALLOW_INEXACT` | `1`/`true`/`yes`/`on` to enable; unset or anything else to disable | **off** | Engage Qwen 3.5 MTP speculative decoding even when the startup exactness probe reports that the multi-token verify block is **not** byte-identical to the single-token decode chain. Before enabling MTP the runtime now measures the property instead of predicting it: one synthetic verify block and the equivalent single-token chain are run from the same state on the loaded checkpoint at the configured `--draft-block-size`, and their logits are compared byte for byte (three independent synthetic inputs, each two short prefills plus `K + 1` forwards; measured 4.9 s for the first call and 1.3 s for a later one per input on a Qwen3.8-27B 4-bit target on an M1 Ultra, the difference being MLX's one-time kernel compilation; more than one input because a kernel pair can disagree by only a byte or two out of ten thousand, at which amplitude a single draw can read as equal; memoized per (model, block width) and warmed at worker startup so it never lands on the request path). A divergence means temperature-0 speculative output would silently differ from `mlxcel generate` without `--draft-model`, so the default is to decline and run classic decode. The static conditions (Metal backend, `supports_metal_gated_delta_kernel` geometry) still apply and are checked first; this probe covers what they cannot, namely which MLX kernel each **quantized projection** dispatches to at `M = K` versus `M = 1`. That choice depends on the GPU generation, the quantization mode, the operand sizes and the block width: `use_qmv_wide` in [`mlx/backend/metal/quantized.cpp`](https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp) sends `M >= 2` to a different reduction whenever `mode != "affine" || arch_gen >= 15`, and `get_qmv_batch_limit` sends `M` above 10, 12, 18 or 32 (by architecture size and generation) to the matrix-matrix kernel. Measured: an affine 4-bit Qwen3.8-27B target on an M1 Ultra is byte-identical at block widths 1 through 11 and diverges at 12 (the `arch_size == 'd'` branch); the same checkpoint on an M5 Max diverges from block width 2, where the `M >= 2` split fires before any batch limit can be observed. Within one checkpoint the limit is per projection, not per model: Gemma 4 12B's attention shapes hold to 17 on an M1 Ultra while its MLP shapes break at 12, so a model's own cliff is the minimum over its shapes, which is why this is measured rather than tabulated. Note the ordering that #1199 introduced: on a failing probe the gate first retries with `qmv_wide` disabled and keeps the narrow kernel when that restores exactness, and only a probe that fails **both** ways consults this flag. On Apple GPU generation 15+ the narrow retry passes, so this flag alone is inert there: the process is pinned narrow, output stays byte-identical, and the log shows the retry's INFO line rather than the override warning (verified live on M3 Ultra, 2026-08-22, byte-identical output with and without the flag; see `benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md`). To research the fast kernel there, set `MLXCEL_QMV_WIDE=1` together with this flag: the pin skips the retry and this flag then engages MTP on the wide kernel, forfeiting byte-identity with the loud WARN. This flag alone is load-bearing only where no exact kernel selection exists at the configured block width. Read once per process. |