perf(speculative): decide the MTP verify width by measured throughput, not an acceptance proxy

docs/environment-variables.md

@@ -188,6 +188,7 @@ The OpenAI audio endpoints (`/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1
 | `MLXCEL_GDN_CHAIN_PARITY` | `0` to disable, any other value (or unset) to enable | on | **Advanced, diagnostic escape hatch.** Gates the chain-parity gated-delta Metal kernel used by Qwen 3.5 MTP's speculative verify and rollback-replay paths (issue #1165). The standard gated-delta kernel carries float32 recurrent state across a `T = K` verify block and rounds it to the storage dtype only once at the end, while the classic single-token decode chain rounds after every token; a `T = K` verify block is therefore NOT bit-identical to `K` consecutive single-token decode steps unless the state is rounded per in-block step. The chain-parity kernel (`gated_delta_step_seqpar`) does that rounding, which is what makes Qwen 3.5 MTP's temperature-0 output byte-identical to classic decode. **Setting this to `0` forfeits that exactness contract**, restoring the pre-#1165 block numerics for A/B attribution of the parity kernel's own cost and acceptance effect; do not set it to `0` in a deployment that needs byte-identical speculative output. Metal-only: the non-Metal ops fallback ignores the flag (the parity guarantee does not exist off Metal today). **The kernel is necessary but not sufficient**: byte-identity also requires every quantized projection to dispatch to the same MLX kernel at `M = block_size` as at `M = 1`, which is not true on every GPU generation or at every block width, so the runtime probe behind `MLXCEL_MTP_ALLOW_INEXACT` is what actually decides whether MTP engages. See `docs/benchmark_results/qwen38-mtp-m1ultra-2026-08-16.md` for the measured kernel cost (inside the dispatch-noise band). |
 | `MLXCEL_MTP_ALLOW_INEXACT` | `1`/`true`/`yes`/`on` to enable; unset or anything else to disable | **off** | Engage Qwen 3.5 MTP speculative decoding even when the startup exactness probe reports that the multi-token verify block is **not** byte-identical to the single-token decode chain. Before enabling MTP the runtime now measures the property instead of predicting it: one synthetic verify block and the equivalent single-token chain are run from the same state on the loaded checkpoint at the configured `--draft-block-size`, and their logits are compared byte for byte (three independent synthetic inputs, each two short prefills plus `K + 1` forwards; measured 4.9 s for the first call and 1.3 s for a later one per input on a Qwen3.8-27B 4-bit target on an M1 Ultra, the difference being MLX's one-time kernel compilation; more than one input because a kernel pair can disagree by only a byte or two out of ten thousand, at which amplitude a single draw can read as equal; memoized per (model, block width) and warmed at worker startup so it never lands on the request path). A divergence means temperature-0 speculative output would silently differ from `mlxcel generate` without `--draft-model`, so the default is to decline and run classic decode. The static conditions (Metal backend, `supports_metal_gated_delta_kernel` geometry) still apply and are checked first; this probe covers what they cannot, namely which MLX kernel each **quantized projection** dispatches to at `M = K` versus `M = 1`. That choice depends on the GPU generation, the quantization mode, the operand sizes and the block width: `use_qmv_wide` in [`mlx/backend/metal/quantized.cpp`](https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp) sends `M >= 2` to a different reduction whenever `mode != "affine" || arch_gen >= 15`, and `get_qmv_batch_limit` sends `M` above 10, 12, 18 or 32 (by architecture size and generation) to the matrix-matrix kernel. Measured: an affine 4-bit Qwen3.8-27B target on an M1 Ultra is byte-identical at block widths 1 through 11 and diverges at 12 (the `arch_size == 'd'` branch); the same checkpoint on an M5 Max diverges from block width 2, where the `M >= 2` split fires before any batch limit can be observed. Within one checkpoint the limit is per projection, not per model: Gemma 4 12B's attention shapes hold to 17 on an M1 Ultra while its MLP shapes break at 12, so a model's own cliff is the minimum over its shapes, which is why this is measured rather than tabulated. Note the ordering that #1199 introduced: on a failing probe the gate first retries with `qmv_wide` disabled and keeps the narrow kernel when that restores exactness, and only a probe that fails **both** ways consults this flag. On Apple GPU generation 15+ the narrow retry passes, so this flag alone is inert there: the process is pinned narrow, output stays byte-identical, and the log shows the retry's INFO line rather than the override warning (verified live on M3 Ultra, 2026-08-22, byte-identical output with and without the flag; see `benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md`). To research the fast kernel there, set `MLXCEL_QMV_WIDE=1` together with this flag: the pin skips the retry and this flag then engages MTP on the wide kernel, forfeiting byte-identity with the loud WARN. This flag alone is load-bearing only where no exact kernel selection exists at the configured block width. Read once per process. |
 | `MLXCEL_QMV_WIDE` | `0`/`false`/`no`/`off` to disable; `1` (or any other value) to pin wide | unset (wide, until the MTP gate's retry turns it off) | Operator pin for MLX's `qmv_wide` kernel, the faster reduction for `M >= 2` quantized matmuls on Apple GPU generation 15+ (overlay in `src/lib/mlx-cpp/patches/mlx/backend/metal/quantized.cpp`, added by #1199). **Setting the variable at all, to any value, counts as an operator pin**: the MTP exactness gate's retry (`retry_without_qmv_wide`) is skipped in both directions, so `MLXCEL_QMV_WIDE=1` keeps the wide kernel and makes a failing probe decline MTP instead of buying exactness back, and `MLXCEL_QMV_WIDE=0` runs the whole process narrow from the start. Unset, the kernel is wide until a failing MTP probe's retry finds the narrow kernel exact and pins the process narrow for good. The pin is process-wide and sits on the dispatch path of every quantized matmul; what non-MTP work pays for the narrow state is measured in `benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md` (nothing measurable on batched decode, about 15 ms per prompt-cache-hit request's suffix prefill). Read once per process at first dispatch; `mlxcel_core::set_qmv_wide` can move it at runtime and the gate is its only caller. |
+| `MLXCEL_MTP_BLOCK_CONTROLLER` | `proxy` to pin the acceptance-proxy gate; `requested` to honour the requested width immediately (measurement mode); any other value (or unset) selects the throughput comparator | throughput | Which controller decides the B=1 MTP verify width when `--draft-block-size` exceeds the drafter's configured depth (issue #1207). The default measures the decision: after a short warm-up the round loop alternates measurement windows (32 rounds) between the configured depth and the requested ceiling, compares emitted tokens per millisecond of round time, holds whichever measures faster (a challenger needs a 2% lead; ties go to the configured depth), and re-challenges the loser on a growing backoff (4, 16, then every 64 windows), with a collapsing challenger aborted after 4 rounds once it trails by more than 35% so a harmful ceiling (the Qwen 3.8 pairing measures 5.80 against 21.30 tok/s at width 12) costs rounds rather than windows. Evidence lives in the generator, so a server process keeps it across requests; drafters that set `prefer_requested_block_size` (Qwen 3.5 MTP) bypass both controllers and always honour the request, exactly as before. Set to `proxy` to restore upstream's fully-accepted-prefix gate, which issue #1207 measured holding the Gemma 4 12B pairing about 5% below its optimum (93.54 against 98.16 tok/s at requested width 5 on an M5 Max) because at 0.585 acceptance the configured prefix is rarely fully accepted no matter how profitable widening is. The batched (B>1) loop stays on the proxy gate regardless: the row-averaged accept length is the only per-round signal it measures today. Set to `requested` to bypass both controllers and draft at the requested width from the first round: this is the width-sweep measurement mode (#1207's own sweep needed a temporary code patch to hold widths; this value is that patch as a switch), not a deployment setting, and the harmful-ceiling protection is off under it. Read once per generator. |
 | `MLXCEL_MTP_TICK_SLICE` | `0`/`false`/`no`/`off` to disable, any other value (or unset) to enable | on | Tick-cooperative B=1 MTP serving (issue #734). When on (the default), a B=1 MTP request on the Gemma 4 family is served one speculative round per scheduler tick, alternating with the classic decode/prefill actions, so concurrent classic-decode rows advance between rounds and the head-of-line stall a speculative request imposes drops from the whole burst to about one round (`burst_wall_ms` in the finalize log reports the max single-tick wall). Tokens stream per round instead of in one end-of-burst lump. Set to an off value to restore the legacy run-to-completion burst (the whole request served inside one tick). The interleaving trades roughly 27% of the speculative request's own aggregate decode throughput (cross-tick round gaps) for that bounded stall, so a deployment serving speculative requests without concurrent classic traffic can turn it off to keep the full-throughput burst. Greedy output, acceptance accounting, and every other env gate are unchanged in both modes; DFlash and the batched B>1 paths always run to completion regardless of this flag. |
 | `MLXCEL_MTP_SLICE_GRANT_ROUNDS` | non-negative integer | `8` | Grant budget for one hold of the tick-slice speculative slot (issue #746), counted in executed slices (slice 0, the prefill + seed, counts as the first slice of a grant). While a slice is in flight, up to 2 further tick-slice-eligible requests park in a grant backlog instead of permanently falling back to classic decode; once the active request has run this many slices with the backlog non-empty, it parks at the next round boundary and the slot is granted to the next request (priority lane first, FIFO within a lane, with an anti-starvation floor: an entry passed over by 2 grant decisions is granted next regardless of lane), so concurrent long streams share speculative acceleration in bounded turns. The budget is read once per grant and per admission decision (cached for the per-round expiry check), so changes apply from the next grant. The budget binds only under contention: a single speculative request never rotates and behaves exactly as under #734. Rotation preserves per-request token streams byte-identically (the drafter is re-armed from the session's own stored verify output at every round). `0` disables rotation and restores the pre-#746 behavior: the active request holds the slot for its whole generation and every concurrent speculative request falls back to classic decode. Unparseable values fall back to the default. |
 | `MLXCEL_SPECULATIVE_STOCHASTIC_ACCEPT` | `1`/`true`/`yes`/`on` to enable; unset or anything else to disable | **off** | Acceptance-optimal speculative acceptance for the classic `SpeculativeGenerator` path (offline `mlxcel generate --draft-model`), issue #902. When on, `temperature > 0` verification uses modified rejection sampling (accept the drafted token `t` iff `u * q(t) <= p(t)` for a fresh `u ~ U[0,1)`, and on the first rejection emit a draw from the normalized residual `relu(p - q)`) instead of the default sampler-match rule (accept iff the draft equals an independent draw from the target sampler). **Both rules are distribution-preserving**: the emitted stream is a target-only sample either way, which is the central correction to the issue's premise. What changes is the acceptance probability, which rises from `sum_x p(x) q(x)` to `sum_x min(p(x), q(x))`, the maximal-coupling ceiling for any correct rule. **Opt-in rather than default** because the gain is the ratio between those two quantities and it collapses toward 1 whenever the drafter is confident (`q(t*) ~ 1` makes `min(p, q)` and `p * q` coincide); measured at about 1.02 on a Llama-3.1-8B / Llama-3.2-1B pair at temperature 0.7, which does not pay for two extra full-vocabulary passes and a host sync per verified position. Check the available gain with `MLXCEL_SPECULATIVE_ACCEPT_DIAG=1` before enabling. Enabling changes the RNG stream, so at an equal seed the emitted tokens differ from a default run even though the distribution is identical. Greedy (`temperature == 0` or `top_k == 1`) never reaches either rule and is byte-identical. The Gemma 4 MTP and DFlash round loops are unaffected: they select the target token by argmax regardless of temperature, so this switch is inert there. `SpeculativeGenerator::with_stochastic_acceptance(bool)` overrides it programmatically. Read once per process. See [`speculative-acceptance.md`](speculative-acceptance.md). |