fix(speculative): the Gemma 4 31B pairing's exactness probe fails both kernels on M3 Ultra, vetoing the burst #1217 just enabled

[inureyes]

Problem

#1217 measured the batch-capable Gemma 4 31B + bf16 assistant pairing at 1.95x to 2.65x on M3 Ultra and switched mtp_b1_default to enable the B = 1 burst for batch-capable targets from Apple GPU generation 15. But that measurement ran at 9e2c6675, which predates #1258's Gemma exactness probe. On current main (dd21ada4) the probe for this pairing fails under qmv_wide and fails again with it disabled, so the default-env gate declines MTP at serve time on the very host and pairing #1217 enabled by default. The static gate says run, the measured gate says decline, and the decline wins, so the 1.95x to 2.65x is unreachable in a default configuration today.

Evidence

Measured 2026-08-22 on a Mac Studio M3 Ultra 512 GB, macOS 26.6.1, during the issue #1261 measurement, recorded in docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md in PR #1278.

Two independent server boots of mlxcel-server -m models/gemma-4-31b-it-4bit --model-draft models/gemma-4-31b-it-assistant-bf16 --draft-block-size 4 with MLXCEL_ENABLE_MTP_B1=1 MLXCEL_MTP_ADAPTIVE=0 both logged:

MTP declined: verify block position 0 differs from the single-token chain in 231782 of 524288 logit bytes. Disabling qmv_wide did not make it exact either. Falling back to classic decode.

The same host's Gemma 4 12B + 4-bit assistant pairing (block 5) and Qwen 3.8 27B + MTP head pairing (block 3) both pass after the narrow retry, so this is specific to the 31B pairing, not to the host or the gate.

A plausible mechanism is the one #1258's "What this does not settle" section names for the M1 Ultra prose divergence: a narrow-kernel divergence involving the 262144-wide LM head. The 31B's head is 5376 to 262144 where the 12B's is 3840 to 262144, so hidden width may be the discriminator, but that is a hypothesis, not a measurement.

Candidate directions

To be decided by measurement, not assumed:

1. Probe at other block widths, since exactness is width-dependent and the burst could run at a width whose probe passes.
2. Characterize the divergence at op level for the 31B head shape to confirm or refute the LM-head mechanism.
3. If no exact configuration exists, decide whether mtp_b1_default's generation 15 batch-capable default should stay (it is harmless but misleading, since the exactness gate vetoes it) and update docs/benchmarks.md's 31B rows, which currently imply the speedup ships by default.
4. Re-measure whether MLXCEL_MTP_ALLOW_INEXACT=1 throughput still matches perf(speculative): re-measure the batch-capable B=1 MTP hardware gate on M1 Ultra and M3 Ultra and adjust mtp_b1_default #1217's rows post-fix(speculative): probe Gemma 4 MTP exactness instead of assuming it #1258.

Acceptance criteria

• The 31B pairing's block-width exactness profile on generation 15 is measured and recorded.
• Either a default configuration exists in which the pairing engages MTP with byte-identity, or the docs and the mtp_b1_default comment state plainly that the pairing requires MLXCEL_MTP_ALLOW_INEXACT=1 on generation 15+ and what that forfeits.
• docs/benchmarks.md's 31B speedup rows carry the qualification.

References

• perf(speculative): re-measure the batch-capable B=1 MTP hardware gate on M1 Ultra and M3 Ultra and adjust mtp_b1_default #1217 (enabled the default)
• fix(speculative): probe Gemma 4 MTP exactness instead of assuming it #1258 (added the probe)
• perf(speculative): the qmv_wide exactness buy-back pins the whole process narrow, taxing batched decode that never asked for it #1261 / PR docs(benchmarks): price the qmv_wide narrow pin's collateral #1278 (where this was found)
• docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md (the record)

[inureyes]

M5 Max cell: same both-kernel failure, so this is not an M3 Ultra quirk

Measured 2026-08-22 on Apple M5 Max 128 GB, macOS 26.6.1, main at 92018a12, offline CLI, gemma-4-31b-it-4bit + gemma-4-31b-it-assistant-bf16.

Probe: fails both kernels at every width tried, with a width-independent signature:

MTP declined: verify block position 0 differs from the single-token chain in
245722 of 524288 logit bytes. Disabling qmv_wide did not make it exact either.

Identical byte count at --draft-block-size 2, 3 and 4 (M3 Ultra read 231782 of 524288). The CLI refuses with the actionable error; a server boot would warm the probe and fall back to classic decode with the WARN.

What the decline costs here (single samples, code prompt, 200 tokens, quiet host):

arm	tok/s
classic decode	27.64
MTP forced (MLXCEL_MTP_ALLOW_INEXACT=1), block 4	62.11 (2.25x)

On this prompt the forced arm's 200 tokens came out byte-identical to classic despite the diverging logits, which is the usual near-tie story: the flips are prompt-dependent (the 2026-08-19 M1 Ultra sweep caught a real one on prose), so the probe's verdict stands and the default stays correct to fail closed.

Two implications for this issue:

1. The gate-vs-gate conflict is generation-15+-wide, not M3U-specific: on every gen 15+ host measured (M3 Ultra, M5 Max), the batch-capable pairing perf(speculative): re-measure the batch-capable B=1 MTP hardware gate on M1 Ultra and M3 Ultra and adjust mtp_b1_default #1217 turned on by default is vetoed by the probe, so the static enable currently delivers nothing anywhere by default, at a measured cost of 1.95 to 2.65x (M3U) and 2.25x (M5 Max) left on the table.
2. The mechanism is not the qmv_wide M-split this family's gate machinery was built around: the divergence survives the narrow retry and its byte count is independent of the block width, which points at a T = 1 versus T > 1 path difference outside the quantized matmul dispatch (candidates worth checking: the single-query maskless SDPA fast path against the masked multi-token path, and the fix(core): f32 matmul and single-query SDPA lose ~fp16 precision on M5, breaking four numeric tests #1065 class of f32 op differences). The Unified 12B pairing passes the narrow retry on the same host, so whatever it is, it is specific to the non-Unified forward or this checkpoint's config (for example final_logit_softcapping).