diff --git a/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.en.md b/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.en.md new file mode 100644 index 000000000..c1ba8ff26 --- /dev/null +++ b/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.en.md @@ -0,0 +1,62 @@ +# Technical Report: PR #1254 - B=1 MTP gate on GPU generation + +**Date**: 2026-08-20 +**Status**: Partial +**Languages**: Rust, Shell, Markdown +**Risk Level**: Medium + +## Executive Summary + +PR #1254 re-measures the batch-capable singleton MTP pairing on M3 Ultra and moves the static gate `mtp_b1_default` off the `has_neural_accelerator` proxy onto Apple GPU generation 15, the `use_qmv_wide` split. The gate was declining a pairing that measures 1.95x to 2.65x on that host. The change is deliberately partial: it does not close issue #1217, whose M1 Ultra acceptance criteria need hardware the measuring host is not. + +## 1. Problem Statement + +The B=1 MTP burst for batch-capable targets ran only where `has_neural_accelerator` held, which is M5 and nothing else. That policy rested on the founding measurement in #165: about 1.2 to 1.4x on M5 Max against a 0.75 to 0.96x regression on M1 Ultra. Both numbers predate #1194, #1199, #1203, #1208 and #1215. M3 Ultra had never been run on the pairing, so a generation-15 part was classified with generation 13 by a binary proxy that does not track the mechanism. + +A second, quieter problem made the first one durable. Neither `scripts/bench_speculative.sh` nor `scripts/bench_block_width.sh` had a case for the Gemma 4 31B + bf16 assistant pairing, so the pairing the gate governs had no path through the #1215 measurement protocol. The founding numbers could not be reproduced or refreshed by the harness that exists to keep such numbers honest. + +## 2. Technical Decisions + +### 2.1 Discriminate on GPU generation rather than the Neural Accelerator + +`AppleSiliconGen::wide_quantized_projections` encodes MLX's `use_qmv_wide` predicate reduced to its chip-dependent half: from generation 15 an affine-quantized projection at `M >= 2` runs as one wide pass, and on generation 13 the verify block runs as `K` narrow passes whose cost grows with the block. This is the mechanism the published round-cost model already credited for the host ordering, so the gate now reads the mechanism instead of a correlate of it. `Unknown` reads false, preserving today's decline for non-Apple hosts where a K-wide verify does not amortize at all (#638) and for Apple generations newer than the enumerated ones. + +### 2.2 Leave generation 13 declining, and say why that is a conclusion + +The width sweep made the conservative choice defensible rather than merely cautious. Round cost fits `0.83 + 0.170 K` classic steps for this pairing on M3 Ultra against `1.14 + 0.090 K` for the 12B pairing on the same host: the bf16 drafter costs about 1.9x as much per extra block position, and the two lines cross at K = 4. A naive transfer of M1 Ultra's published block-4 round cost of 2.71 across pairings would have predicted that generation 13 now clears break-even; carrying the slope ratio instead puts a block-4 round near 3.6 classic steps there, which the emitted tokens would only just cover, consistent with the founding regression. + +### 2.3 Make the safety argument a test rather than a claim + +The new predicate is strictly more permissive than the one it replaced, so no host loses a path it previously had. That is what makes it legitimate to change the gate on evidence from one host without re-measuring the others, so a unit test asserts the implication over every enumerated generation rather than leaving it as prose. + +## 3. Change Summary + +| Area | Change | +|---|---| +| `src/lib/mlxcel-core/src/hardware.rs` | New `AppleSiliconGen::wide_quantized_projections` predicate plus two unit tests (generation split, and the weaker-than-NA implication). | +| `src/server/batch/speculative_burst.rs` | `mtp_b1_default` third parameter becomes `wide_quantized_projections`; `mtp_b1_burst_enabled` reads it from `silicon_gen`; docstring rewritten to the new measurements with the two evidence limits stated; two new unit tests. | +| `src/server/batch/mtp_policy.rs` | Field renamed and re-sourced, so the adaptive policy's ambiguous-window fallback follows the same predicate. | +| `src/server/batch/scheduler.rs` | Decline-path comment updated to the new policy. | +| `scripts/bench_speculative.sh`, `scripts/bench_block_width.sh` | New `gemma31b` case in both, so the protocol can reach the pairing the gate governs. | +| `docs/benchmarks.md`, `docs/environment-variables.md` | 31B section rewritten with the measured rows; the claim that the pairing's speedup comes from B>1 windows and that its single-stream acceptance is too low is removed as falsified. | +| `docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md` | New dated record: environment, three-prompt table, width sweep, round-cost fits, dispatch-path verification, and an explicit list of what was not measured. | + +## 4. Review Findings + +Two corrections were made during the work rather than after it. + +The first was an analytical error caught by the width sweep. An earlier draft of the docstring and the env-var reference asserted that the round-cost model predicted post-#1203 M1 Ultra would now clear break-even on this pairing. That inference transferred a round cost between two pairings with different drafter dtypes, which the sweep then showed have slopes differing by 1.9x. Both statements were corrected before commit, and the results record now warns explicitly against transferring a round cost between these pairings at any width other than the crossing point. + +The second was a PR-body defect. The sentence stating that the PR does not close #1217 itself contained the substring `close #1217`, which GitHub's closing-keyword parser matched without regard to the negation, so the PR was initially registered as auto-closing the issue it was written to leave open. The body was reworded and `closingIssuesReferences` re-verified as empty. + +## 5. Validation + +`cargo fmt --all --check` clean. `cargo clippy --profile test-fast --features metal,accelerate --all-targets` clean. 361 `server::batch` unit tests pass, plus the new `hardware` and `mtp_b1_default` tests. + +Beyond the pure test seam, the gate was exercised through `Scheduler::mtp_b1_should_run` on the real checkpoints on the measuring host, with `MLXCEL_MTP_ADAPTIVE=0` so the static gate decides. `main` at `9e2c6675` declined the burst; this branch ran it (block 4, 80 tokens over 21 rounds, acceptance 0.921); `MLXCEL_ENABLE_MTP_B1=0` still declined. That before-and-after is on one host with one pair of checkpoints, which is what the acceptance criterion asked for and no more. + +Recorded as part of validation: the offline `mlxcel generate` path does not consult this gate at all. `mtp_b1_default` has exactly one caller and `MtpPolicy` is built only in the server worker, so the bench harness runs the burst unconditionally. This is why the harness is the right instrument for deciding the gate, and why the two env vars the issue prescribes are inert there. + +## 6. Related Work + +Issue #1217 remains open by design. Its M1 Ultra rows and its M5 Max re-measurement need hardware that was not available, and the M3 Ultra Qwen rows it also asks for were already measured on current main under this protocol in #1215. Predecessors: #165 (the founding gate), #333 (adaptive policy, which falls back to this static default on an ambiguous window), #1203 (drafter projections quantized at load), #1215 (the measurement protocol and its guards). diff --git a/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.ko.md b/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.ko.md new file mode 100644 index 000000000..0aca1e17f --- /dev/null +++ b/TECHNICAL_REPORTS/1254-mtp-b1-gate-gpu-generation-20260820.ko.md @@ -0,0 +1,62 @@ +# 기술 보고서: PR #1254 - GPU 세대 기반 B=1 MTP 게이트 + +**작성일**: 2026-08-20 +**상태**: 부분 완료 +**언어**: Rust, Shell, Markdown +**위험도**: Medium + +## 요약 + +PR #1254는 배치 가능 타깃의 단일 요청 MTP 페어링을 M3 Ultra에서 다시 측정하고, 정적 게이트 `mtp_b1_default`의 판별 기준을 `has_neural_accelerator` 대리 지표에서 Apple GPU 15세대의 `use_qmv_wide` 분기로 옮긴다. 기존 게이트는 해당 호스트에서 1.95배에서 2.65배까지 빨라지는 페어링을 거부하고 있었다. 이 변경은 의도적으로 부분 구현이며, 이슈 #1217을 닫지 않는다. 그 이슈의 M1 Ultra 관련 완료 조건은 측정 호스트가 갖추지 못한 하드웨어를 요구하기 때문이다. + +## 1. 문제 정의 + +배치 가능 타깃의 B=1 MTP 버스트는 `has_neural_accelerator`가 참인 곳, 즉 M5에서만 동작했다. 이 정책의 근거는 #165의 최초 측정값이다. M5 Max에서 약 1.2배에서 1.4배, M1 Ultra에서 0.75배에서 0.96배의 성능 저하. 두 수치 모두 #1194, #1199, #1203, #1208, #1215보다 앞선다. M3 Ultra는 이 페어링으로 측정된 적이 아예 없었고, 그 결과 15세대 칩이 메커니즘을 반영하지 못하는 이진 대리 지표에 의해 13세대와 같은 부류로 묶였다. + +문제를 오래 지속시킨 원인은 따로 있다. `scripts/bench_speculative.sh`와 `scripts/bench_block_width.sh` 어느 쪽에도 Gemma 4 31B + bf16 assistant 페어링 항목이 없었다. 게이트가 관장하는 바로 그 페어링이 #1215 측정 프로토콜을 통과할 경로를 갖지 못한 것이다. 수치의 신뢰성을 지키려고 만든 하니스가 정작 그 수치를 재현하거나 갱신할 수 없었다. + +## 2. 기술적 선택과 그 이유 + +### 2.1 Neural Accelerator 대신 GPU 세대로 판별 + +`AppleSiliconGen::wide_quantized_projections`는 MLX의 `use_qmv_wide` 조건에서 칩에만 의존하는 부분을 추려 담는다. 15세대부터는 affine 양자화 프로젝션이 `M >= 2`에서 한 번의 넓은 패스로 처리되고, 13세대에서는 검증 블록이 `K`개의 좁은 패스로 나뉘어 블록이 커질수록 비용이 함께 늘어난다. 이미 문서화된 라운드 비용 모델이 호스트 간 순서를 설명할 때 지목한 메커니즘이 바로 이것이므로, 게이트는 이제 상관 지표가 아니라 메커니즘 자체를 읽는다. `Unknown`은 거짓으로 읽는다. K 폭 검증이 아예 상각되지 않는 비 Apple 호스트(#638)와 아직 열거하지 않은 최신 Apple 세대에서 기존의 거부 동작을 그대로 유지하기 위해서다. + +### 2.2 13세대는 계속 거부하되, 그것을 결론으로 만들기 + +폭 스윕 덕분에 보수적인 선택이 단순한 신중함이 아니라 근거 있는 판단이 되었다. 이 페어링의 M3 Ultra 라운드 비용은 `0.83 + 0.170 K` classic step으로 맞춰지고, 같은 호스트의 12B 페어링은 `1.14 + 0.090 K`다. bf16 드래프터는 블록 위치 하나당 약 1.9배를 더 쓰며, 두 직선은 K = 4에서 교차한다. M1 Ultra의 공개된 블록 4 라운드 비용 2.71을 페어링만 바꿔 그대로 가져다 쓰면 13세대도 이제 손익분기를 넘는다는 결론이 나오지만, 기울기 비율을 반영하면 그곳의 블록 4 라운드 비용은 3.6 classic step 근처가 된다. 라운드가 내보내는 토큰 수로는 겨우 상쇄될 수준이고, 이는 최초 측정의 성능 저하와 부합한다. + +### 2.3 안전성 논거를 주장이 아니라 테스트로 + +새 조건식은 교체 대상보다 엄격하게 더 관대하므로, 기존에 열려 있던 경로를 잃는 호스트는 없다. 한 호스트의 근거만으로 게이트를 바꾸어도 되는 이유가 바로 이 성질이다. 그래서 이 함의를 산문으로 남기지 않고, 열거된 모든 세대에 대해 단위 테스트로 검증한다. + +## 3. 변경 요약 + +| 영역 | 변경 내용 | +|---|---| +| `src/lib/mlxcel-core/src/hardware.rs` | `AppleSiliconGen::wide_quantized_projections` 조건식 추가와 단위 테스트 2건(세대 분기, NA보다 약한 조건이라는 함의). | +| `src/server/batch/speculative_burst.rs` | `mtp_b1_default`의 세 번째 인자를 `wide_quantized_projections`로 변경, `mtp_b1_burst_enabled`가 `silicon_gen`에서 값을 읽도록 수정, 새 측정값과 근거의 두 가지 한계를 명시하도록 문서 주석 재작성, 단위 테스트 2건 추가. | +| `src/server/batch/mtp_policy.rs` | 필드 이름과 출처 변경. 적응형 정책이 프로파일링 결과가 모호할 때 쓰는 기본값도 같은 조건식을 따른다. | +| `src/server/batch/scheduler.rs` | 거부 경로 주석을 새 정책에 맞게 갱신. | +| `scripts/bench_speculative.sh`, `scripts/bench_block_width.sh` | 두 스크립트에 `gemma31b` 항목 추가. 게이트가 관장하는 페어링에 프로토콜이 닿을 수 있게 되었다. | +| `docs/benchmarks.md`, `docs/environment-variables.md` | 31B 절을 측정값으로 재작성. 이 페어링의 속도 향상이 B>1 윈도에서 나오며 단일 스트림 수용률이 너무 낮다는 서술은 사실과 달라 삭제. | +| `docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md` | 날짜가 붙은 새 기록. 환경, 세 프롬프트 표, 폭 스윕, 라운드 비용 적합, 실제 디스패치 경로 검증, 측정하지 못한 항목의 명시. | + +## 4. 리뷰 발견 사항 + +작업 도중 두 가지를 바로잡았다. + +첫째는 폭 스윕이 잡아낸 분석 오류다. 문서 주석과 환경 변수 문서의 초고는 라운드 비용 모델이 #1203 이후의 M1 Ultra가 이제 손익분기를 넘을 것으로 예측한다고 단언했다. 이 추론은 드래프터 dtype이 다른 두 페어링 사이에서 라운드 비용을 그대로 옮긴 것인데, 스윕은 두 페어링의 기울기가 1.9배 차이 난다는 사실을 보여주었다. 두 서술 모두 커밋 전에 수정했고, 측정 기록에는 교차점이 아닌 폭에서 두 페어링의 라운드 비용을 서로 옮겨 쓰지 말라는 경고를 명시했다. + +둘째는 PR 본문의 결함이다. 이 PR이 #1217을 닫지 않는다고 적은 문장 자체에 `close #1217`이라는 문자열이 들어 있었고, GitHub의 종료 키워드 파서는 부정 표현을 고려하지 않고 이를 인식했다. 열어 두려고 쓴 문장 때문에 PR이 이슈를 자동으로 닫도록 등록된 것이다. 본문을 고쳐 쓰고 `closingIssuesReferences`가 비어 있음을 다시 확인했다. + +## 5. 검증 + +`cargo fmt --all --check` 통과. `cargo clippy --profile test-fast --features metal,accelerate --all-targets` 경고 없음. `server::batch` 단위 테스트 361건과 새로 추가한 `hardware`, `mtp_b1_default` 테스트가 모두 통과한다. + +순수 함수 테스트 경계를 넘어, 측정 호스트에서 실제 체크포인트로 `Scheduler::mtp_b1_should_run` 경로를 직접 통과시켰다. 정적 게이트가 판단하도록 `MLXCEL_MTP_ADAPTIVE=0`을 설정한 상태에서, `9e2c6675` 시점의 `main`은 버스트를 거부했고, 이 브랜치는 버스트를 실행했으며(블록 4, 21라운드에 80토큰, 수용률 0.921), `MLXCEL_ENABLE_MTP_B1=0`은 여전히 거부했다. 한 호스트와 한 쌍의 체크포인트에서 확인한 전후 비교이며, 완료 조건이 요구한 범위가 정확히 거기까지다. + +검증 과정에서 함께 기록한 사실이 하나 있다. 오프라인 `mlxcel generate` 경로는 이 게이트를 전혀 참조하지 않는다. `mtp_b1_default`의 호출자는 하나뿐이고 `MtpPolicy`는 서버 워커에서만 생성되므로, 벤치 하니스는 모든 호스트에서 버스트를 무조건 실행한다. 게이트를 판단하는 계측 도구로 이 하니스가 적절한 이유이자, 이슈가 지정한 두 환경 변수가 그곳에서는 아무 효과가 없는 이유이기도 하다. + +## 6. 관련 작업 + +이슈 #1217은 의도적으로 열린 상태로 둔다. M1 Ultra 측정과 M5 Max 재측정은 확보하지 못한 하드웨어가 필요하고, 함께 요구된 M3 Ultra Qwen 측정값은 이미 #1215에서 같은 프로토콜로 현재 main에 기록되어 있다. 선행 작업: #165(최초 게이트), #333(적응형 정책, 프로파일링이 모호할 때 이 정적 기본값으로 되돌아온다), #1203(로드 시점 드래프터 프로젝션 양자화), #1215(측정 프로토콜과 그 보호 장치). diff --git a/docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md b/docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md new file mode 100644 index 000000000..501274a57 --- /dev/null +++ b/docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md @@ -0,0 +1,189 @@ +# B=1 MTP gate re-measurement on M3 Ultra, 2026-08-20 + +The static per-hardware gate for the singleton MTP burst, `mtp_b1_default` in +`src/server/batch/speculative_burst.rs`, ran batch-capable targets only where +`has_neural_accelerator` held, which is M5 and nothing else. That policy rested +on two numbers taken in the founding measurement (#165): about 1.2 to 1.4x on +M5 Max, against a 0.75 to 0.96x regression on M1 Ultra. Both predate #1194, +#1199, #1203, #1208 and #1215, and M3 Ultra had never been run on the pairing +at all. Issue #1217 asked what the numbers say now. + +Headline: the gate was wrong about M3 Ultra by a wide margin. The batch-capable +Gemma 4 31B target with its bf16 assistant measures **1.95x to 2.65x** there, on +a host the gate declined, and it beats the M5 Max figure the gate was built to +enable. A verify round costs 1.51 classic decode steps on this host and the +rounds emit 2.96 to 3.99 tokens, so the pairing clears break-even by roughly +double on every prompt. + +The discriminator is not the Neural Accelerator. It is the `use_qmv_wide` split +documented in `src/models/speculative_exactness.rs`: from Apple GPU generation +15 an affine-quantized projection at `M >= 2` runs as one wide pass, and below +it the verify block runs as `K` narrow passes whose cost grows with the block. +M3 Ultra is generation 15 and the old predicate grouped it with generation 13 +anyway. + +## Environment + +| Field | Value | +|---|---| +| Host | Mac Studio, Apple M3 Ultra, 512 GB unified memory, macOS 26.6.1 (25G76) | +| Build | `cargo build --release --features metal,accelerate` | +| Branch | `update/issue-1217-mtp-b1-gate` at `9e2c6675` (`main` tip, unmodified for the measurement) | +| Harness | `scripts/bench_speculative.sh gemma31b`, six samples per arm, ABBA blocks, two warm-ups discarded | +| Sweep | `scripts/bench_block_width.sh gemma31b`, 8 interleaved rounds, rotating start width | +| Wrapper | `scripts/with_indexers_paused.sh` (17 indexers suspended for the duration) | +| Target | `models/gemma-4-31b-it-4bit` (`model_type: gemma4`, `supports_batching() == true`) | +| Drafter | `models/gemma-4-31b-it-assistant-bf16` | +| Sampling | `--temp 0` | + +Neither pairing existed in the bench scripts before this run. `bench_speculative.sh` +covered the 12B and Qwen pairings only, and `bench_block_width.sh` the same two, +so the pairing the gate actually governs had no path through the #1215 protocol. +Both scripts gained a `gemma31b` case as part of this work; that absence is most +of why the founding numbers went stale without anyone noticing. + +## Results + +Gemma 4 31B + bf16 assistant, block 4, greedy. Round cost is the identity +`emitted per verify / speedup`, in units of this host's own classic decode step, +which is the same quantity the cross-host table in `docs/benchmarks.md` uses. + +| Output | Tokens | Block | acceptance | emitted/verify | classic | MTP | speedup | round cost | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| enumeration | 400 | 4 | 1.000 | 3.990 | 31.5 | 83.6 | **2.65x** | 1.51 | +| source code | 300 | 4 | 0.882 | 3.646 | 31.8 | 76.6 | **2.41x** | 1.51 | +| prose | 400 | 4 | 0.656 | 2.956 | 31.7 | 61.9 | **1.95x** | 1.52 | + +Spreads were 0.5% to 1.0% on both arms of all three rows, against the harness +limit of 4%. The contention watch flagged brief spikes in a handful of samples +and nothing sustained enough to report. + +Three prompts with nothing in common agree on the round cost to within 0.7%, +which is the control that this is a property of the host and the block width +rather than of the prompt. It also lands on the 1.50 to 1.51 that the 12B +pairing measures at block 4 on this same host. Read that coincidence carefully: +the width sweep below shows the two pairings have different slopes and merely +cross near block 4, so it is not evidence that the drafter's dtype stopped +mattering. + +## Block-width sweep + +Source-code prompt, 300 tokens, 8 interleaved rounds with a rotating start +width, against the same classic arm of 31.76 tok/s. + +| width | decode tok/s | spread | acceptance | emitted per verify | speedup | round cost | vs peak | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 2 | 53.3 | 0.7% | 0.942 | 1.942 | 1.68x | 1.16 | -31.2% | +| 3 | 67.3 | 1.2% | 0.897 | 2.794 | 2.12x | 1.32 | -13.0% | +| 4 | 76.6 | 0.9% | 0.882 | 3.646 | 2.41x | 1.51 | -1.0% | +| 5 | 77.4 | 1.7% | 0.873 | 4.096 | 2.44x | 1.68 | **peak** | +| 6 | 75.3 | 1.1% | 0.870 | 4.530 | 2.37x | 1.91 | -2.7% | +| 8 | 73.8 | 0.5% | 0.800 | 4.983 | 2.32x | 2.14 | -4.6% | + +The peak is at 5 rather than the drafter's declared 4, but only by 1.0% against +a 1.7% spread on that row, so the two are tied and the declared width stands as +a default. Everything from 3 to 8 gains by more than a factor of two, so no +width in this range is a bad choice; only width 2 gives up real throughput, and +it does so by starving the round of positions to amortise over rather than by +losing acceptance, which is highest there. + +Acceptance is nearly flat from width 2 to 6 (0.942 down to 0.870) and only +falls off at 8. That is the bf16 assistant staying accurate deep into a block, +and it is why emitted per verify keeps climbing across the whole table while +throughput turns over at 5: the round is emitting more each time, and past 5 the +verify is simply costing more than the extra emission is worth. + +Fitting `cost = a + b K` over the six widths gives: + +``` +round cost = 0.83 + 0.170 K classic steps (largest residual 0.064) +``` + +Beside the two published fits this is the useful comparison, and it corrects a +reading the three-prompt table on its own invites: + +| Pairing | Host | fit | cost at K=4 | +| --- | --- | --- | ---: | +| Gemma 4 31B + **bf16** assistant | M3 Ultra | `0.83 + 0.170 K` | 1.51 | +| Gemma 4 12B + 4-bit assistant | M3 Ultra | `1.14 + 0.090 K` | 1.50 | +| Gemma 4 12B + 4-bit assistant | M1 Ultra | `1.35 + 0.346 K` | 2.73 | + +The 31B and 12B pairings on this host agree at block 4 to within 0.01 classic +steps, which looks like the drafter's dtype not mattering. It is not: the +per-position slope is 1.9 times steeper for the bf16 drafter, and the two lines +merely cross near K = 4. Either side of that they separate, the 31B pairing +being cheaper per round below the crossing and dearer above it. Anyone +transferring a round cost between these two pairings at some other width will +be wrong in a direction that depends on which side they are on. + +## What this does and does not settle + +It settles the batch-capable gate for generation 15. The pairing gains 1.95x to +2.65x on M3 Ultra with every row inside a 1.0% spread, so `mtp_b1_default` now +reads `wide_quantized_projections` (Apple GPU generation 15 and up) in place of +`has_neural_accelerator` (M5 only). M4 is grouped with M3 by the shared +`use_qmv_wide` dispatch rather than by measurement, which is an inference and is +labelled as one in the code. + +It does not settle generation 13, and the sweep is what makes that worth saying +carefully. The naive move is to take M1 Ultra's published block-4 round cost of +2.71, note that this pairing emits 2.96 to 3.99 tokens per verify, and conclude +that post-#1203 M1 Ultra would now clear break-even. The sweep shows why that is +not sound: 2.71 belongs to the 12B pairing with a 4-bit drafter, and this +pairing's slope is 1.9 times steeper. Carrying that ratio onto generation 13's +`1.35 + 0.346 K` puts a block-4 round near 3.6 classic steps there, which 2.96 +to 3.99 emitted tokens would only just cover, and which is consistent with the +0.75 to 0.96x the founding measurement recorded. So the founding decline for +generation 13 reads as sound rather than merely stale, and it stays. + +That estimate extrapolates across a pairing and a generation at once, so it is +a reason to leave generation 13 alone, not a result. Measuring it needs an +M1 Ultra host with these two checkpoints, which is tracked separately; the +prediction above is stated in a falsifiable form so that run can settle it. + +## Gate exercised through the real dispatch path + +Unit tests cover `mtp_b1_default` as a pure function, which is not the same as +showing that the running scheduler consults it. This is the same host, the same +two checkpoints, and the server rather than the offline CLI, with +`MLXCEL_MTP_ADAPTIVE=0` so the static gate is what decides and no +`MLXCEL_ENABLE_MTP_B1` unless stated. One `/v1/completions` request each. + +| Binary | `MLXCEL_ENABLE_MTP_B1` | Scheduler outcome | +|---|---|---| +| `main` at `9e2c6675`, before the change | unset | declined, fell back to classic decode | +| this branch | unset | ran the B=1 burst | +| this branch | `0` | declined, fell back to classic decode | + +The declines are the scheduler's own log line, `MTP B=1 speculative burst +declined for seq seq-0 ... falling back to classic decode`. The run that +proceeded reports block 4 at effective block 4, 80 tokens over 21 rounds, +acceptance 0.921, 3.762 tokens emitted per verify, and no decline line anywhere +in its log. + +That is the before-and-after the gate change is for: an M3 Ultra that declined +this pairing now runs it, and the override still pins the decision in both +directions. + +The offline `mlxcel generate` path does **not** exercise this gate, which is +worth recording because it changes how the throughput rows above should be +read. `mtp_b1_default` has exactly one caller, `Scheduler::mtp_b1_should_run`, +and `MtpPolicy` is built only in the server worker; the offline path gates on +exactness alone (`run_offline_mtp`). So the bench harness runs the burst +unconditionally on every host, which is what makes it the right instrument for +deciding the gate, and it is also why `MLXCEL_ENABLE_MTP_B1=1` and +`MLXCEL_MTP_ADAPTIVE=0` are inert there. They are set in the harness so the arm +matches what a server runs with the gate forced on, not because the harness +needs them. + +## Not measured here + +- **M1 Ultra, this pairing.** No generation-13 host was available. Nothing in + this record changes M1 or M2 behaviour. +- **M5 Max, re-measured.** Its ~1.2 to 1.4x is the founding figure and is also + pre-#1203. M5 keeps the path it already had either way, so the gate does not + depend on it, but the number in the table is old and the M3 Ultra rows above + now exceed it, which is itself a reason to re-run it. +- **Qwen MTP pairing on M3 Ultra.** Already measured on current main under this + protocol in #1215 and recorded in `docs/benchmarks.md` (1.67x at block 3, full + width sweep). Not re-run here. diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 2c7c800f5..09556db23 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -641,8 +641,10 @@ on either. A declining probe falls back to classic decode unless `MLXCEL_MTP_ALLOW_INEXACT=1` is set. B=1 (single-request) MTP runs by default for every MTP target; the Gemma 4 Unified target cannot batch at all, so B=1 is also its only decode path. The batch-capable 31B + bf16 -assistant measures ~1.2 to 1.4x on M5 Max. Set `MLXCEL_ENABLE_MTP_B1=0` to -opt out on hardware where the B=1 verify forward does not pay for itself. +assistant measures ~1.2 to 1.4x on M5 Max and 1.95x to 2.65x on M3 Ultra (see +below); it runs by default from Apple GPU generation 15 since #1217. Set +`MLXCEL_ENABLE_MTP_B1=0` to opt out on hardware where the B=1 verify forward +does not pay for itself. Gemma 4 is not probed yet (#1188), so the rows above are the fast kernel rather than the byte-identical one. Keeping byte-identity on the code row, by dropping @@ -876,13 +878,46 @@ enabling this pairing on generation 13. ### Gemma 4 31B + bf16 assistant -The 31B text target is batch-capable, and its MTP speedup comes from batched -(B>1) verify windows rather than the singleton path. The scheduler declines B=1 -MTP there because the bf16 assistant's single-stream acceptance is too low to -offset the extra drafter forward per token. This pairing is wired into -`speculative_bench` (`REACHABLE_PAIRINGS`) and runs once the -`gemma-4-31b-it-4bit` and `gemma-4-31B-it-assistant-bf16` checkpoints are present -in the model store. +The 31B text target is batch-capable, which is the case the B=1 static gate +(`mtp_b1_default`) governs. Until issue #1217 that gate ran the singleton path +only where `has_neural_accelerator` held, on the reading that this pairing's +speedup came from batched (B>1) verify windows and that the bf16 assistant's +single-stream acceptance was too low to offset its extra drafter forward. Both +halves of that reading were measured before #1194, #1199, #1203, #1208 and +#1215, and neither survived re-measurement. + +M3 Ultra, 2026-08-20, block 4, greedy, under the protocol above +(`scripts/bench_speculative.sh gemma31b`): + +| Host | Output | Tokens | Block | acceptance | emitted/verify | classic | MTP | speedup | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| M3 Ultra (512 GB) | enumeration | 400 | 4 | 1.000 | 3.990 | 31.5 | 83.6 | **2.65x** | +| M3 Ultra (512 GB) | source code | 300 | 4 | 0.882 | 3.646 | 31.8 | 76.6 | **2.41x** | +| M3 Ultra (512 GB) | prose | 400 | 4 | 0.656 | 2.956 | 31.7 | 61.9 | **1.95x** | + +Single-stream acceptance is 0.66 to 1.00, not too low, and the singleton path +gains on every prompt. All three rows measure a verify round at 1.51 to 1.52 +classic decode steps and emit 2.96 to 3.99 tokens, so they clear break-even by +roughly double. The width sweep fits `0.83 + 0.170 K` classic steps, largest +residual 0.06, and peaks at width 5 with width 4 tied inside its spread. + +Beside the 12B pairing's `1.14 + 0.090 K` on the same host, the bf16 drafter +costs about 1.9x as much per extra block position and the two lines cross near +K = 4, which is the only reason the block-4 round costs match. Do not carry a +round cost between these two pairings at any other width. + +The gate now reads Apple GPU generation instead: on from generation 15 (M3, M4, +M5), classic decode on generation 13 (M1, M2). M4 is grouped by the shared +`use_qmv_wide` dispatch rather than measured. Generation 13 was not re-measured +for want of a host, and carrying the slope ratio above onto its +`1.35 + 0.346 K` puts a block-4 round near 3.6 classic steps, which the emitted +tokens would only just cover, so its founding 0.75 to 0.96x reads as sound +rather than stale and it keeps declining. Full record and method: +`benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md`. + +The pairing is also wired into `speculative_bench` (`REACHABLE_PAIRINGS`), which +runs once the `gemma-4-31b-it-4bit` and `gemma-4-31B-it-assistant-bf16` +checkpoints are present in the model store. ### Adaptive B=1 MTP policy diff --git a/docs/environment-variables.md b/docs/environment-variables.md index ad9326c38..63f99d2eb 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -181,7 +181,7 @@ The OpenAI audio endpoints (`/v1/audio/speech`, `/v1/audio/transcriptions`, `/v1 | `MLXCEL_DRAFT_KIND` | `dflash`, `mtp` | auto/none | Alias for `--draft-kind` when the CLI flag and `LLAMA_ARG_DRAFT_KIND` are absent. | | `MLXCEL_DRAFT_BLOCK_SIZE` | unsigned integer | per drafter (`4` for MTP, `16` for DFlash) | Alias for `--draft-block-size` when the CLI flag and `LLAMA_ARG_DRAFT_BLOCK_SIZE` are absent. | | `MLXCEL_MTP_ADAPTIVE` | `0`/`false`/`no`/`off` to disable, any other value (or unset) to enable | on | Adaptive B=1 MTP policy (issue #333). When on, the server profiles the first few B=1 MTP bursts of each (target, drafter, hardware, block_size) pairing (acceptance length, verify latency, drafter latency, batch size, prompt shape) and settles to a data-driven enable/decline verdict that overrides the static per-hardware gate when the measured profile is clearly favorable or unfavorable, falling back to the static default otherwise. The verdict (enable/decline plus the coarse acceptance rate, no prompt data) is persisted at `${MLXCEL_CACHE_DIR:-$HOME/.cache/mlxcel}/mtp-policy/.json` (hint format v3; v2 hints were settled against the pre-#725 verify kernel and the pre-#736 estimator, so they are ignored and the pairing re-profiles once), so profiling runs once per pairing and a restart reuses the verdict. Changing `MLXCEL_DRAFT_BLOCK_SIZE` changes the block_size dimension of the key, so the old hint is discarded and profiling restarts for the new K. Set to an off value to disable profiling and use the pre-#333 static per-hardware gates. `MLXCEL_ENABLE_MTP_B1` still pins the decision and, when set, suppresses profiling. The experimental batched (B>1) path is unaffected and stays behind `MLXCEL_ENABLE_MTP_BATCH`. The speedup estimate is measured, not modeled (issue #736): while profiling, each burst runs a couple of classic-step probe rounds (drafterless rounds whose `[1, 1]` verify forward is shape-identical to a classic decode step; each emits one real greedy token, so nothing is wasted and temperature-0 output stays byte-identical), and the estimator compares the measured speculative round cost (verify + drafter + walk/finalize overhead) against the measured classic step time, which is taken as the median (not the mean) of the per-burst probe means so the first burst's one-time CUDA kernel/graph compilation for the `[1, 1]` verify shape cannot skew the estimate toward a falsely slow classic step. This makes the verdict correct across backends and kernel eras without hardware heuristics: on GB10 with the multirow qmv verify (`MLXCEL_QMV_MULTIROW`, issue #725) the Gemma 4 12B pairing profiles to about 1.5× and enables, while the same pairing on the pre-#725 per-row verify profiles to about 0.5× and declines with margin. When a window collects no probe signal the pre-#736 shape heuristic (issue #638: 1.0 on Apple Silicon, `sqrt(K)` elsewhere) remains as the fallback. | -| `MLXCEL_ENABLE_MTP_B1` | `0`/`false`/`no`/`off` to disable, any other value to force on | adaptive (per hardware) | Manual override for the singleton (B=1) MTP burst, in both directions. When set it pins the decision and disables adaptive profiling (issue #333). When unset, the adaptive policy decides (see `MLXCEL_MTP_ADAPTIVE`); with `MLXCEL_MTP_ADAPTIVE=0` the decision is the static per-hardware default (issue #165): non-batchable targets (`gemma4_unified` 12B pairs, whose only decode path is B=1) default **on** everywhere (measured across three prompts: 1.90x to 3.14x on M5 Max, 1.74x to 2.61x on M3 Ultra, 0.95x to 1.48x on M1 Ultra, where the prose end is a 5% loss rather than a gain); batch-capable targets (the 31B + bf16 assistant) default **on only on M5+** (Neural Accelerator generation): M5 Max measured ~1.2 to 1.4×, while M1 Ultra measured a consistent ~0.75 to 0.96× regression, so pre-M5 chips fall back to classic decode. | +| `MLXCEL_ENABLE_MTP_B1` | `0`/`false`/`no`/`off` to disable, any other value to force on | adaptive (per hardware) | Manual override for the singleton (B=1) MTP burst, in both directions. When set it pins the decision and disables adaptive profiling (issue #333). When unset, the adaptive policy decides (see `MLXCEL_MTP_ADAPTIVE`); with `MLXCEL_MTP_ADAPTIVE=0` the decision is the static per-hardware default (issue #165): non-batchable targets (`gemma4_unified` 12B pairs, whose only decode path is B=1) default **on** everywhere (measured across three prompts: 1.90x to 3.14x on M5 Max, 1.74x to 2.61x on M3 Ultra, 0.95x to 1.48x on M1 Ultra, where the prose end is a 5% loss rather than a gain); batch-capable targets (the 31B + bf16 assistant) default **on from Apple GPU generation 15** (M3, M4, M5), and fall back to classic decode on generation 13 (M1, M2). This was `has_neural_accelerator` (M5 only) until issue #1217, on the strength of ~1.2 to 1.4× on M5 Max against a ~0.75 to 0.96× regression on M1 Ultra, both measured before #1194/#1199/#1203/#1208/#1215 and neither re-measured since. M3 Ultra, which the old predicate lumped with M1 Ultra, was never measured on this pairing at all until #1217 did it: 1.95× (prose), 2.41× (source code) and 2.65× (enumeration) on 2026-08-20 under the #1215 protocol, at a verify-round cost of 1.51 classic steps against 2.71 on M1 Ultra. The discriminator is the `use_qmv_wide` split (an affine-quantized projection at `M >= 2` runs as one wide pass from generation 15 and as `K` narrow passes below it), not the Neural Accelerator. M4 is grouped by that shared dispatch rather than measured. Generation 13 keeps declining and the width sweep supports that: round cost fits `0.83 + 0.170 K` classic steps for this pairing on M3 Ultra against `1.14 + 0.090 K` for the 12B pairing there, so the bf16 drafter costs about 1.9x as much per block position, and carrying that ratio onto generation 13's `1.35 + 0.346 K` puts a block-4 round near 3.6 classic steps, which 2.96 to 3.99 emitted tokens would only just cover. See `docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md`. | | `MLXCEL_ENABLE_MTP_BATCH` | truthy value | off | **Advanced.** Forces the batched Gemma 4 MTP burst path for parity/debug testing. Not governed by the adaptive policy (issue #333), which scopes to the validated, byte-identical B=1 path. | | `MLXCEL_ENABLE_MTP_DEFERRED` | `1` | off | **Advanced.** Enables the deferred greedy verifier path for Gemma 4 MTP when sampling settings allow it. | | `MLXCEL_METAL4_ATTENTION` | `0`/`false`/`no`/`off` to disable; unset or any other value to enable | on (where the hardware has it) | **Advanced, diagnostic kill switch.** Forces the M5 neural-accelerator fused attention route off on hardware that has it, so `layers::metal4_causal_attention` is skipped and the ordinary SDPA path runs instead. Off-switch only: it cannot turn the route on where `has_neural_accelerator && macos_supports_na` is false, so setting it on an M1 is inert. This route is the first suspect whenever an M5 disagrees numerically with an earlier Apple GPU generation (issue #1065), and before this switch the only way to A/B the hypothesis was to patch `should_use_metal4_attention` and rebuild, which is what the #1182 M5 investigation had to do. Read once per process, so set it before starting `mlxcel` or `mlxcel-server`. Inert on non-Metal builds. | diff --git a/scripts/bench_block_width.sh b/scripts/bench_block_width.sh index d94e6ae5b..5625478e8 100755 --- a/scripts/bench_block_width.sh +++ b/scripts/bench_block_width.sh @@ -4,6 +4,7 @@ # Usage: # ./scripts/bench_block_width.sh qwen # widths 2 3 4 5 6 8 # ./scripts/bench_block_width.sh gemma # widths 3 4 5 6 8 10 12 +# ./scripts/bench_block_width.sh gemma31b # widths 2 3 4 5 6 8 # ./scripts/bench_block_width.sh gemma 4 5 6 # explicit widths # # Run it through scripts/with_indexers_paused.sh, the same way the throughput @@ -58,8 +59,15 @@ case "${1:-}" in gemma) TARGET=models/gemma-4-12b-it-4bit; DRAFTER=models/gemma-4-12b-it-assistant-4bit DEFAULT_WIDTHS=(3 4 5 6 8 10 12) ;; + gemma31b) + # The batch-capable pairing the B=1 static gate governs (issue #1217). Its + # declared drafter width is 4; the sweep brackets it the way the qwen one + # does rather than reaching for the 12B pairing's wide tail, because a bf16 + # drafter costs far more per position than a 4-bit one. + TARGET=models/gemma-4-31b-it-4bit; DRAFTER=models/gemma-4-31b-it-assistant-bf16 + DEFAULT_WIDTHS=(2 3 4 5 6 8) ;; *) - sed -n '2,9p' "$0"; exit 1 ;; + sed -n '2,10p' "$0"; exit 1 ;; esac shift WIDTHS=("$@"); [ ${#WIDTHS[@]} -eq 0 ] && WIDTHS=("${DEFAULT_WIDTHS[@]}") diff --git a/scripts/bench_speculative.sh b/scripts/bench_speculative.sh index f695c99bc..4b118cb66 100755 --- a/scripts/bench_speculative.sh +++ b/scripts/bench_speculative.sh @@ -5,6 +5,7 @@ # Usage: # ./scripts/bench_speculative.sh # every pairing it can find # ./scripts/bench_speculative.sh gemma # one pairing +# ./scripts/bench_speculative.sh gemma31b # the batch-capable pairing # ./scripts/bench_speculative.sh --reps 4 # more samples per arm # ./scripts/bench_speculative.sh --no-wait # do not wait for a quiet host # @@ -53,7 +54,7 @@ while [ $# -gt 0 ]; do --reps) REPS="$2"; shift 2 ;; --no-wait) WAIT_FOR_QUIET=0; shift ;; --spread-limit) SPREAD_LIMIT="$2"; shift 2 ;; - -h|--help) sed -n '2,12p' "$0"; exit 0 ;; + -h|--help) sed -n '2,13p' "$0"; exit 0 ;; *) ONLY="$1"; shift ;; esac done @@ -184,6 +185,31 @@ if [ -z "$ONLY" ] || [ "$ONLY" = "gemma" ]; then "enumeration" "$PROMPT_LIST" 400 fi +# The batch-capable pairing, which is the one the B=1 static gate governs +# (`mtp_b1_default` in src/server/batch/speculative_burst.rs). It was missing +# here until issue #1217, so the gate's founding numbers were never reproducible +# through this protocol; that is most of why they went stale unnoticed. +# +# The two env vars are set for the server's benefit, not this script's: the +# offline `generate` path this harness drives never consults the B=1 gate or +# the adaptive policy (they live in `Scheduler::mtp_b1_should_run` and +# `MtpPolicy`, both server-only), so the burst runs here regardless. Setting +# them makes the arm the same one a server would run with the gate forced on, +# and keeps the command copy-pasteable into a server session. +if [ -z "$ONLY" ] || [ "$ONLY" = "gemma31b" ]; then + export MLXCEL_ENABLE_MTP_B1=1 MLXCEL_MTP_ADAPTIVE=0 + measure_pairing "Gemma 4 31B + bf16 assistant" \ + models/gemma-4-31b-it-4bit models/gemma-4-31b-it-assistant-bf16 4 \ + "source code" "$PROMPT_CODE" 300 + measure_pairing "Gemma 4 31B + bf16 assistant" \ + models/gemma-4-31b-it-4bit models/gemma-4-31b-it-assistant-bf16 4 \ + "prose" "$PROMPT_PROSE" 400 + measure_pairing "Gemma 4 31B + bf16 assistant" \ + models/gemma-4-31b-it-4bit models/gemma-4-31b-it-assistant-bf16 4 \ + "enumeration" "$PROMPT_LIST" 400 + unset MLXCEL_ENABLE_MTP_B1 MLXCEL_MTP_ADAPTIVE +fi + if [ -z "$ONLY" ] || [ "$ONLY" = "qwen" ]; then measure_pairing "Qwen 3.8 27B + its 4-bit MTP head" \ models/qwen3.8-27b-4bit models/qwen3.8-27b-mtp-4bit 3 \ diff --git a/src/lib/mlxcel-core/src/hardware.rs b/src/lib/mlxcel-core/src/hardware.rs index 4f8eb2e18..39f195ea4 100644 --- a/src/lib/mlxcel-core/src/hardware.rs +++ b/src/lib/mlxcel-core/src/hardware.rs @@ -41,6 +41,35 @@ impl AppleSiliconGen { matches!(self, AppleSiliconGen::M5) } + /// Whether this generation runs an affine-quantized projection at `M >= 2` + /// as one wide pass rather than as narrow per-position passes. + /// + /// This is MLX's `use_qmv_wide` predicate (`mode != "affine" || arch_gen + /// >= 15` in `mlx/backend/metal/quantized.cpp`) reduced to the part that + /// depends only on the chip: Apple GPU generation 15 and newer, which is + /// M3, M4 and M5. Generation 13 (M1, M2) has no such path and runs a + /// verify block as `K` narrow passes whose cost grows with the block. + /// + /// That difference is what decides whether a speculative verify round pays + /// for itself, so it is the discriminator behind the B=1 MTP gate + /// (`mtp_b1_default` in the `mlxcel` crate); see `docs/benchmarks.md` for + /// the round-cost measurements that order the generations by it. + /// + /// `Unknown` reads false, which is deliberate in both directions it + /// covers: a non-Apple GPU (CUDA / GB10), where a K-wide verify does not + /// amortize at all (issue #638), and an Apple generation newer than the + /// enumerated ones, which `parse_silicon_gen` also maps to `Unknown`. Both + /// decline rather than assume. Extend the enum when a new Apple generation + /// ships; this carries the same staleness contract as + /// [`Self::has_neural_accelerator`]. + #[inline] + pub fn wide_quantized_projections(self) -> bool { + matches!( + self, + AppleSiliconGen::M3 | AppleSiliconGen::M4 | AppleSiliconGen::M5 + ) + } + /// Returns the expected Metal GPU family version (3 for M1–M4, 4 for M5+). #[inline] pub fn metal_version(self) -> u32 { @@ -790,6 +819,43 @@ mod tests { assert!(!AppleSiliconGen::Unknown.has_neural_accelerator()); } + #[test] + fn wide_quantized_projection_flag_splits_at_generation_15() { + // Generation 13 runs a verify block as narrow per-position passes. + assert!(!AppleSiliconGen::M1.wide_quantized_projections()); + assert!(!AppleSiliconGen::M2.wide_quantized_projections()); + // Generation 15+ takes the one wide pass at `M >= 2`. + assert!(AppleSiliconGen::M3.wide_quantized_projections()); + assert!(AppleSiliconGen::M4.wide_quantized_projections()); + assert!(AppleSiliconGen::M5.wide_quantized_projections()); + // Non-Apple and not-yet-enumerated Apple parts decline rather than + // assume; on CUDA a K-wide verify does not amortize at all (#638). + assert!(!AppleSiliconGen::Unknown.wide_quantized_projections()); + } + + #[test] + fn wide_quantized_projections_is_weaker_than_neural_accelerator() { + // The B=1 MTP gate moved from `has_neural_accelerator` to this + // predicate (issue #1217). That is only safe if it is strictly more + // permissive: no generation may lose a capability the old gate + // granted, so every NA generation must also read wide here. This is + // the property that lets the gate change on M3 Ultra evidence alone + // without touching what M1/M2 hosts do. + for chip in [ + AppleSiliconGen::M1, + AppleSiliconGen::M2, + AppleSiliconGen::M3, + AppleSiliconGen::M4, + AppleSiliconGen::M5, + AppleSiliconGen::Unknown, + ] { + assert!( + !chip.has_neural_accelerator() || chip.wide_quantized_projections(), + "{chip} has a Neural Accelerator but does not read as wide-projection", + ); + } + } + #[test] fn ops_per_buffer_default_gates_on_hardware_class() { // Pre-M5 Apple Silicon (Metal family 3) raises the command-buffer op cap diff --git a/src/server/batch/mtp_policy.rs b/src/server/batch/mtp_policy.rs index 6f02b91d0..695482564 100644 --- a/src/server/batch/mtp_policy.rs +++ b/src/server/batch/mtp_policy.rs @@ -18,8 +18,9 @@ //! //! Before #333 the singleton (B=1) MTP burst was gated by a purely static //! per-hardware rule ([`super::speculative_burst::mtp_b1_default`], issue -//! #165): non-batchable 12B targets default on everywhere, batch-capable 31B -//! targets default on only on M5+. Those static gates are correct for the +//! #165, revised by #1217): non-batchable 12B targets default on everywhere, +//! batch-capable 31B targets default on from Apple GPU generation 15. Those +//! static gates are correct for the //! pairings they were measured on, but they leave performance on the table //! when a new (target, drafter, hardware) pairing is favorable yet the static //! rule declines it, and they keep running MTP on a pairing that turns out @@ -773,7 +774,11 @@ enum PolicyState { pub(crate) struct MtpPolicy { key: PolicyKey, target_supports_batching: bool, - has_neural_accelerator: bool, + /// The host's Apple GPU generation reduced to whether an affine-quantized + /// projection at `M >= 2` runs as one wide pass (generation 15+). Feeds + /// [`super::speculative_burst::mtp_b1_default`] when a profiling window + /// comes out ambiguous; see that function for the measurements behind it. + wide_quantized_projections: bool, /// True on compute-bound (non-Apple-Silicon, e.g. CUDA / GB10) hardware, /// where a K-wide verify forward does not amortize to one classic decode /// forward. Drives the backend-specific verify-cost multiple (issue #638). @@ -806,7 +811,7 @@ impl MtpPolicy { } let key = PolicyKey::new(target_id, drafter_id, hardware_label(), block_size); let hw = mlxcel_core::hardware::get_hardware(); - let has_neural_accelerator = hw.has_neural_accelerator; + let wide_quantized_projections = hw.silicon_gen.wide_quantized_projections(); // Compute-bound = non-Apple-Silicon (CUDA / GB10): the runtime hardware // probe reports `AppleSiliconGen::Unknown` off Apple GPUs. On such hosts // the K-wide verify does not amortize (issue #638), so the policy @@ -814,7 +819,7 @@ impl MtpPolicy { // also maps Apple generations newer than the enumerated ones to // `Unknown`, so the "Apple byte-identical" guarantee is scoped to the // enumerated gens; extend the enum when a new Apple generation ships - // (same staleness contract as `has_neural_accelerator`). + // (same staleness contract as `wide_quantized_projections`). let compute_bound = matches!( hw.silicon_gen, mlxcel_core::hardware::AppleSiliconGen::Unknown @@ -824,7 +829,7 @@ impl MtpPolicy { Some(Self::from_parts( key, target_supports_batching, - has_neural_accelerator, + wide_quantized_projections, compute_bound, force, store, @@ -838,7 +843,7 @@ impl MtpPolicy { pub(crate) fn from_parts( key: PolicyKey, target_supports_batching: bool, - has_neural_accelerator: bool, + wide_quantized_projections: bool, compute_bound: bool, force: Option, store: PolicyStore, @@ -859,7 +864,7 @@ impl MtpPolicy { Self { key, target_supports_batching, - has_neural_accelerator, + wide_quantized_projections, compute_bound, state, store, @@ -903,7 +908,7 @@ impl MtpPolicy { super::speculative_burst::mtp_b1_default( None, self.target_supports_batching, - self.has_neural_accelerator, + self.wide_quantized_projections, ) } diff --git a/src/server/batch/mtp_policy_tests.rs b/src/server/batch/mtp_policy_tests.rs index 8aef93da5..670abf580 100644 --- a/src/server/batch/mtp_policy_tests.rs +++ b/src/server/batch/mtp_policy_tests.rs @@ -298,22 +298,24 @@ fn drive(mut policy: MtpPolicy, s: MtpBurstProfile, n: usize) -> MtpPolicy { policy } -fn adaptive_policy(static_default_batching: bool, has_na: bool) -> MtpPolicy { +fn adaptive_policy(static_default_batching: bool, wide_projections: bool) -> MtpPolicy { // force = None → adaptive; no-dir store keeps these state-machine tests // off the filesystem (persistence is exercised separately). Bandwidth-bound // (Apple) hardware unless the test opts into the compute-bound path. - adaptive_policy_hw(static_default_batching, has_na, false) + // `wide_projections` is the generation-15 split the static default reads + // (issue #1217): false models an M1/M2-class host, true an M3/M4/M5 one. + adaptive_policy_hw(static_default_batching, wide_projections, false) } fn adaptive_policy_hw( static_default_batching: bool, - has_na: bool, + wide_projections: bool, compute_bound: bool, ) -> MtpPolicy { MtpPolicy::from_parts( key(), static_default_batching, - has_na, + wide_projections, compute_bound, None, PolicyStore::with_dir(None), @@ -322,7 +324,8 @@ fn adaptive_policy_hw( #[test] fn favorable_profile_enables_overriding_static_decline() { - // Batch-capable target + no neural accelerator → static default DECLINE. + // Batch-capable target on Apple GPU generation 13 (no wide quantized + // projection path) → static default DECLINE. let policy = adaptive_policy(true, false); assert!( !policy.static_default(), diff --git a/src/server/batch/scheduler.rs b/src/server/batch/scheduler.rs index 4278662e2..333e0aca3 100644 --- a/src/server/batch/scheduler.rs +++ b/src/server/batch/scheduler.rs @@ -4406,11 +4406,14 @@ impl BatchScheduler { // verdict, overriding the static per-hardware gate where the // measured profile is clearly favorable or unfavorable. Without a // policy (MLXCEL_MTP_ADAPTIVE off) this falls back to the static - // per-hardware default (issue #165): non-batchable 12B targets keep - // B=1 MTP on everywhere; batch-capable 31B targets default it on - // only on M5+, since pre-M5 GPU cores measured a consistent - // regression. `MLXCEL_ENABLE_MTP_B1` overrides in both directions; - // on decline the request falls back to classic decode. + // per-hardware default (issue #165, revised by #1217): + // non-batchable 12B targets keep B=1 MTP on everywhere; + // batch-capable 31B targets default it on from Apple GPU + // generation 15, where a quantized projection at `M >= 2` runs as + // one wide pass, and off on generation 13, which runs the verify + // block as narrow per-position passes and measured a regression. + // `MLXCEL_ENABLE_MTP_B1` overrides in both directions; on decline + // the request falls back to classic decode. let seq = window.into_iter().next().expect("singleton window"); tracing::info!( "MTP B=1 speculative burst declined for seq {} (adaptive policy \ diff --git a/src/server/batch/speculative_burst.rs b/src/server/batch/speculative_burst.rs index 40e5652b9..98ce308a0 100644 --- a/src/server/batch/speculative_burst.rs +++ b/src/server/batch/speculative_burst.rs @@ -509,7 +509,7 @@ pub(crate) fn mtp_prefill_suffix_start( /// Whether the Gemma 4 MTP B=1 (single-request) burst path runs for a target /// with the given batching capability. /// -/// Default policy (issue #165, per-hardware): +/// Default policy (issue #165, revised by #1217, per-hardware): /// - Non-batchable targets (the 12B Unified family, whose only decode path is /// B=1): **on** everywhere. Measured across three prompts: 1.90x to 3.14x /// on M5 Max, 1.74x to 2.61x on M3 Ultra, and 0.95x to 1.48x on M1 Ultra. @@ -517,13 +517,50 @@ pub(crate) fn mtp_prefill_suffix_start( /// assumes on every prompt, and it is what the adaptive policy (#333) /// exists to catch. See `docs/benchmarks.md` for the rows and the /// round-cost model that predicts them. -/// - Batch-capable targets (the 31B + bf16 assistant): **on only on M5+** -/// (Neural Accelerator generation). M5 Max measured ~1.2 to 1.4x, but -/// M1 Ultra measured a consistent regression (~0.75 to 0.96x, four greedy -/// 160-token prompts), so pre-M5 chips default to classic decode. The -/// discriminator is GPU compute generation rather than memory bandwidth: -/// M1 Ultra has datacenter-class bandwidth yet the drafter + K-wide verify -/// forwards do not pay for themselves on its older GPU cores. +/// - Batch-capable targets (the 31B + bf16 assistant): **on from Apple GPU +/// generation 15** ([`AppleSiliconGen::wide_quantized_projections`], which +/// is M3, M4 and M5), classic decode below it. +/// +/// The batch-capable half was `has_neural_accelerator` (M5 only) until #1217, +/// on the strength of ~1.2 to 1.4x on M5 Max against a ~0.75 to 0.96x +/// regression on M1 Ultra. Both of those predate #1194, #1199, #1203, #1208 +/// and #1215, and M3 Ultra had never been measured on this pairing at all. It +/// has now: 2026-08-20 on current main, three greedy prompts under the #1215 +/// protocol, **1.95x (prose), 2.41x (source code), 2.65x (enumeration)** on an +/// M3 Ultra that the old predicate declined. The rows are in +/// `docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md`. +/// +/// The discriminator is the `use_qmv_wide` split documented in +/// `crate::models::speculative_exactness`, not the Neural Accelerator: from +/// generation 15 an affine-quantized projection at `M >= 2` runs as one wide +/// pass, while generation 13 runs the verify block as `K` narrow passes. That +/// shows up directly as the cost of a verify round in classic decode steps, +/// which is what a round has to out-emit to pay for itself. All three M3 Ultra +/// rows above measure that round cost at 1.51 to 1.52 and emit 2.96 to 3.99 +/// tokens per verify, so they clear break-even by roughly double; M1 Ultra +/// measures 2.71 at the same block width on the 12B pairing. +/// +/// Two limits on this evidence, both deliberate: +/// - **M4 is inferred, not measured.** It is grouped here because it shares +/// generation 15's `use_qmv_wide` dispatch with M3, not because anyone ran +/// the pairing on one. +/// - **Generation 13 was not re-measured**, for want of the hardware. Its +/// founding regression is stale in the way M5 Max's founding gain was, but +/// the width sweep argues it is not therefore wrong. Round cost fits +/// `0.83 + 0.170 K` classic steps for this pairing on M3 Ultra against +/// `1.14 + 0.090 K` for the 12B pairing on the same host: the bf16 drafter +/// costs about 1.9x as much per extra block position as the 4-bit one, and +/// the two lines happen to cross at K = 4. Carrying that slope ratio onto +/// generation 13's `1.35 + 0.346 K` puts this pairing near 3.6 classic steps +/// per round at K = 4 there, which the 2.96 to 3.99 tokens a round emits +/// would only just cover. So M1 and M2 keep declining, and the estimate is +/// an extrapolation across both a pairing and a generation rather than a +/// result. This predicate is strictly more permissive than the one it +/// replaced, so no host lost a path it previously had. +/// +/// This is also the value [`super::mtp_policy::MtpPolicy`] falls back to when +/// a profiling window is ambiguous, so it steers real decisions even with the +/// adaptive path (#333) on, which is the default. /// /// `MLXCEL_ENABLE_MTP_B1` overrides the default in both directions: any value /// other than `0`/`false`/`no`/`off` forces it on, those values force it off. @@ -533,15 +570,21 @@ pub(crate) fn mtp_b1_burst_enabled(target_supports_batching: bool) -> bool { mtp_b1_default( std::env::var("MLXCEL_ENABLE_MTP_B1").ok().as_deref(), target_supports_batching, - mlxcel_core::hardware::get_hardware().has_neural_accelerator, + mlxcel_core::hardware::get_hardware() + .silicon_gen + .wide_quantized_projections(), ) } /// Pure decision core of [`mtp_b1_burst_enabled`], separated for unit testing. +/// +/// `wide_quantized_projections` is the host's Apple GPU generation reduced to +/// the one property that decides whether a verify block amortizes; see +/// [`mtp_b1_burst_enabled`] for what it is and what measured it. pub(crate) fn mtp_b1_default( env_override: Option<&str>, target_supports_batching: bool, - has_neural_accelerator: bool, + wide_quantized_projections: bool, ) -> bool { if let Some(v) = env_override { return !matches!(v, "0" | "false" | "FALSE" | "no" | "off"); @@ -549,7 +592,7 @@ pub(crate) fn mtp_b1_default( if !target_supports_batching { return true; } - has_neural_accelerator + wide_quantized_projections } /// Whether to force the Gemma 4 MTP B>1 batched burst path. Off by default. @@ -2867,30 +2910,101 @@ fn model_variant_label(model: &LoadedModel) -> &'static str { #[cfg(test)] mod tests { - /// Issue #165: per-hardware B=1 MTP default decision table. + /// Issue #165, revised by #1217: per-hardware B=1 MTP default decision + /// table. #[test] fn mtp_b1_default_policy_table() { use super::mtp_b1_default; // Env override wins in both directions, regardless of hardware. for &batching in &[true, false] { - for &na in &[true, false] { - assert!(mtp_b1_default(Some("1"), batching, na)); - assert!(mtp_b1_default(Some("on"), batching, na)); - assert!(!mtp_b1_default(Some("0"), batching, na)); - assert!(!mtp_b1_default(Some("false"), batching, na)); - assert!(!mtp_b1_default(Some("off"), batching, na)); - assert!(!mtp_b1_default(Some("no"), batching, na)); + for &wide in &[true, false] { + assert!(mtp_b1_default(Some("1"), batching, wide)); + assert!(mtp_b1_default(Some("on"), batching, wide)); + assert!(!mtp_b1_default(Some("0"), batching, wide)); + assert!(!mtp_b1_default(Some("false"), batching, wide)); + assert!(!mtp_b1_default(Some("off"), batching, wide)); + assert!(!mtp_b1_default(Some("no"), batching, wide)); } } // No override: non-batchable targets stay on everywhere (B=1 is their // only decode path and measured profitable on both chip classes). assert!(mtp_b1_default(None, false, true)); assert!(mtp_b1_default(None, false, false)); - // Batch-capable targets: on only on M5+ (Neural Accelerator) chips. + // Batch-capable targets follow the wide-projection split: on from + // Apple GPU generation 15, classic decode below it. assert!(mtp_b1_default(None, true, true)); assert!(!mtp_b1_default(None, true, false)); } + /// Issue #1217: the batch-capable half of the table, stated in terms of + /// the chip generations it actually decides for, so a change to + /// `wide_quantized_projections` cannot silently re-route this gate. + /// + /// M3 is the row this issue moved, and it moved on measurement: the + /// 31B + bf16 assistant pairing runs 1.95x to 2.65x on an M3 Ultra that + /// the previous `has_neural_accelerator` predicate declined + /// (`docs/benchmark_results/mtp-b1-gate-m3ultra-2026-08-20.md`). + #[test] + fn mtp_b1_default_batch_capable_follows_gpu_generation() { + use mlxcel_core::hardware::AppleSiliconGen; + + let decides = |chip: AppleSiliconGen| { + super::mtp_b1_default(None, true, chip.wide_quantized_projections()) + }; + + // Generation 13: the verify block runs as narrow per-position passes + // and its founding regression has not been re-measured, so it keeps + // declining. + assert!(!decides(AppleSiliconGen::M1)); + assert!(!decides(AppleSiliconGen::M2)); + // Generation 15+: measured on M3, inferred for M4 from the shared + // dispatch, and M5 keeps the path it already had. + assert!(decides(AppleSiliconGen::M3)); + assert!(decides(AppleSiliconGen::M4)); + assert!(decides(AppleSiliconGen::M5)); + // Non-Apple / not-yet-enumerated hosts decline, unchanged by #1217. + assert!(!decides(AppleSiliconGen::Unknown)); + + // Non-batchable targets are generation-independent, on every host. + for chip in [ + AppleSiliconGen::M1, + AppleSiliconGen::M3, + AppleSiliconGen::M5, + AppleSiliconGen::Unknown, + ] { + assert!(super::mtp_b1_default( + None, + false, + chip.wide_quantized_projections() + )); + } + } + + /// Issue #1217: no host may lose the B=1 path it had before the gate + /// changed. The new predicate must be strictly more permissive than + /// `has_neural_accelerator`, which is what makes it safe to flip on + /// M3 Ultra evidence without an M1 Ultra re-measurement. + #[test] + fn mtp_b1_default_never_revokes_a_previously_enabled_host() { + use mlxcel_core::hardware::AppleSiliconGen; + + for chip in [ + AppleSiliconGen::M1, + AppleSiliconGen::M2, + AppleSiliconGen::M3, + AppleSiliconGen::M4, + AppleSiliconGen::M5, + AppleSiliconGen::Unknown, + ] { + let before = super::mtp_b1_default(None, true, chip.has_neural_accelerator()); + let after = super::mtp_b1_default(None, true, chip.wide_quantized_projections()); + assert!( + !before || after, + "{chip} ran B=1 MTP before #1217 and would decline after", + ); + } + } + use super::*; #[test]