docs(benchmarks): price the qmv_wide narrow pin's collateral

[inureyes]

Summary

Issue #1261 gated any scoping of the qmv_wide narrow pin on first measuring what the rest of the process pays for it, with an explicit exit condition: if the tax is small, document the number and stop. Measured on the required generation 15 host (M3 Ultra, 512 GB): the B = 2/4/8 batched-decode tax is at most 1% on Gemma 4 31B and at most 0.2% on Qwen 3.8 27B (8 and 4 samples per arm per cell, every spread at or under 1%, both arms pinned explicitly via MLXCEL_QMV_WIDE). The reason is structural rather than numerical: both families the pin can fire on decode batches as per-sequence M = 1 forwards (Gemma 4 dispatches one row per forward through forward_with_sequence_id: on the VLM route the measured checkpoint takes, via Gemma4VLModel's forward_batched_with_context_and_ids override and forward_batched_with_seq_ids_dispatch; on the text-only route, via the trait default's per-row loop. Qwen 3.5's override takes its per-row branch for every single-token input), so batched decode never dispatches the kernel the pin disables. The collateral cost that is real is one prompt-cache-adopted suffix prefill per request, whose M lands in the qmv window: +15.4 ms per forward on Gemma (+33% on cache-hit TTFT), +12.6 ms on Qwen (+28%), each fitting a single per-forward constant across all concurrency levels, with uncached and chunked prefill measured as controls (+0.2 to +1.1%). The mixed arm (one MTP stream holding the speculative slot plus four classic streams, Qwen pairing) bounds the bystander cost at 0.1 to 2.6%, and the loss tracks the MTP stream's own narrow verify occupying the shared worker, not the classic streams' kernels. Per the issue's exit condition, Step 2 is documented as not built; the boundary that would reopen it (a real joint batched decode for an MTP family, as Gemma 3 and Llama 4 already have) is named in the record.

The measurement also settled the stale MLXCEL_MTP_ALLOW_INEXACT recipe. Since #1199 the gate retries without qmv_wide before the override is consulted, so on generation 15+ hosts where the narrow retry passes the override alone is inert: verified live with all four env recipes on the pairing #1258 measured, with the log lines, byte-identical output between the default env and the override-alone run, and the 117 vs 139 tok/s throughput split as evidence. The fast-kernel recipe is MLXCEL_QMV_WIDE=1 MLXCEL_MTP_ALLOW_INEXACT=1. Note the issue body's own description of its mixed arm ("MLXCEL_MTP_ALLOW_INEXACT=1 with the switch left wide") needs the same correction; the measured arm B used the two-variable recipe. The docs are corrected accordingly, and the merged PR descriptions of #1199 and #1258 carry a prepended dated note pointing here.

Found on the way and recorded for follow-up: on current main the 31B + bf16 assistant pairing probes non-identical under BOTH kernels on this host ("Disabling qmv_wide did not make it exact either"), so the default-env gate declines the batch-capable burst #1217 enabled; #1217's 1.95x to 2.65x rows were measured at 9e2c6675, which predates #1258's Gemma probe.

What changed

• scripts/bench_qmv_wide_pin.sh (new): ABBA boot driver for both measurement arms (sweep: pinned wide/narrow B-sweep without a drafter; mixed: one MTP stream plus N classic streams, with each boot's arm identity grepped from the gate log lines).
• scripts/bench_qmv_pin_mixed.py (new): mixed-workload client; classic decode rates are the reported quantity, and a window is invalid unless the MTP stream decoded through at least 95% of it.
• scripts/bench_serving_concurrency.py: count reasoning_content deltas alongside content; a reasoning model that spends its budget thinking previously reported no TTFT and no decode rate at all.
• docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md (new): the full measurement record, including the contaminated long-context cell reported as text-divergence-contaminated rather than averaged in.
• docs/benchmarks.md: link the record from the pin discussion; correct the fast-row reproduction recipe and the "declining probe" sentence for the post-feat(speculative): let the MTP gate buy back exactness by dropping qmv_wide #1199 retry ordering.
• docs/environment-variables.md: correct the MLXCEL_MTP_ALLOW_INEXACT row and add the missing MLXCEL_QMV_WIDE row.
• docs/benchmark_results/mtp-drafter-step-profile-m5max-2026-08-17.md: dated note that its reproduction recipe predates the feat(speculative): let the MTP gate buy back exactness by dropping qmv_wide #1199 retry.
• .gitignore: ignore the raw bench-results/ run directories the driver writes.
• scripts/with_indexers_paused.sh: refuse to suspend anything without a resume list, validate INDEXER_RESUME_DEADLINE as an integer instead of splicing it into a detached bash -c string, and record each pid before it is stopped.
• TECHNICAL_REPORTS/1278-qmv-wide-pin-tax-measurement-20260822.{en,ko}.md (new): the bilingual technical report for this measurement.

Changes during review

The measurement and its conclusion are unchanged; the review cycles hardened the harnesses that produced it and corrected one code citation.

• The Gemma 4 mechanism originally cited the text-only trait default. The measured checkpoint carries embed_vision.* weights, so it routes through Gemma4VLModel, which does override forward_batched_with_context_and_ids. That override delegates to forward_batched_with_seq_ids_dispatch, itself an explicit per-row loop, so the per-sequence M = 1 conclusion holds on the route the run actually took. The Summary above cites both routes.
• The driver could not be interrupted: a trap handler that returned resumed the loop, so an interrupt mid-run killed one server and booted the next arm, completing all boots and exiting 0. Handlers now exit, and HUP was added so a dropped connection cannot orphan a 31B server.
• wait_ready probed the health endpoint before checking that the launched child was alive, so a server already holding the port would answer and every boot would measure one kernel while reporting success. Arm identity is the only property the ABBA design establishes, so this is now a port preflight followed by a liveness check.
• An inherited MLXCEL_* export could silently collapse the two arms of the mixed run. It is now rejected up front, and the run records the inherited environment.
• A TypeError from a malformed usage field was outside the surrounding except tuple in bench_serving_concurrency.py and propagated through the gather, killing every remaining concurrency level. It is now handled per level.
• Harness exit status was discarded in both modes, so a failed window could unbalance the design silently. Failures are now recorded and surfaced.

Test plan

• B-sweep, Gemma 4 31B: 8 boots ABBA, 1 warm-up + 2 measured passes each, under scripts/with_indexers_paused.sh, Time Machine off; per-stream generation lengths verified equal across arms (59 tokens) via usage counts.
• B-sweep confirmation, Qwen 3.8 27B: 4 boots ABBA, same protocol.
• Mixed arm, Qwen pairing: 4 boots A B B A, 1 warm-up + 2 measured windows each; arm identity verified from the exactness-gate log line of every boot; window validity (MTP overlap at least 99.7%) checked.
• Gate recipes: 4 env recipes, 2 interleaved samples each, on gemma-4-12b-it-4bit + assistant; log lines, byte-identity diffs, and throughput recorded.
• python3 -m py_compile on both Python harnesses; bash -n on the driver. No Rust source changed.
• Post-review: bash -n on both shell scripts, python3 -m py_compile on both Python harnesses, and cargo fmt --check all clean. Interrupt handling, port preflight, and the mktemp failure path each verified against a replica or stub.

Closes #1261

[inureyes]

Implementation Review Summary

Intent

Measure what the rest of a server process pays for the MTP exactness gate's process-wide qmv_wide narrow pin (#1261 Step 1), and take the issue's sanctioned exit if the tax is small: record the number and do not build Step 2.

The two load-bearing code claims, verified against source

The whole argument rests on neither MTP family dispatching an M = B projection on batched decode. One citation held, one did not.

Qwen 3.5: correct. src/models/qwen3_5.rs:3388 is if batch_caches.len() <= 1 || shape[1] <= 1, which takes the per-row loop on every decode step, and Qwen35VLModel (src/vision/qwen3_5_vl.rs:393) delegates straight through to the same function, so the citation covers both variants.

Gemma 4: off-path, corrected in 5c5fa1f. The record said Gemma 4 "does not override forward_batched, so it inherits the trait default (src/lib/mlxcel-core/src/generate.rs:736)". True of src/models/gemma4.rs, but models/gemma-4-31b-it-4bit carries embed_vision.* weights, so gemma4_has_vision_weights (src/models/detection.rs) routes it to LoadedModel::Gemma4VLM. execute_batched_decode (src/server/batch/scheduler.rs:7300) calls forward_batched_with_context_and_ids with Some(seq_ids), and Gemma4VLModel overrides that entry point (src/vision/gemma4_vl.rs:687) in favour of forward_batched_with_seq_ids_dispatch (src/multimodal/batched_dispatch.rs:60). The trait default is never reached on the measured configuration.

The conclusion survives, and on firmer ground: that helper is itself an explicit per-row loop over forward_with_sequence_id, so M = 1 per forward on both routes. But a reader following the citation would have found the override and concluded the claim was false, which is the one thing a measurement record cannot afford. The record now describes both routes and names the one the run took.

Findings addressed

• Gemma 4 batched-decode mechanism cited a route the measurement did not take (HIGH)
• ~23% on Gemma 4 quoted without naming the quantity, in a doc that carries a verify-forward 23% and an unrelated end-to-end-decode 23% and explicitly warns against conflating them (MEDIUM)
• Composite 17 to 23% range quoted in the mixed-workload section, which measures the Qwen pairing alone; now quotes that pairing's own 17 to 20% verify-forward figure (MEDIUM)
• Suffix-forward fit claimed agreement to within 0.1 ms on every row; the B = 8 row is 0.2 ms off, 4.5 x 15.4 = 69.3 against a measured 69.5 (LOW)

Verified accurate, no change needed

• qmv_wide_pinned_by_operator() is std::env::var("MLXCEL_QMV_WIDE").is_ok(), so the new env row's claim that any value including 0 counts as a pin is exact, and the values column matches the C++ parsing in mlxcel_qmv_wide_flag() character for character.
• retry_without_qmv_wide runs at line 287 and let decision = exact || allow_inexact(); at 290, so the corrected ordering claim in docs/environment-variables.md and docs/benchmarks.md is right, including the subtle case that a retry which fails restores the wide kernel, which is why the override alone does reach the fast kernel on the 31B pairing.
• All four gate-recipe log lines quoted in the record match the tracing calls, including the pinned-operator decline suffix.
• set_qmv_wide has exactly one caller (speculative_exactness.rs:207), so the env row's "the gate is its only caller" holds, and with no drafter the sweep arms cannot be flipped by anything.
• qmv batch limit is 12 for every projection of the 31B target on an applegpu_g15d part: hidden 5376, intermediate 21504, q at 8192 and the 262144 LM head all fall past the D <= 4096 && O <= 4096 branch of the arch_size == 'd' table.
• mtp_capable_target covers the Gemma 4 and Qwen 3.5 variants only, and the Gemma 3 (gemma3.rs:1302) and Llama 4 (llama4.rs:1801) counterexamples do take a joint forward_batched_decode_with_caches when shape[1] == 1 and no mask.
• Protocol: both B-sweep arms pinned explicitly, two balanced ABBA blocks (1 0 0 1 0 1 1 0), warm-up pass discarded, spreads reported against the 4% limit, indexers paused, Time Machine off, full environment table, mixed-arm identity grepped from each boot's gate line.
• The reasoning_content change cannot alter previously recorded measurements: the key is absent on non-reasoning models, so delta.get("content") or delta.get("reasoning_content") is identical to the old expression there, and where usage is present the token count never came from the delta counter anyway.
• Exclusions are stated with reasons: the long-context Gemma cell is reported as text-divergence-contaminated (130 tokens wide against 182 narrow) and kept out of the tax, with its TTFT columns retained as the chunked-prefill control.

Remaining items

• Window 2's classic-stream loss reads 2.6% from the rounded ratio 0.974; the unrounded means give 2.54%. Inside the displayed precision, left as is. (LOW)
• Sweep-mode arm identity is sound by construction, but the driver captures no artifact for it the way run_mixed greps gate lines. The record's own evidence (arms generate different text, equal 59-token lengths from usage counts) covers it. (LOW)

Verification

• All stated requirements implemented (acceptance criterion 2 satisfied through its documented-and-accepted branch, per the issue's exit condition; criterion 3 holds by construction since no Step 2 design shipped)
• No placeholder/mock code remaining
• Integrated into project code flow (record linked twice from docs/benchmarks.md, harnesses reuse bench_serving_concurrency.py and with_indexers_paused.sh, raw run directories gitignored)
• Project conventions followed (no attribution anywhere, no em dashes, PR body not hard-wrapped, Closes #1261 present)
• Existing modules reused where applicable
• No unintended structural changes (no Rust source touched)
• Tests pass (python3 -m py_compile on both harnesses, bash -n on the driver; no runtime code changed, so no Rust suite is in scope)

A bilingual technical report is committed at TECHNICAL_REPORTS/1278-qmv-wide-pin-tax-measurement-20260822.{en,ko}.md.

[inureyes]

Security and performance review

Scope: the three harness scripts. No Rust changed, so there is no production code path to price here, and no runtime-performance findings are claimed against benchmark scripts. What matters in a harness is cleanup and whether a failure is loud, and that is where every finding below sits. Two of them were reproduced on this host, using the leftovers of this PR's own measurement runs as evidence.

Fixed in d749ad5.

HIGH

The driver could not be interrupted. trap cleanup EXIT INT TERM runs the handler and then resumes the script where the signal landed. A Ctrl-C part way through an ABBA run therefore killed the current server and booted the next arm, finishing the whole 8-boot sweep and exiting 0. Verified against a replica of the trap structure: old form ran all three iterations and exited 0, new form aborts mid-iteration with 130. The signal handlers now exit, and HUP is trapped so an SSH drop or a closed terminal cannot orphan a 31B server.

A stale server on the port was measured as if it were the freshly booted arm. wait_ready probed /v1/models before checking that the boot it had just launched was still alive. Anything already holding port 8113 (most plausibly a server leaked by an aborted run, which the finding above made likely) answers the probe, every subsequent launch dies on EADDRINUSE, and all eight boots measure one kernel while the run reports success. Arm identity is the only thing the ABBA design establishes. There is now a preflight that refuses to start when the port already answers (verified against a stub on 8113), and the liveness check runs before the readiness probe inside the loop.

with_indexers_paused.sh: an unchecked mktemp could leave the machine's indexers suspended with no way back. If mktemp fails, PIDFILE is empty, the daemons are SIGSTOPped, and their pids are recorded nowhere, so neither the trap nor the detached backstop can restore them. Confirmed by simulation. The wrapper now refuses to suspend anything when it cannot create the resume list, and records each pid before sending the STOP rather than after, so a failed write can never produce an unrecorded suspension.

MEDIUM

INDEXER_RESUME_DEADLINE was a shell injection sink. It was spliced into a detached nohup bash -c string. INDEXER_RESUME_DEADLINE='0; touch /tmp/pwned' executed the injected command in a process that outlives the wrapper; confirmed. It is now validated as a positive integer, and the deadline and pid list are passed as positional arguments instead of into the program text. $PIDFILE was interpolated into single quotes on the same line, which a TMPDIR containing a quote would have broken; same fix.

Shutdown was SIGTERM-only with an unbounded wait, and the pid was not cleared after reaping. A server that does not act on SIGTERM (a 31B model mid-Metal-call) made wait block forever, hanging the harness with the model resident. And because cleanup() never cleared SERVER_PID, a second handler pass could signal a pid the OS had recycled. Shutdown now escalates to SIGKILL after SHUTDOWN_GRACE_S (default 60), and clears the pid before returning.

Inherited MLXCEL_* silently redefined the arms. run_mixed arm A is "the default env" and run_sweep assumes MLXCEL_QMV_WIDE is the only difference between boots. Neither sanitized the operator's shell. An export left over from testing a gate recipe by hand, which is exactly how this PR's recipe section was produced, would collapse both arms onto one kernel with no symptom. The driver now aborts when either variable is already set, and env.txt records the inherited MLXCEL_/MLX_/LLAMA_ARG_ environment.

Python harness exit status was discarded. A bench_qmv_pin_mixed.py window that raised SystemExit ("MTP stream produced no token"), or a bench_serving_concurrency.py pass that errored, left a traceback in its output file while the loop marched on, costing one arm a sample invisibly at aggregation time. Failures now write a HARNESS-FAILED rc=N marker into the file, plus a run-level warning naming the count.

bench_serving_concurrency.py could abort a whole sweep on one malformed field. int(usage["completion_tokens"]) raises TypeError for a non-numeric value, and TypeError is not in the surrounding except tuple, so it propagated out of the executor, through the gather in run_level, and killed every remaining concurrency level. Reproduced against a stub emitting completion_tokens as a dict: pre-fix exit 1 with a traceback, post-fix the level completes on the delta count. Same guard added to bench_qmv_pin_mixed.py.

A leaked MTP stream could silently change the next window's shape. mtp.join(timeout=15) discarded its outcome. A stream that has not ended may still hold the tick-slice slot, which makes the next window's "MTP stream" a classic one measured under the same label: exactly the silent substitution the overlap check exists to catch. mtp_drained is now part of window validity and appears in the JSON.

The backstop leaked one process per run. Eight bash plus sleep pairs were resident on this host when the review ran, one per measurement run tonight, the oldest 2h41m old, each holding until its deadline. A stale one is not only litter: it wakes hours later and signals pids the OS may have recycled. It is now retired by the trap on any ending that reaches it, while still surviving a SIGKILL of the wrapper, which is its whole purpose.

LOW

Arm tokens flowed unvalidated into output paths (SWEEP_ARMS and MIXED_ARMS are word-split, which also glob-expands); now rejected unless 0|1 and A|B respectively. No preflight on the binary or the output directory, so a missing build or an unwritable $OUT surfaced 600s later or as silently discarded results; both are checked now, matching benchmark_paged_decode_production.sh. SIGQUIT was missing from the wrapper's trap list. The exception that ends a stop-induced read is now recorded rather than swallowed, so a genuine parsing bug is distinguishable from a clean stop.

Field evidence worth keeping

Two of the wrapper's leftover pid files on this host still held pids, from the 02:09 and 02:56 runs, which means resume() never ran for them and only the deadline backstop stood between the machine and indefinitely suspended Spotlight. They were rescued incidentally, because a long-lived mdworker appears in several sequential runs' lists and a later run's resume reached it. Had either been the last run of the night, the raised INDEXER_RESUME_DEADLINE would have left it suspended for two hours. The header comment claimed the "already suspended, leaving it alone" guard prevents runs from touching each other's work; that guard is at suspend time only, and the comment now says so.

Looked at, not changed

The byte-at-a-time SSE read loop (resp.read(1) plus buf += chunk) costs client CPU proportional to bytes received across five GIL-bound threads on the machine under measurement. It is left alone deliberately: the committed record was produced with this exact loop, the reported quantities are between-arm ratios where the client cost cancels, and changing it would break reproducibility of the tracked result document for no measurement gain.

delta.get("content") or delta.get("reasoning_content") counts one token when a delta carries both channels. Real servers emit one channel per delta, so this is noted rather than changed.

The .gitignore addition is clean: /bench-results/ is root-anchored, nothing tracked lives under it, and it does not shadow docs/benchmark_results/ (git check-ignore confirms). No secrets or absolute home paths in the new documents or scripts.

Verification

bash -n under macOS system bash 3.2.57 for both shell scripts, py_compile for both Python harnesses, and negative tests for every new guard: no-arg and bad-mode usage, inherited-env rejection, arm-token rejection including SWEEP_ARMS="../../oops", port preflight against a live stub, deadline injection and non-numeric and zero rejection, backstop retirement (/usr/bin/true run leaves zero new backstops, no daemon in state T, pid file removed), and the two Python regressions reproduced pre-fix and cleared post-fix against stub servers. No Rust source, documentation, or .gitignore was touched.

[inureyes]

PR Finalization Complete

Summary

• Tests: not added. Judged not warranted; see rationale below.
• Documentation: one cosmetic fix (docs/benchmarks.md paragraph rewrap, ca9ed648). Everything else checked was already correct.
• Lint/Format: bash -n, python3 -m py_compile, and cargo fmt --check all pass clean. No further fixes needed.

What was checked

Documentation completeness. Both new harness scripts (scripts/bench_qmv_wide_pin.sh, scripts/bench_qmv_pin_mixed.py) already carry full header comments matching the house style of bench_speculative.sh/bench_decode.sh: usage, both arms, the output layout, and the reasoning behind the design choices (why both arms pin the env explicitly, why the mixed arm needs MLXCEL_MTP_SLICE_GRANT_ROUNDS=0, etc). docs/benchmarks.md links the new results doc at both relevant sections (the MTP-cost paragraph and the batched-decode discussion). docs/environment-variables.md's MLXCEL_MTP_ALLOW_INEXACT correction and new MLXCEL_QMV_WIDE row both read correctly and check out against the source (qmv_wide_pinned_by_operator, retry_without_qmv_wide, mlxcel_core::set_qmv_wide), and the new row is placed in the logical MTP-exactness cluster in the Speculative-decoding section rather than out of place. One cosmetic fix: a paragraph in docs/benchmarks.md was left with two short orphaned lines after the MLXCEL_QMV_WIDE insertion, not matching the file's ~80-column wrap convention; rewrapped in ca9ed648.

Bilingual docs. Checked docs/README.md: the bilingual mkdocs manual sources (docs/en, docs/ko) are explicitly maintained in a separate documentation tree and are not part of this repository (mkdocs.ko.yml in this repo only carries the docs_dir/nav config for that external tree). The docs/*.md files this PR touches (benchmarks.md, environment-variables.md, benchmark_results/*.md) are GitHub-facing English-only documents by established convention; grepping the whole repo for *.ko.md confirms Korean counterparts exist only under TECHNICAL_REPORTS/, which this PR already provides in both languages. No Korean translation work was skipped.

Lint/format. bash -n on both changed shell scripts, python3 -m py_compile on both changed Python scripts, and cargo fmt --check (no Rust source changed, so no clippy/build/test run) all pass with no output. shellcheck and ruff/black are not configured for scripts/ in this repo (the python.yml CI only lints the python/ client package), so bash -n/py_compile are the applicable bar here.

Test coverage. Judged not warranted. The one precedent in this repo for testing bench-script internals, tests/test_bench_decode_oom_classifier.py, works because is_oom_failure() is already an isolated pure function (rc, err_text) -> bool with several named branches whose regression would silently misclassify results. Nothing in this diff has that shape: bench_qmv_wide_pin.sh's logic is process-lifecycle orchestration (boot/wait/kill a real server), already exercised end to end by the security review; bench_qmv_pin_mixed.py's only pure function (build_prompt) is a trivial deterministic string builder; and its one real piece of decision logic (the window overlap/valid check) is inline inside a threaded I/O function, not isolated the way is_oom_failure is, so testing it would mean a refactor rather than a natural test addition. This is one-off measurement tooling read by a human against printed valid=/overlap= diagnostics, not a decision silently embedded in an exit code.

Results doc consistency. Confirmed the filename is docs/benchmark_results/qmv-wide-pin-tax-m3ultra-2026-08-22.md (not -m3ttra-). Structure matches mtp-b1-gate-m3ultra-2026-08-20.md: an Environment table, protocol description, per-arm results tables, and explicit statements of what was and was not settled (the "What this settles, and what would reopen it" section walks the issue's three acceptance criteria one by one and names the boundary condition that would reopen Step 2).

All checks passing. Ready for review.