Eval bug: qwen4exp (Qwen3.8-Flash-Next) on Metal: decode emits 1 token then EOS at long context — silent empty output, stochastic at threshold, threshold moves with model quant / KV quant / n_ctx

[dongzhang777-source]

Name and Version

0.3.0-dev (build 37, commit 8e53fce), built with AppleClang 21.0.0 for Darwin arm64 (Metal, default flags, Release)

Operating systems

Mac

GGML backends

Metal

Hardware

2x Apple M1 Max (24-core and 32-core GPU), 64 GB unified memory each — reproduced on both

Models

• unsloth-style AD-3.84bpw-IQ4_XS-M64 (28 shards) — 16K/64K OK, 128K FAIL
• AD-4.27bpw-Q4_K_M-M64 (33 shards) — 64K FAIL (q4v4 KV) / 16K FAIL (q8v8 KV)

Problem description & steps to reproduce

llama-server, single slot, GPU flash attention (auto). With prompts beyond a config-dependent threshold:

• prefill completes normally (e.g. 118-140 tok/s at 64K)
• decode emits exactly 1 token then stops (EOS): eval time = ... / 1 tokens, answer empty
• server returns HTTP 200 — silent failure, no error anywhere
• at/near the threshold the failure is STOCHASTIC: same server, same config, same prompt size → 64K x3 = PASS, PASS, FAIL on consecutive runs
• threshold moves with model quant (Q4_K_M << IQ4_XS), KV quant (q8v8+b256: 16K vs q4v4+b512: 64K), and possibly n_ctx window size (64K probes 3/3 @ n_ctx 208896 vs 2/3 @ n_ctx 262144, same 64K prompts)
• with mixed-precision KV (-ctk q8_0 -ctv q4_0) Metal rejects FA → attention falls back to CPU → 200K needle tests pass 3/3 (correct but slow) — so the failure is specific to the GPU attention path
• -fa off cannot be tested with quantized KV ("quantized V cache requires flash_attn to be enabled"); with f16 KV + -fa off decode fails immediately with Compute error. off = 0, n_batch = 256, ret = -3 on the first token
• memory pressure excluded (failures observed with < 600 MB swap)
• needle-in-haystack probe (3 pins at 10/50/90% depth), warmup, temperature 0

Matrix (all llama-server, GPU FA, IQ4_XS unless noted):

model	KV	-b/-ub	n_ctx	16K	64K	128K
IQ4_XS	q4v4	512	208896	3/3	3/3,3/3,3/3	0/3 (1 tok)
IQ4_XS	q4v4	1024	208896 (A machine, +ngram spec)	—	3/3 (32K/64K)	0/3 (1 tok)
IQ4_XS	q8v8	1024	262144	—	3/3,3/3,0/3 (stochastic!)	not reached
Q4_K_M	q4v4	512	208896	—	0/3	—
Q4_K_M	q8v8	256	208896	0/3	—	—

First Bad Commit

Unknown; qwen4exp support landed via #27742 (2026-08-27) — all failures observed on builds containing it. Retested on master df03399 (0.4.0-dev build 300, contains #27941, #28023, #28098): the bug reproduces identically. 64K x1 = 3/3 (prefill 153.8 tok/s, decode 11.1 — sparse-FA perf improvements visible), 128K x1 = 0/3, decode 0.0 (1 tok) — same signature, same threshold. So #27941 (block position keying), #28023 (indexer head summation) and Metal sparse FA do not resolve it.

Related

#27856 (HIP: depth-decay only, output correct), #28734 (CUDA: dense top-k TODO), #25722 (Metal: degenerate-token silent empty output, different arch), #21450 (Metal mixed KV rejection — the mechanism that pushes attention to CPU and avoids this bug)

Disclosure: issue text prepared with AI assistance (GLM-5.3-Flash via ZCode), verified against server logs and raw probe JSONs.

[dongzhang777-source]

Update (2026-09-12): on 0.4.0-dev (build 300, commit df03399) the failure is not exclusive to the Metal flash-attention kernel.

Same machine/model (M1 Max 64GB, IQ4_XS-M64), with mixed-precision KV (-ctk q8_0 -ctv q4_0 — rejected by Metal, forcing the entire attention to CPU):

• 64K x1: 3/3 pins OK (prefill 71.3 tok/s — the CPU-attention speed fingerprint)
• 128K x1: 0/3, decode = exactly 1 token — prefill completes normally (62.0 tok/s), identical silent 1-token-EOS signature

So on 0.4.0 the failure reproduces on the CPU attention path as well. The earlier 200K-passing run (same mixed-KV CPU-attention setup, posted above) was on 0.3.0-dev b37 / 8e53fce with the Q4_K_M quant — so either a 0.4.0 regression on the CPU path or a model-quant dependence (IQ4_XS vs Q4_K_M) is involved; not yet isolated. 64K remains reliable on every tested config; the boundary sits in 64K–128K across all of them.

[dongzhang777-source]

Cross-linking the active rework PRs + our full failure matrix (for regression use)

PRs #28213 (gather-based sparse attention for QSA decode) and #28699 (incremental pooled-key cache for the QSA indexer) rewrite exactly the decode path where this failure lives. Sharing our measured matrix so it can double as a regression test for those rewrites.

Failure signature (every run: prompt prefill completes normally; the FIRST decode step returns exactly 1 token (EOS) and the response ends silently; HTTP 200; nothing in the server log):

config (M1 Max 64GB)	window	64K	128K	258K
AD-IQ4_XS-M64, Metal FA, q4v4 KV, b512, 8e53fce	208896	3/3	0/3	—
AD-IQ4_XS-M64, Metal FA, q8v8 KV, b1024, 8e53fce	262144	2/3	—	—
AD-Q4_K_M-M64, Metal FA, 8e53fce	208896	0/3	—	—
AD-IQ4_XS-M64, CPU FA (k8v4), df03399 0.4.0-dev300	208896	3/3	0/3	—
oMLX 0.7.0.dev2 (different engine, same model, MLX PLE-SSD + 50% expert offload)	262144	3/3 @16K/64K	3/3	0/3 (deterministic memory-pressure — a different, reproducible failure mode)

Pin-retrieval probes, single-shot fresh haystacks, cache_prompt=false, 3 independent runs per cell.

Observations relevant to the rewrites:

• The threshold moves with quant, KV dtype and window size, and the boundary is random (same config, same length: 2/3 and 3/3 across repeats) — this looks like state/index corruption in the indexer top-k selection + full-window masked-attention path rather than a clean length check. qwen4exp : gather-based sparse attention for QSA decode #28213's overview confirms the current decode materializes a mask over the entire KV; a mask/index bug at large contexts (out-of-range or uninitialized selected indices, int32 overflow in mask sizing) would produce exactly this signature.
• The MLX reference implementation runs the same checkpoint at 128K fine (3/3 needle retrieval), so per-layer golden intermediates are obtainable by cross-checking against oMLX/MLX if useful.

We keep a ~3-hour repro harness on M1 Max 64GB (single 257K-token prompt kills it deterministically at the 258K rung) and can run candidate patches or nightly builds against the full matrix on request.

Comment drafted with AI assistance (GLM-5.3-Flash via ZCode), reviewed and posted by the account owner.

[dongzhang777-source]

More data: 8e53fce + thinking-low + exact-seed reruns (M1 Max 64GB, IQ4_XS-M64 q4v4, -c 208896, Metal FA)

Needle haystack (3 PINs at 10/50/90%, temp 0, fresh random filler per seed, thinking=low via reasoning_effort):

prompt	max_tokens	seed	result
82,089	1024	A	3/3, 185 tok out
98,500	1024	A	3/3
98,923	1024	B	1/3, thinking exhausted budget
99,077	1024	C	3/3, 509-char thinking, clean
131,220	1024	A	0/3, budget maxed, empty content
130,937	2048	D	3/3 in 187 tok out, 285-char thinking
131,220	2048	A (same text as the 0/3 run)	1/3, 1205-char thinking drowned

Two findings for the matrix:

1. Budget does not decide it. Same 131K text: dead at 1024, and a different 131K text passes crisply at 2048 — but the same dead text still fails at 2048. Seed/config noise dominates, consistent with the index-corruption hypothesis in the original report.
2. The 128K failure mode here is thinking-drown, not instant EOS. The failing runs show the model mis-scoping the instruction ("repeat the full needle content" reinterpreted as spanning to the next marker) and rambling until the budget dies. No 1-token-EOS observed on this config (thinking=low); the 1-token signature in the original report used default (xhigh) thinking.

Methodology note: the same-seed rerun hit prefix cache (prefill skipped), so treat its prefill number as invalid; decode/thinking ran live.

Net for our fleet: capping the A node at 64K verified-correct (96K accepted 2.5/3 across seeds — not shippable), >96K rejected loudly at the router. Happy to run candidate patches against the 96–131K stochastic band on request.