arm: uniform q8_0 KV (main+draft) vs production mixed pin — measured loss on the rig

[ddvnguyen]

Config-only arm. Doc: docs/arms/arm-uniform-q8-kv.md (in this branch).

Question

Does -ctv q8_0 -ctvd q8_0 (all-main KV q8_0, K already q8_0) decode faster than the production mixed pin (-ctv q5_1 -ctvd q5_1)?

Answer: NO — uniform q8_0 is a measured LOSS

protocol	mixed pin (control)	uniform q8_0	delta
multiturn-growth 1×12, turns 1-9 mean	24.26 tok/s	18.75 tok/s	−22.7%
pure decode ms/tok @ turn 1	20.80	21.88	+5%
pure decode ms/tok @ deep turns 7-9	26.9–31.2	33–36.6	+15–24%
single-request 150-tok eval @ ~16K ctx	34.17/34.38/34.45	33.32/33.19/33.09	−3.2%
prefill @ 16K	817 tok/s	819 tok/s	par

Mechanism: q5_1 V = 7.5 bit/elem vs q8_0 = 8.25 bit/elem ⇒ uniform q8_0 packs ~+10% more V bytes (+17% KV buffer bytes measured: main KV 2176 vs 1856 MiB @ 65k/np1); FA kernels dequantize V to f16 in-kernel either way, so uniformity buys no speed and the extra bytes tax every attention step, amplified with depth (per the strict per-turn decode-latency table in the doc).

Production-shape fit

Uniform q8_0 BOOTS at -c 262144 -np 2 -ts 27,38: no OOM, KV 8704 MiB main + 544 draft; VRAM at the same 15847/11910 ceiling via UM net. Fit is NOT the blocker — speed is.

"40 tok/s for the production pin" provenance (requested grep)

No single documented figure exists. Closest sources:

1. docs/investigations/740-results-report.md (arms 102/103/104, 146176×3 ctx, kv_unified OFF): single-request 150-token eval band 40.5–49.7 tok/s ("arm102's 46.2–46.3 class"); 2-concurrent 17.6 tok/s/slot.
2. PROJECT_STATUS.md Doesn't compile due to missing headers for memcpy and assert ggml-org/llama.cpp#381 timeline: Arm 017 "36–39 tok/s decode" (multiturn, 98K ctx shape), older Doesn't compile due to missing headers for memcpy and assert ggml-org/llama.cpp#381 ~31 tok/s sustained.
3. CLAUDE.md hardware table "~200 tok/s" = single-GPU ceiling (different class).
   A "reproduce 40 tok/s" request needs to pin the protocol — the 740 single-shot eval at its 146k shape is the closest documented band (note: that shape is NOT the current 747.4 262k production pin).

Rig protocol honored both bounds: stop pod → drain (1 MiB/1 MiB) → fit boot → kill → boot/bench → kill → drain → pod start → health 200 ×2 → 15847/11911 verified.