feat(cuda): arm117 — port adaptive-KV-streaming onto baseline + Gate G/R/B pass; rig cells pending (#117)

[ddvnguyen]

Executes the port (S0/S1/Gate-B) for #117's docs/arms/arm-adaptive-kv-streaming.md.

Port, not rebase — results materially BETTER than the doc's risk budget:

• Their feature/adaptive-kv-stream branch is a linear rebase-carry whose upstream lineage includes 5266f24da (v0.4.0 = our baseline base) AND sparse-FA CUDA + ggml: add sparse-fa for DSV4/GLM ggml-org/llama.cpp#27970, and continues to ~2026-09-06. The feared pre-sparse-fa fattn.cu skew does not exist.
• Port = 61 streaming-series commits cherry-picked in order, on clean baseline (d50efc6f); the one hidden feature dependency (7fa262b3 UM-mirror) included; the 6 irrelevant upstream carries excluded. Only 2 trivial conflict resolutions.
• Fidelity proof: all 44 non-carry, non-fork-delta files byte-identical (blob SHA equal) to source HEAD. PR feat(server): --parallel-ctx-threshold admission gate + UM prefetch net (hydra#747 baseline) #110 fork delta intact.

Gate G (code read): zero touch on recurrent/SSM/GDN paths (gated_delta_net.cu, ssm-*.cuh, llama-memory-recurrent*.cpp untouched). Draft KV not streamed by upstream design.

Gate R verdict: runtime is per-CUDA-device per-process, proc-address based; the rpc-server peer has no llama-layer, so stock RPC composition is impossible upstream-side. Implemented (lower-risk path, doc fallback respected): LLAMA_KV_STREAM_DEVICE=CUDA0 device filter — CUDA0 layers stream, 3060 share fully resident, production RPC topology intact. Plus LLAMA_KV_STREAM_ALLOW_MULTISEQ=1 opt-in so -np 2 (this fork's load-bearing production config, upstream-untested) is measurable rather than hard-rejected.

Gate B: llama-server + ggml-rpc-server build OK at 86;120, CUDA 13.2.2, FA_ALL_QUANTS. CPU-side unit tests of the ported feature all green (plan 241 asserts / config 18 / softmax).

Rig cells: not started. One physical rig; PR #116's arm holds it (rig-coordination protocol). Parity control, Gate T probe, and cells A/C/D run once the rig is released.

Arm doc updated with ## Results on fork/arm-adaptive-kv-streaming at fa111588.

[ddvnguyen]

Rig pass complete. Rig restored & verified (health 200 ×2, VRAM 15847/11911 exact production signature, pod_llama-baseline Running).

Verdicts

• Gate B (build + vanilla parity): PASS. Port tree with --kv-stream-stage-mib 0 is byte-identical to the pre-d50efc6f baseline binary on the production RPC shape (greedy 2208-tok kernel prompt at ctx 65536).
• Gate T: q8_0/q5_1 → ATTENTION_DIRECT (both direct_attention under FA_ALL_QUANTS; no F16-conversion path). Boot evidence: CUDA_KV_Stream_Host KV buffer = 1044 MiB, 9/16 kv layers streamed on CUDA0, 7 resident on RPC0, draft KV not streamed.
• Gate G: recurrent/GDN/SSM files untouched (code-read + ported-file list proof).

Cell A

• np1: PASS — OFF vs ON byte-identical greedy outputs (3 comparison pairs incl. fresh-boot repeats; ON self-repeat identical).
• np2: FAIL (H2-class), two independent reproducible signatures:

1. First request on fresh boot landing on slot 1 diverges from the byte-stable OFF control — reproducible across boots; NOT a cache-restore issue (blocked-cache runs with 2204 cached tokens reproduce OFF exactly; and fresh-request divergence has cached_tokens=0). Suspect slot>0 streaming page/dirty-row accounting or first-decode stream layout.
2. Hard crash under genuine 2-slot concurrency: GGML_ASSERT(ggml_cuda_kv_stream_fattn_fits(dst)) failed (ggml-cuda.cu:1898, ggml_cuda_graph_evaluate_and_capture); backtrace preserved. Server abort → port was released.

Per the doc's FAIL bar, concurrency violation stands regardless of the clean single-slot verdict. The LLAMA_KV_STREAM_ALLOW_MULTISEQ=1 knob did its job: converted an untested upstream assumption into a concrete, reproducible failure signature.

Artifacts

• Arm doc ## Results updated (fork/arm-adaptive-kv-streaming @ 40b41d6b).
• Cell logs/JSON: arm117-artifacts/ (Aoff/Aon np1+np2 server logs, snapshots, crash log with the assert backtrace).
• Cosmetic boot nit (logged, no impact): fit-probe ctx throws the device-filter error before device wiring; fit-params retry recovers.
  Follow-ups: (1) slot-aware streaming accounting fix or hard-disable multiseq upstream-side; (2) fork issue for full RPC-peer streaming composition (protocol extension); (3) optional q4_0-V diagnostic cell if desired.

[ddvnguyen]

Round 2: root cause + numbers + q8_0 verdict (rig verified, rig restored)

1) np2 crash — root-caused, fixed (ef3b119f)

The printed guard k_runtime/v_runtime identity was a red herring. Instrumented fits() (pointer+buft+runtime logging) + live repro split it into two mechanisms:

• The actual abort: ggml_cuda_flash_attn_ext_streamed_supported() (fattn.cu:1216) requires Q/K/V ne[3]==1 — a single-stream ubatch assumption. Cont-batching 2 busy slots builds K/V 4D views with ne[3]==ns==2 (probe print: dst ne=[256,24,2,1]). The dispatcher (ggml-cuda.cu:3053) routes FA to kv_stream_fattn on buffer membership only, so the geometry-rejected node walked into the assert at :1898 instead of falling back. Upstream never runs -np>1, so the path was never tested. Fix: dispatch now requires the full fits(); otherwise FALLBACK to ordinary flash_attn_ext (kv-stream buffer is host-mapped pinned storage → zero-copy reads are semantically exact) + WARN. Post-fix repro: no crash, server survived, both slots coherent.
• Slot-1 divergence (separate bug, same subsystem): resident-page bookkeeping has no stream axis (mark_dirty_rows maps row/page_tokens → page, marks layers without any stream index, fattn.cu:709-756). Slot 0/1 tokens alias the same (layer, page); last writer wins for both streams. This is the first-request slot-1 divergence (fresh-boot cached_tokens=0) and is why pool=2048 cached restores on slot 0 stay byte-exact. Real fix = a (layer × stream) resident layout — an upstream design change, not portable with a small fork patch.
• Both mechanisms are quant-size agnostic.

2) Real decode numbers (multiturn-growth-test.sh, 1 session × 12 turns, ~8K/turn, 750 out; turns 10-12 ctx-cap for BOTH — excluded by harness)

config	mean tok/s (turns 1-9)	turn 1	turn 9
baseline binary (d50efc6f, resident)	24.26	31.54	21.89
port + streaming ON (9/16 layers streamed, stage 2048 MiB)	mean 17.25	31.52	21.41

• Streaming costs ≈ −28.9% mean decode at this shape.
• Depth curve: baseline flat ~21-24 tok/s at 40-60K resident; streaming dips to 8.79-10.78 tok/s at 40-53K (host pool round-trips dominate once the 2048 MiB resident pool is exceeded), then recovers at 60K.
• turn-1 parity (31.52 vs 31.54) confirms the shallow-context path is unchanged.

3) q8_0 K+V question — definite NO

• Crash mechanism is a geometry check (ne[3]==1), not type support; Gate T already proved q8_0 and q5_1 both DIRECT. In the diagnostic print the K/V of the same layer always share ONE buffer pointer — no mixed-runtime identity possible in the failing node; fits() nil/nil case was the prepare_graph probe on a plain-buffer node (not dispatch).
• page_bytes are quant-size dependent only through ggml_cuda_kv_stream_page_bytes() geometry math, which self-adjusts; no mixed-size sensitive path (2D stride checks are per-type row-size formulas).
• A q8_0/q8_0 np2 pre-fix boot would merely reproduce the same abort (disallowed per doc); the correct fix is the geometry gate + multistream layout work, not the quant.

Rig: restored — health 200 ×2, VRAM 15847/11911 exact, prod pod Running.

Artifacts: arm117-artifacts/crashdiag-Aon-np2-server.log (instrumented assert traces), fallback-Aon-np2-server.log (post-fix concurrent repro), perf-on.txt + perf-base.txt (turn-by-turn). Doc updated (fork/arm-adaptive-kv-streaming @ 58de2e67).