arm: adaptive-KV-streaming correctness/perf on the 2x-RTX RPC-split topology (design phase)

[hydra-z]

Summary

Design-only arm spec (no rig actions, no merge to main): evaluate the
adaptive KV streaming technique
(RaymondHuang210129/llama.cpp-adaptive-kv-streaming,
branch feature/adaptive-kv-stream, discussed at ggml-org#28216)
on our production topology — 2× RTX same-host RPC split
(5060 Ti 16 GB main + 3060 12 GB rpc peer, -ts 27,38), parallel: 2,
Qwen3.8-27B hybrid (GDN + attention), baseline d50efc6f0.

This is the successor candidate to the context-shift lever
(--context-shift is a config-no-op for qwen35 — IMROPE/n_pos_per_embd=4
hard gate — and force-enabling it is lossy/discard-based with a real
GDN-rollback risk; see the context-shift arm doc). Streaming is exact and
lossless, attention-only, and their tested shape is literally our model
family on the same VRAM class — but upstream tested single GPU, single
process, single slot.

Topology-portability finding (the crux — do not assume)

1. Two processes is the hard part, not two GPUs. Under the production
   layer split, FLASH_ATTN_EXT for RPC0-owned KV layers runs inside the
   ggml-rpc-server process on the 3060, which today has no path to receive
   --kv-stream-stage-mib. The streaming branch touches zero RPC
   transport code (ggml-rpc.cpp/transport*: 0-line diffs; an unrelated
   ggml-rpc.h version bump rides along). so per-process enablement
   (env-keyed CUDA-backend activation vs a small rpc-server argv extension)
   is an explicit Gate R deliverable of the execution phase, not an
   assumption. The spec defines a restricted mode (main-device-only
   streaming + peer fully resident) as the fallback evaluation shape.
2. -sm row is unsupportable on CUDA (PR105.0 rig-validation) —
   so composition must be evaluated on the production layer split, and
   PR105.0 also proved the in-process single-machine topology is a false
   refuge (14× UM-split collapse / −16% best-case vs RPC). RPC split is the
   only valid production shape for this arm.
3. GDN sidestep looks real (recurrent state is O(1), never streamed) and
   is pinned as Gate G (code-read proof + our post-fix: include recurrent/hybrid models in checkpoint-creation gate (hydra_vortex #316) #5/fork: revert hybrid-model checkpoint-creation gate (hydra_vortex #346, warmup hang on qwen35moe) #9 checkpoint
   semantics vs their 7dc5fcd5 cache-restore support) — verified, not
   assumed.
4. KV type mode risk pinned (Gate T): production V q5_1 is not their
   tested q4_0; the header exposes a DIRECT vs F16 attention-mode
   capability probe, which the arm runs before any probe traffic.
5. parallel: 2 is mandatory (cells B/D), not assumed-generalizable —
   consistent with the session's llama-kt development ggml-org/llama.cpp#469/The quantize.py script is not needed anymore. Just fetch the latest code and do this as a quantization step: ggml-org/llama.cpp#641/[Work Group] Add RLHF like ColosallChat on bigger dataset to achieve ChatGPT quality ggml-org/llama.cpp#743-Add "-e"/"--eval-threads" to distinguish thread counts for single-token eval and prompt eval ggml-org/llama.cpp#744 bug-class precedent.

Merge / integration risk (plainly stated, not a clean rebase)

• Their branch sits on upstream master ~Aug 24; our baseline on upstream
  Sep 4 + PR feat(server): --parallel-ctx-threshold admission gate + UM prefetch net (hydra#747 baseline) #110. fattn.cu was churned in between by upstream
  (sparse-fa for DSV4/GLM, CUDA + ggml: add sparse-fa for DSV4/GLM ggml-org/llama.cpp#27970, 2026-09-02) — and it's their most-edited
  file (+2274-line diff). Cherry-pick will NOT merge cleanly: expect a
  manual transplant of streaming hooks onto post-CUDA + ggml: add sparse-fa for DSV4/GLM ggml-org/llama.cpp#27970 fattn (days, not
  hours; highest-risk step).
• Their branch also carries unrelated unmerged work (ggml-cpu iqp.cpp,
  RPC RCE patch RPC GRAPH_COMPUTE Arbitrary RCE patch ggml-org/llama.cpp#20908, transport-apple.*, ...) that WOULD collide with
  our GGML_RPC=ON rpc-server build → wholesale merge prohibited by design;
  the port commits cherry-pick the KV-stream subset only.
• Local branches order (explicit): stream-port first on clean baseline,
  PR103 second (trivial — 11-line env gate in ggml_cuda_try_gdn_cache_fusion),
  PR104 third (moderate — +560 lines in ggml-cuda.cu re-entering
  ggml_cuda_compute_forward/graph-capture region + common.cuh
ggml_tensor_extra_gpu fields; regions are mostly disjoint from the
  streaming hunks, semantic risk is the graph-capture machinery both touch).
  Reversing order maximizes churn in the most delicate file region.

What the doc contains (per existing arm conventions)

Background/premise, confirmed facts table, feature port plan (S0/S1/S2),
merge-risk table incl. our own fork/pr103-gdn-fusion-impl and
fork/pr104-mixed-quant-arm + baseline UM-prefetch commit, hypotheses
(H1 portability/H2 concurrency/H3 rpc-plumbing/H4 kv-type), rig + launch
spec (747.4 pin + --kv-stream-stage-mib, cells with per-VRAM/pinned-RSS
budget logging), Gates B/T/R/G, test cells A–F (parity at 65536 total ctx;
prod-shape 262144; capability cell 524288; 2-concurrency depth via the
PR105.0 2×10 multiturn harness; conditional interplay cells), pass/fail
bars (greedy byte-identical parity, both-marker recall ≥5/5 as the
no-discard proof, 0 shift/eviction events, VRAM+host budget ceilings,
throughput floor vs the UM-thrash collapse signature), correctness gates,
and sequencing for the runner.

Reviews sought BEFORE anyone runs this on the rig

1. Gate R mechanism preferences (env vs argv extension) and whether a
   restricted cell (main-device-only streaming) is worth early execution if
   full composition plumbing stalls.
2. q5_1 V vs q4_0 V as the streaming-native primary cell.
3. Merge-order confirmation: stream-port → PR103 rebase → PR104 rebase.

No rig actions, no build, no test harness executed in this PR. Per arm
convention: no merge without a rig execute pass + explicit user
confirmation.