Arm spec: context-shift hybrid-state correctness for Qwen3.8 (design phase)

[ddvnguyen]

Arm spec — --context-shift hybrid-state correctness (Qwen3.8-27B GDN hybrid)

Design/spec phase only. Not executed. Do not merge to main; no rig actions.

Closes the question: does --context-shift produce CORRECT output for Qwen3.8-27B once triggered, under realistic concurrent load (parallel: 2 = production), or does it silently corrupt the recurrent/GDN layers' state while attention layers shift fine?

Key premises (verified in source @ d50efc6f0)

1. Recurrent rollback is bounded: llama-memory-recurrent.cpp::seq_rm accepts a partial rollback only if rollback <= n_rs_seq; with production --spec-type draft-mtp, n_rs_seq = draft.n_max = 3 (a 3-token snapshot window) vs a shift discard of typically thousands of tokens.
2. Hybrid short-circuit: llama_memory_hybrid::seq_rm runs the recurrent cache first and returns false WITHOUT touching the attention cache on recurrent failure.
3. The server ignores the failure: tools/server/server-context.cpp:2970 calls slot.mem.seq_rm(...) without checking the return, then proceeds with seq_add + token-buffer trim → attention-vs-recurrent desync, silently. No log anywhere on the seq_rm refus (documented diagnosability gap, deliberately not fixed in this arm).
4. NEW empirical finding from boot logs: every 747.x production boot prints KV cache shifting is not supported for this context, disabling KV cache shifting (common/common.cpp:1457) — i.e. production's context_shift: on is silently converted to OFF at every boot. Gate 0 of this arm verifies whether a small-ctx shape can boot with shift actually enabled, and which lever flips that gate.

Arm design summary

• Cells: A1/A2 (MTP on = production, n_rs_seq=3) at concurrency 1/2; C1/C2 (--no-spec, n_rs_seq=0) at 1/2. Plus a no-shift baseline control session.
• Probe: needle-in-haystack two-marker recall. Plant marker M1 (pig Wilbur / chartreuse / Borzoi-san) early → falls before the eviction boundary. Keep generating past shift; plant M2 (squirrel Zurnif-8 / Pavdeel / Felarn) after the boundary. Post-shift: expect honest forget of M1 and reliable recall of M2 (all 3 facts, 5/5 probes). Red flags = silent GDN corruption: garbled output, flaky/hologram recall of never-evicted M2, byte-mismatch on temp-0 repeats (P4), cross-session marker leakage (sessions use disjoint casts).
• Concurrency focus: both sessions grow past n_ctx and fire their own shifts while interleaving spec-decode snapshot rollbacks in the same llama_memory_recurrent instance; compares against the 1-concurrency control.
• Launch: 747.4 production pin with ctx 262144 → 16384 (2×8192 — shift fires within ~10 chat turns/~20 K tokens instead of 262 K; no UM oversubscription at this size, so UM is dropped as a confound). RPC topology 27,38 unchanged.
• Bars: concrete PASS (clean eviction boundary, M2 5/5, M1 ≤1/5 forget, byte-determinism, zero leakage, A2≡A1 verdict) vs FAIL (Gate-0 blocked / boundary-blind recall / coherence crack / leakage / shift-count thrash). A1-fail-but-A2-pass (or vice versa) = concurrent-interference finding, hard fail in either direction.

Same bug-class as fork findings ggml-org#469 / ggml-org#641 and upstream ggml-org/llama.cpp ggml-org#22384 / ggml-org#24055 / ggml-org#20428.

[ddvnguyen]

Updated (commit 1a991ca8d) per review:

1. Root cause pinned in the doc — llama_model_rope_type() maps qwen35/qwen35moe unconditionally to LLAMA_ROPE_TYPE_IMROPE → n_pos_per_embd() = 4 → llama_kv_cache::get_can_shift() hard-fails → common/common.cpp:1457 disables ctx_shift at every boot. No config lever exists. (Also confirmed general.architecture = qwen35 in the production GGUF header.)

2. New gate discovered during the trace: llama_kv_cache::seq_add() and seq_div() carry GGML_ASSERT(hparams.n_pos_per_embd() == 1) — a get_can_shift() bypass alone would hard-abort at the very first shift. So the test patch covers all three sites.

3. Test-only patch added (docs/arms/arm-context-shift-hybrid-testpatch.patch, 37 lines, one file, un-applied on the branch):

   • env gate LLAMA_TEST_FORCE_SHIFT_QWEN35 (dormant/unset = byte-identical behavior to vanilla baseline);
   • scoped tightly to arch == QWEN35 || QWEN35MOE inside the IMROPE prohibition branch — other IMROPE arches (qwen3vl etc.) stay prohibited;
   • seq_add/seq_div asserts downgraded to warning under the same env gate, proceeding with a scalar cell-pos shift — justified for text-only chats where all 4 M-RoPE axes hold the same value (mixed media would silently corrupt non-temporal axes → hence TEST HARNESS ONLY);
   • explicit "do not ship to production; IMROPE K-shift 4-axis correctness is itself unverified, a second separate risk" comments at all three sites;
   • verified git apply clean against baseline d50efc6f0; revert = git checkout -- src/llama-kv-cache.cpp.

4. Gate 0 revised: now a patch-based gate with a built-in negative control (patch applied + env UNSET must still print the disabling warning), the shape-lever confirmation runs are optional (expected to fail per roof cause), and a no-abort check at the first shift event.

No rig actions taken; still design phase. Ready for second review.