feat(rollout): segment-based kernel injection for hybrid-engine greedy decode

[delock]

Summary

Adds a segment-KI framework that accelerates greedy decode in the hybrid-engine rollout by injecting native CUDA kernels into comm-free segments of the model, plus a C++ decode loop that runs the full autoregressive generation with near-zero per-step Python overhead.

Segment detection: structural, not per-model

Detection is attribute-name based: any HF family whose gated MLP uses gate_proj/up_proj/down_proj (LLaMA, Qwen, Mistral, DeepSeek, Gemma...) or whose GatedDeltaNet block uses the in_proj_{qkv,z,b,a} set is picked up without per-model code. Projections carrying TP collectives are never fused (a defensive isinstance check against the Allreduce layer hierarchy).

Co-location by construction

Both GLU (dual-weight GEMV) and GDN (quad-weight GEMV) replacements read the original weight Parameters directly through autograd.Functions with exact backward, so training and generation share one forward path: zero weight copies, no inject/eject switching between rollout and training. Verified end-to-end by an integration test (below): generate → train one step → generate, output changes, no sync calls.

Components

File	Role
csrc/module_inject/fused_glu.cu	dual/quad-weight GEMV (MLP / GDN input projections), GDN gates, decode attention (warp-per-head, KV split-K, online softmax), fused residual+RMSNorm, triple QKV GEMV, graph-capturable decode step
csrc/module_inject/decode_loop.cu	C++ generation loop taking a backend-agnostic std::function replay callable (CUDA graph, eager forward, or future SYCL); argmax and buffer updates run as a private kernel; EOS checked via amortized D2H sync every 16 steps
deepspeed/module_inject/segment_ki.py	segment detection + forward replacement (apply_segment_ki)
deepspeed/module_inject/kernel_reference.py	pure-PyTorch executable specifications for every kernel — test oracle, runtime fallback, porting reference
deepspeed/ops/module_inject/{fused_glu,decode_loop}.py	op builders (JIT compile, lazy load)
deepspeed/utils/static_cache.py	hybrid-slot static cache: real static KV layers for full attention, pass-through slot reusing HF's cudagraph-safe GDN state buffers by reference
deepspeed/runtime/rollout/hybrid_engine_rollout.py	prefill → cache setup → graph capture → C++ decode loop, with layered fallback (full-step graph for b=1, graph+argmax for b>1, C++ loop, Python loop)

Performance

Qwen3.5-4B, RTX 4080 SUPER, greedy, b=1:

	tok/s	relative
HF eager	22.5	1.0×
DeepSpeed (this PR)	69.4	3.1×
vLLM	74.9	—

Tests

tests/unit/module_inject/test_segment_ki_kernels.py — 11 tests, all passing (executed on RTX 4080 SUPER):

• Kernel vs reference consistency (GPU, 6 tests): every kernel compared against its kernel_reference.py oracle on identical inputs (dual_gemv, quad_gemv, gdn_gates, triple_gemv, fused_add_norm, decode_attn)
• Injection invariants (CPU-runnable, 4 tests): injected forward matches un-injected output, no weight copies installed, composite autograd path, backward scatters gradients to the original Parameters
• Rollout → train → rollout integration (GPU, Qwen3.5-0.8B-Base): generate with graph capture + segKI, one SGD step through the same injected model, generate again with changed output — no inject/eject/sync calls anywhere in between
• Multi-GPU (AutoTP) validation (planned follow-up)

Notes

• SYCL/XPU porting: kernel_reference.py serves as the executable spec; decode_loop's std::function interface is backend-agnostic (graph replay or eager forward); SYCL kernels would follow the same op-builder pattern.
• Related upstream fix split into a standalone PR: fix: attention_unfused alpha missing norm_factor squaring #8666 (attention_unfused alpha squaring).