Skip to content

Add chunked Qwen prefill with backend KV cache reuse - #549

Merged
0xShug0 merged 1 commit into
mainfrom
feature/qwen-chunked-prefill
Sep 14, 2026
Merged

0xShug0 merged 1 commit into
mainfrom
feature/qwen-chunked-prefill

Conversation

@0xShug0

@0xShug0 0xShug0 commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Summary

Extract reusable framework support for bounded, single-sequence Qwen prefill directly into the backend KV cache.

Additions

  • Add QwenCausalDecodeRuntime::prefill_embeddings_into_cache() to process embedding inputs in bounded blocks and continue token decoding from the same cache, without exporting and reimporting KV state through host memory.
  • Add multi-token cache updates through FastKVSetRowsModule::build_block() and QwenDecoderLayerModule::build_with_static_cache_block().
  • Add TransformerKVCache::clear_on_backend() using the existing ggml tensor-memset API. It zeroes key/value tensors and resets position counters while retaining their allocations.
  • Keep the block-prefill graph owned and reused by the runtime, with cleanup integrated into its existing graph lifecycle.
  • Add a focused unit test and CMake registration for chunked prefill, continued decoding, repeated use, cache clearing, and capacity handling.

Scope

The new prefill path is explicit opt-in and requires DirectSetRows with full last-token readback. Existing prefill entry points remain available. This PR contains no model integrations, model specifications, UI changes, or ggml modifications.

@0xShug0
0xShug0 merged commit b46fe6b into main Sep 14, 2026
6 checks passed
@0xShug0
0xShug0 deleted the feature/qwen-chunked-prefill branch September 14, 2026 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant