feat(pflash): Qwen3.5-0.8B drafter scorer, segment probe and strict budget selection

[Graffioh]

Draft. Runtime changes for PFlash on the Qwen3.5-0.8B drafter: a strict token-budget selector over variable-length segments, a segment probe and an optional scoring head loaded from GGUF, and a chat-first scorer query derived from the rendered prompt. The previous Qwen3-0.6B path is untouched and remains the default.

What changes

Qwen3.5-0.8B drafter (qwen35_drafter.cpp/h, qwen35_loader.cpp, new; qwen3_drafter.cpp keeps the Qwen3-0.6B path and dispatches on DrafterArch; shared helpers in qwen3_drafter_common.cpp/h)

• --prefill-drafter accepts a Qwen3.5-0.8B GGUF (selected by qwen3.5/qwen35 in the file name). The drafter runs only its first fifteen blocks and scores the context with block 15's NoPE Q/K projections — the same attention-mass scorer the Qwen3-0.6B block-13 head uses — on the model's native 262K context, so inputs beyond the 0.6B drafter's 32K window can be scored.
• PFLASH_SCORING_HEAD_GGUF loads a replacement block-15 Q/K head (GGUF architecture pflash_scoring_head, keys scoringhead.*, schema qwen3_5_0_8b_nope_qk_mass_v1); the file is bound to the drafter by sha256 and refused otherwise.
• PFLASH_QWEN35_LEGACY_SCORER=1 keeps the all-layer running-max scorer available on this drafter.
• Drafter compute errors and non-finite probabilities fail closed: the request errors instead of serving an unscored prompt.

Strict budget selection (pflash_selection.cpp/h, new)

• PFLASH_SELECT_MODE=budget_only|top_p fills an exact token budget from the keep ratio / --prefill-curve (or a cumulative-mass cutoff), over fixed chunks (PFLASH_SELECT_CHUNK_SIZE, scheduled by input length when unset) or probe segments.
• PFLASH_SELECT_SCORE=auto|sum|density ranks candidates by total mass or mass density; auto picks density with probe segments, sum with fixed chunks.
• The query span and required instruction spans are mandatory and always retained; the selector reports stop=budget_reached|mandatory_query_exceeds_budget|… and the re-encode is retried when the rendered prompt overshoots the ceiling.
• Two-scorer split budget (PFLASH_SELECT_SCORER=head|legacy|split, PFLASH_SELECT_SPLIT): the head ranks first, the all-layer scorer fills the remainder.

Segment probe (PFLASH_SEGMENT_PROBE_GGUF)

• A 264K-parameter network on the block-14 tap scores every token for "a new unit of text starts here" (schema qwen3_5_0_8b_segment_probe_v1; threshold, min/max segment length and conv taps come from the GGUF metadata, sub-unit head for the oversize split rule). The context is cut at boundaries above threshold plus the query start and instruction-span edges, and the selector ranks whole functions, classes, files or paragraphs instead of fixed windows, skipping segments that do not fit the remaining budget. Falls back to fixed chunks when fewer than four boundaries are found. PFLASH_SELECT_SEGMENTS=auto|fixed|probe overrides.

Chat-first scorer query (http_server.cpp/h, prefix_cache.cpp/h markers)

• Production PFlash is chat-first. The scorer query is the prompt's last token: the end of the generation prompt, where the model starts answering, having read the whole request. Nothing is parsed out of the user's text, so the question can sit anywhere in the message (before a pasted document, in the middle of it, among the user's own sentences). Turns are found by the rendered prompt's own control markers (Qwen <|im_start|>…<|im_end|>, DeepSeek <｜User｜>/<｜Assistant｜>, Gemma <|turn>…<turn|>, Laguna <user>…</user>) rather than by message bookkeeping. Marker strings are resolved on the target tokenizer and searched in the decoded prompt, so a drafter whose vocabulary lacks the control tokens still maps correctly.
• An assistant turn left open at the prompt end is the generation prompt, whatever think or channel prefix the template adds (<|im_start|>assistant\n<think>\n, <｜Assistant｜></think>, <|turn>model\n<|channel>thought…). It is never the query. Tool output wrapped in a user turn (<tool_response>, <tool_result>) doesn't count as the user's turn.
• The tail (PFLASH_SELECT_QUERY_TOKENS) of the latest user turn scores as a second query window at the same weight: the last token reads the whole request, the user's own tokens match literal strings (an identifier, a described function) the last token does not carry. On SCBench's multi-turn chats (36 chats, 171 turns, Jev/SCBench metrics) this answers 135 turns against 112 for the last token alone, 127 with SCBench's question sent as pflash_query, and 140 for full prefill.
• The generation prompt and the latest user turn's role header (<|im_start|>user\n) are kept; the latest user turn follows the same skeleton rule as every other turn (whole when short, otherwise scored). A prompt without chat markers scores the tail (PFLASH_SELECT_QUERY_TOKENS, default 8) of its content.
• Agent loops: assistant and tool turns after the user's turn are ordinary context, scored like the rest of the conversation; only the generation prompt stays pinned.
• Measured on the dev chats re-laid out with the question first, in the middle, among the user's own sentences, two questions, and an email thread (252 turns, Jev-judged): 197/252 against 155/252 for the previous tail query and 231/252 for full prefill; turn 1 36/42 vs 20/42. The original layout is unchanged (37/42).
• PFLASH_SELECT_QUERY_PARSER=arbitrary_tail is the default; latest_user remains selectable for benchmark comparisons. Marker-less prompts and latest_user still find the boundary through sentinel re-renders.
• pflash_query stays a benchmark-only override. It replaces the derived query, is mapped against the decoded prompt text (BPE merges such as What stay inside the span), and is kept whole, with the scorer consuming only its bounded tail. pflash_required spans are kept verbatim under compression.
• Tests build their prompts with render_chat_template for all four families, with thinking on and off, instead of hand-written strings.
• Compression trace (PFLASH_TRACE_PATH, JSONL): input ids, segments, per-candidate scores, selected chunks, forced chunks, budget, mass, parser record (input kind, selection rule, parser name, fingerprints, content/query bounds).

Budget over droppable tokens

• The strict budget is kept + keep_ratio × (input − kept). Kept means the system and developer messages, tool definitions, the query with its turn's envelope, and the generation prompt, counted with the selector's own mandatory-chunk rule. Both the selector budget and the target-token ceiling follow it. Before, a system prompt larger than keep_ratio × prompt failed the request.
• Auto mode compares --prefill-threshold with the droppable tokens.

Multi-turn ownership and views

• With a strict-selection environment, PFlash owns every turn instead of failing closed or routing to FlowKV. Only request-scoped FlowKV disk compression (disk_cache_policy.compress) is still rejected. Without a selection environment the legacy path is unchanged: continuations go to FlowKV and full prefix-cache restores stay enabled.
• Views. The server remembers the prompt it served for each conversation, matched by raw prompt prefix up to the old generation prompt, since most clients send no session id. The next turn serves that view plus the new turns, and the target restores its prefix-cache snapshot of the view.
• Snapshot hint. The view asks for its snapshot at the start of its generation prompt (PreparedPrompt::snapshot_cut); the default second-to-last boundary lands on the system prompt of a first turn.
• Recall. The fresh compression still runs each turn. Segments it keeps for the new question that the view lacks (CompressResult::kept_spans) are recalled as excerpts at the start of the new user turn, after everything cached, so the view always contains the fresh selection. Agent steps (assistant call plus tool output) append without recalling.
• Recall gate. Recall takes every out-of-view segment the new question attends to at lift >= PFLASH_CHAT_RECALL_MIN_LIFT (default 2; lift = mass per token relative to uniform attention); there is no token cap. A question that needs more than a third of a fresh selection rebuilds the view from it instead. On the development chats (Jev-judged): gate 2 212/252 against 200 at 4 and 197 at 8 on the query-layout chats (full prefill 231), 37 against 37 of 42 on the original chats, ~190 recalled tokens per turn, later-turn TTFT 1.01 s (full prefill 1.42 s). Non-finite head scores (seen once, not reproduced) rescore the prompt from scratch before failing.
• Rebuild. The view is rebuilt from the fresh prompt when it grows past twice the fresh prompt or past the context. PFLASH_CHAT_VIEW=0 serves the fresh compression every turn.
• Live on the R9700 (27B target, 0.8B drafter, keep 0.1, 20.6K-token conversation, three turns): all answers correct. Turn 2 recalls the segment turn 1 dropped. Target prefill falls from 7.7 s to 2.8 s on turn 2 and from 2.7 s to 0.9 s on turn 3. The drafter still reads the whole history each turn.

Dialogue skeleton and history queries

• Under strict selection every other turn keeps its role header, and user turns and assistant answers up to PFLASH_CHAT_SKELETON_TOKENS (default 256) stay whole. Like developer text, the skeleton is scored as context when it alone would not fit.
• The tails of the last PFLASH_CHAT_HISTORY_QUERIES (default 3) earlier user turns score the context alongside the query (CompressRequest::history_query_spans), their masses mixed in at weights 1/2, 1/4, 1/8. Drafter sessions keep the rows of up to 8 recent query windows.
• Live (20.6K tokens, 3 turns): turn 2 recalls 64 tokens instead of 1472 and takes 1.0 s instead of 3.9 s; all answers correct.

usage.timings.pflash

• Compressed responses report compress time, drafter input, kept and compressed tokens, effective keep ratio, query rule, history-query count, drafter session resume, new tokens and forward time, and the view outcome (fresh, continue, rebuild or repeat, with served, reused, delta, recalled and fresh tokens).

Drafter scoring sessions

• The strict scorer keeps up to PFLASH_DRAFTER_SESSIONS (default 2) per-conversation sessions: a blocks-0..14 cache, the block-15 NoPE keys, the probe's raw logits, the last query window's rows, and a recurrent-state checkpoint 64 tokens before the prompt end.
• A prompt sharing a session's prefix runs only its new tokens through the drafter and scores against every stored key.
• Sessions live with the loaded drafter, so they need --draft-residency persistent (plus --prefill-skip-park where both models fit).
• Live: with 20.6K tokens and three turns, drafter forward goes 3.9 s → 0.04 s on turn 2 and 3.3 s → 0.13 s on turn 3. Selections, served prompts and answers match scoring from scratch, and turns take 3.9 s and 3.3 s instead of 8.8 s and 6.2 s. An agent step reuses the stored query rows and runs only the new tool output.

Adaptive keep ratio (adaptive_keep_ratio.h): a new session starts from the configured ratio for its prompt length (--prefill-keep-ratio / --prefill-curve) instead of a fixed 10 %; the DFlash-acceptance controller then moves it within 2.5–20 %.

IPC daemon (pflash_drafter_ipc*.cpp/h, pflash_drafter_ipc_daemon.cpp, pflash_types.h, model_backend.h): drafter architecture, probe and head paths and the selection config travel over the drafter IPC; pflash_daemon runs the drafter standalone.

Tests: test_pflash_selection.cpp (selection modes, mandatory spans, density vs sum, probe segments, fallback, parser defaults), test_server_unit.cpp (chat-tail span extraction for Qwen/DeepSeek/open-ended/marker-less prompts, query mapping, required text, strict continuation ownership, FlowKV fallback, trace), test_pflash_drafter_ipc.cpp (config transport), test_bandit_integration.cpp.

Not in this PR

• No model files: the drafter, probe and head GGUFs are loaded from paths given on the command line / environment.
• The Qwen3.5 attention runs dense through ggml_flash_attn_ext; the block-sparse FlashPrefill kernels still dispatch head dimension 128 only.

Known limitations

• The scorer query width is no longer capped at 8 tokens on the drafter IPC (the previous validator); widths 1–512 round-trip, 0 and negatives fail closed.
• PFlash never compresses the system prompt: a system prompt that alone does not fit the context fails the request, and one that fits is kept whole (a document pasted into it is not compressed). Developer messages and tool definitions that alone would not fit lose their pin and are scored like any other context.
• An explicit pflash_query far from the end of its own user turn makes the kept suffix exceed the budget, and the request fails the same way.
• The remote drafter IPC and the running-max scorer (alone or in the split) keep the old kept-suffix contract, so agent-loop turns after the query are not scored there. The remote drafter reports no kept spans, so multi-turn views are off with it.
• Multi-turn: a large new tool output that would more than double the view triggers a rebuild, so it is compressed on arrival instead of kept verbatim. Drafter sessions are freed with the drafter under the default residency.
• The document-mass prior and forced document headers from earlier commits were reverted: PFlash scores the prompt against the query and nothing else.

🤖 Generated with Claude Code