feat(vision): speculative decoding for image requests (Qwen3.8 and DeepSeek V4 Vision)

[davide221]

What

Image requests now decode with the model's drafter, like text requests, for both vision models. Until now every image request decoded one token at a time.

Qwen3.5 / Qwen3.8 (DFlash2)

Image tokens take 2D rotary positions, so after an image the rotary position runs rope_delta_ ahead of the KV position. Plain decode already applied that offset, but the DFlash verify target did not. So the HTTP layer and the backend both forced image requests to plain decode.

• Qwen35DFlashTarget reads the backend's per-request rope_delta_ and shifts the M-RoPE positions of chain and tree verify by it. The offset is zero for text, so text requests are unchanged.
• The blanket plain-decode force for image requests is removed.

One R9700, Qwen3.8-27B-IQ4_XS-pure.gguf + Q8_0 projector, 12 images (6 ChartQA, 6 AI2D), 256-token answers, greedy, streamed:

engine	first token	whole answer	decode
this PR	0.55 s	4.03 s	76 tok/s
llama.cpp 7ab4ee7 + the same DFlash2 drafter	0.68 s	5.54 s	53 tok/s
main	0.53 s	7.58 s	36 tok/s
llama.cpp, no drafter	0.63 s	8.36 s	33 tok/s

Faster than llama.cpp with the drafter on all 12 images, from 1.21x to 1.58x.

Quality:

• 220-question image eval: 188, the same as main, with 218 answers identical.
• One to four images per request: 9/9.

DeepSeek V4 Flash Vision (DSpark)

The image prefill graph takes no DSpark capture hooks, so image requests had no drafter features. This PR makes three changes:

• Image chunks end at their last image, so the text after the image prefills and captures as ordinary chunks.
• The feature window is cleared at each image chunk, so the drafter always reads one contiguous tail.
• The !req.images guard on the DSpark path is dropped.

Strix Halo alone, Vision-Exp ROCMFPX MIX, the published DSpark launch, the same 12 images:

	first token	whole answer	decode
this PR	4.90 s	13.3 s	30.4 tok/s
main	4.13 s	15.7 s	22.1 tok/s

Feature capture during prefill adds about 0.7 s before the first token, the same cost text requests already pay. One-word answers therefore come back slightly later.

Quality:

• 220 questions: 181 (AI2D 86, ChartQA 55 and 40), with 209 answers identical to plain decode.
• One to four images: 9/9.
• Text decode speed unchanged.

Exactness

Greedy output with a drafter is not byte-identical to plain decode on this build, for text or images: some answers change a word after a few hundred characters. llama.cpp's DFlash run shows the same against its own plain decode.

🤖 Generated with Claude Code

[cubic-dev-ai]

1 issue found across 6 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/deepseek4/deepseek4_backend.cpp">

<violation number="1" location="server/src/deepseek4/deepseek4_backend.cpp:2849">
P2: For prompts that end with an image (the standard "describe this image:" shape) or have less than `n_swa` text tokens after the last image, `spec_feat_window_` ends up empty or short at the end of prefill, yet the PR now routes these requests into `run_deepseek4_dspark_spec_decode` (the `!req.images` guard was removed). In `run_deepseek4_dspark_spec_decode` this yields `win_len = 0` / `feat_count = 0` — the drafter's feature ring (`feat_win`) is fully zero-initialized and `ctx_len = 0` on the first steps, so the drafter drafts with no context at all until its own emitted features fill the window. Previously image requests always fell back to AR decode, so this state is newly reachable. This is both a quality/accept-rate risk (poor drafts on the first steps for the most common vision prompt shape) and a robustness question: verify that a zero-context draft/verify graph is legal (a zero-row SWA window). Consider keeping an AR fallback when `win_len == 0` (or when the trailing text after the last image is shorter than `n_swa`), e.g. guarding the spec path with a non-empty `spec_feat_window_`.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: For prompts that end with an image (the standard "describe this image:" shape) or have less than n_swa text tokens after the last image, spec_feat_window_ ends up empty or short at the end of prefill, yet the PR now routes these requests into run_deepseek4_dspark_spec_decode (the !req.images guard was removed). In run_deepseek4_dspark_spec_decode this yields win_len = 0 / feat_count = 0 — the drafter's feature ring (feat_win) is fully zero-initialized and ctx_len = 0 on the first steps, so the drafter drafts with no context at all until its own emitted features fill the window. Previously image requests always fell back to AR decode, so this state is newly reachable. This is both a quality/accept-rate risk (poor drafts on the first steps for the most common vision prompt shape) and a robustness question: verify that a zero-context draft/verify graph is legal (a zero-row SWA window). Consider keeping an AR fallback when win_len == 0 (or when the trailing text after the last image is shorter than n_swa), e.g. guarding the spec path with a non-empty spec_feat_window_.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/deepseek4/deepseek4_backend.cpp, line 2849:

<comment>For prompts that end with an image (the standard "describe this image:" shape) or have less than `n_swa` text tokens after the last image, `spec_feat_window_` ends up empty or short at the end of prefill, yet the PR now routes these requests into `run_deepseek4_dspark_spec_decode` (the `!req.images` guard was removed). In `run_deepseek4_dspark_spec_decode` this yields `win_len = 0` / `feat_count = 0` — the drafter's feature ring (`feat_win`) is fully zero-initialized and `ctx_len = 0` on the first steps, so the drafter drafts with no context at all until its own emitted features fill the window. Previously image requests always fell back to AR decode, so this state is newly reachable. This is both a quality/accept-rate risk (poor drafts on the first steps for the most common vision prompt shape) and a robustness question: verify that a zero-context draft/verify graph is legal (a zero-row SWA window). Consider keeping an AR fallback when `win_len == 0` (or when the trailing text after the last image is shorter than `n_swa`), e.g. guarding the spec path with a non-empty `spec_feat_window_`.</comment>

<file context>
@@ -2823,10 +2824,29 @@ int DeepSeek4Backend::do_prefill(const std::vector<int32_t> & tokens,
+            }
+            // The drafter reads the newest rows as one contiguous window, so
+            // rows from before an image cannot sit next to rows after it.
+            if (chunk_has_image) spec_feat_window_.clear();
         }
 
</file context>

[davide221]

Fixed in c2b02a3: when an image request has no text after its last image to seed the drafter window, DeepSeek4 falls back to plain decode (image_without_draft_context). The chunk boundary now comes from vision::last_image_end_in, with unit checks.