🕵️Community Testing Request: DFlash/DFlash2/DSpark Speculative Decoding Feedback🤗

[JamePeng]

Community Testing Request: DFlash/DFlash2/DSpark Speculative Decoding Feedback

Hi everyone,

I am preparing experimental DFlash、DFlash2 and DSpark speculative decoding support for llama-cpp-python.

This functionality has not been officially released yet and should currently be treated as part of the upcoming 0.3.49-preview development version. I would appreciate community testing across different models, quantizations, GPUs, backends, and workloads before the final 0.3.49 release.

More Information see wiki: Llama Speculative Decoding

Current implementation

The preview currently provides:

• External DFlash draft model support
• External DSpark draft model support
• Target-layer feature extraction and fusion
• Non-causal mask-block draft decoding
• DFlash probability filtering
• DSpark acceptance-confidence filtering
• Backend-side candidate sampling
• Native recurrent snapshot rollback where available
• Partial on-device checkpoint fallback for recurrent or hybrid draft models
• Accepted-prefix replay after checkpoint restoration
• Detailed acceptance, checkpoint, rollback, and throughput statistics
• A dedicated ordinary-versus-speculative benchmark example

The same LlamaDFlashDecoding engine handles both DFlash and DSpark, with behavior selected from the configured algorithm and draft GGUF metadata.

Important preview limitations

The current implementation is:

• Text-only
• Limited to one sequence using seq_id=0
• Dependent on a compatible external DFlash or DSpark GGUF
• Not compatible with MTMD or multimodal embedding batches yet
• Still subject to API, validation, checkpoint, and performance changes before the final 0.3.49 release

Please use it in a testing environment rather than a production deployment.

Models tested so far

Initial testing currently covers compatible:

• Qwen3.6 target and DFlash sidecar pairs
• Qwen3.8 target and DFlash2/DSpark sidecar pairs

Support for other compatible architectures may already work, but needs more validation. In particular, feedback for Nemotron DFlash, different DSpark variants, hybrid/recurrent models, and reduced-vocabulary sidecars would be very useful.

How to test DFlash

python -m examples.high_level_api.high_level_api_dflash_dspark_speculative \
  --algorithm dflash \
  --model path/to/target.gguf \
  --draft-model path/to/dflash.gguf \
  --draft-tokens 7 \
  --warmup-tokens 128 \
  --max-tokens 512 \
  --runs 3

How to test DFlash2

python -m examples.high_level_api.high_level_api_dflash_dspark_speculative `
  --algorithm dflash2 \
  --model path/to/target.gguf \
  --draft-model path/to/dflash2.gguf \
  --max-tokens 512 \
  --runs 3 \
  --warmup-tokens 64 \
  --n-ctx 4096 \
  --n-batch 512 \
  --n-ubatch 512 \
  --draft-tokens 7 \
  --draft-n-min 0 \
  --draft-p-min 0 \

For fixed-length throughput testing, add:

--ignore-eos

How to test DSpark

python -m examples.high_level_api.high_level_api_dflash_dspark_speculative \
  --algorithm dspark \
  --model path/to/target.gguf \
  --draft-model path/to/dspark.gguf \
  --draft-tokens 7 \
  --warmup-tokens 128 \
  --max-tokens 512 \
  --runs 3

Run the example with -h to see all available options:

python -m examples.high_level_api.high_level_api_dflash_dspark_speculative -h

Draft-length testing

The best draft length depends on the model, draft block size, GPU, backend, quantization, prompt, and acceptance rate.

If possible, please compare several values:

1, 2, 3, 4, 5, 6, 7, 8

A longer block is not automatically faster. Please compare final sustained throughput rather than relying only on the acceptance rate.

Feedback requested

Please include as much of the following information as possible:

llama-cpp-python version or commit:
Operating system:
Python version:
Backend:
GPU:
GPU memory:
Target model:
Target quantization:
Draft model:
Algorithm: DFlash / DSpark

n_ctx:
n_batch:
n_ubatch:
draft_n_max:
draft_p_min:
backend sampling enabled:
ignore_eos:

Prompt or workload type:
Generated token count:
Ordinary throughput:
Speculative throughput:
Speedup:

Draft tokens proposed:
Draft tokens accepted:
Acceptance rate:
Verification steps:
Rollback count:
Target decode/sync time:
Draft time:
Process time:
Checkpoint capture/restore time:

Outputs deterministic across repeated runs:
First divergence from ordinary decoding:
Errors or warnings:

The full summary printed by the example is also welcome.

Correctness notes

For the cleanest comparison, use deterministic sampling:

temperature = 0
top_k = 1
repeat_penalty = 1

Even with deterministic sampling, ordinary and speculative output may diverge because target verification uses a different batch shape, which can affect floating-point tie-breaking. Please report the first divergent token instead of reporting only whether the complete outputs match.

More important warning signs include:

• Different outputs across repeated runs with the same configuration
• Invalid tokens
• Draft cache position errors
• Failed memory_seq_rm() operations
• Checkpoint restoration failures
• Output corruption after rejection
• Increasing memory usage between runs
• Crashes during close() or interpreter shutdown
• Large unexplained process() or synchronization time

Performance feedback

I am especially interested in:

• Which draft length provides the best throughput
• Whether backend sampling improves large-vocabulary models
• How acceptance changes between structured and low-repetition prompts
• The cost of target decode and synchronization
• Native rollback versus on-device checkpoint behavior
• Performance on CUDA, ROCm, Vulkan, Metal, and CPU backends
• Results at 512, 1024, and 2048 generated tokens

Thank you to everyone willing to test this preview. Your results will help improve model compatibility, correctness, rollback behavior, and performance before the final 0.3.49 release.

— JamePeng

[craftingmod]

[Setup]
llama-cpp-python version or commit: 7b78e397789191cff45fb2f75b48b6005e2bbf51 (0.3.49-nightly)
Operating system: Windows 11 26H2 (26300.9168)
Python version: 3.13.12
Backend: CUDA 13.2 (SM86)
GPU: NVIDIA RTX 3090 Ti
memory: 24GB VRAM (with 64GB DDR4 RAM)
Target model: Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf (from meta official)
Target quantization: Q4_K_M
Draft model: dflash-Muse-Glimmer-30B-Q4_K_M.gguf
Algorithm: DFlash

[test Logs]
llama-cpp-python on git main [!] via cmake v4.4.2 via py v3.13.12 (llama-cpp-python) took 1m48s 
> python -m examples.high_level_api.high_level_api_dflash_dspark_speculative `
. --algorithm dflash `
. --model "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" `
. --draft-model "dflash-Muse-Glimmer-30B-Q4_K_M.gguf" `
. --draft-tokens 15 `
. --warmup-tokens 128 `
. --max-tokens 512 `
. --runs 5
Target model   : Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf
Draft model    : dflash-Muse-Glimmer-30B-Q4_K_M.gguf
Algorithm      : dflash
Prompt         : 'Write a detailed 1800-word technical article about speculative decoding. Use ten titled sections, include pseudocode, performance analysis, limitations, and a conclusion. Do not finish early.'
Measured runs  : 5 x 512 tokens per mode
Warmup         : 128 tokens per mode
Draft settings : max=15, p_min=0
Backend sample : enabled
Ignore EOG     : False
Limitations     : text-only, seq_id=0
[llama-cpp-python] loaded bundled OpenMP runtime: libomp140.x86_64.dll
[llama-cpp-python].find_library: loaded library from ggml-base.dll
[llama-cpp-python].find_library: loaded library from ggml.dll
[llama-cpp-python].find_library: loaded library from llama.dll
load: setting token '<|message|>' (200023) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|start|>' (200022) attribute to USER_DEFINED (16), old attributes: 8
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
load: setting token '<|message|>' (200023) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|start|>' (200022) attribute to USER_DEFINED (16), old attributes: 8
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
load: setting token '<|message|>' (200023) attribute to USER_DEFINED (16), old attributes: 8
load: setting token '<|start|>' (200022) attribute to USER_DEFINED (16), old attributes: 8
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
llama_kv_cache_iswa: using full-size SWA cache (ref: https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)

[ordinary]
  model load          : 5.703 s (not benchmarked)
  run 1              :  512 tokens / 11.015 s = 46.48 token/s
  run 2              :  512 tokens / 11.021 s = 46.46 token/s
  run 3              :  512 tokens / 11.027 s = 46.43 token/s
  run 4              :  512 tokens / 11.058 s = 46.30 token/s
  run 5              :  512 tokens / 11.037 s = 46.39 token/s
  aggregate           : 46.41 token/s
  median run          : 46.43 token/s

[dflash]
  model load          : 6.803 s (not benchmarked)
  run 1              :  512 tokens / 6.627 s = 77.26 token/s
  run 2              :  512 tokens / 6.626 s = 77.27 token/s
  run 3              :  512 tokens / 6.642 s = 77.09 token/s
  run 4              :  512 tokens / 6.738 s = 75.98 token/s
  run 5              :  512 tokens / 6.671 s = 76.75 token/s
  aggregate           : 76.87 token/s
  median run          : 77.09 token/s
  draft proposed      : 11850
  draft verified      : 11850 in 790 steps
  draft accepted      : 1770 (14.9%)
  draft rollbacks     : 775 (native: 775)
  target decode+sync  : 26.467 s
  draft phase         : 4.119 s
  process phase       : 2.063 s
  checkpoint          : 12.771 ms

[comparison: dflash vs ordinary]
  output tokens match : False
  runs deterministic  : True
  first divergence    : generated token 84
  speedup             : 1.656x
  throughput change   : +65.6%

I've tested draft-tokens in 1,2,3,5,7,15 and

15(65.6%) > 2 (about 40%) > 1 (about 39%) > else

was good :)

[JamePeng]

👌

[JamePeng]

@craftingmod 0-day Support DFlash2 selector and M-RoPE execution, you can make some tests for dflash2

[craftingmod]

Case of DFlash2

Setup

llama-cpp-python version or commit: 6220f97188143d9c828345c13df3384389bc344d (0.3.49-nightly)
Operating system: Windows 11 26H2 (26300.9278)
Python version: 3.13.12
Backend: CUDA 13.2 (SM86)
GPU: NVIDIA RTX 3090 Ti
memory: 24GB VRAM (with 64GB DDR4 RAM)
Target model: Qwen3.8-27B-Q4_K_M.gguf (ggml-org/Qwen3.8-27B-GGUF)
Target quantization: Q4_K_M
Draft model: Qwen3.8-27B-DFlash2-Q4_K_M.gguf (incoai/Qwen3.8-27B-DFlash2-GGUF)
Algorithm: DFlash2

Execute

llama-cpp-python on git main via cmake v4.4.2 via py v3.13.12 (llama-cpp-python) took 2m14s
> python -m examples.high_level_api.high_level_api_dflash_dspark_speculative `
. --algorithm dflash2 `
. --model "Qwen3.8-27B-Q4_K_M.gguf" `
. --draft-model "Qwen3.8-27B-DFlash2-Q4_K_M.gguf" `
. --max-tokens 512 `
. --runs 5 `
. --warmup-tokens 64 `
. --n-ctx 4096 `
. --n-batch 512 `
. --n-ubatch 512 `
. --draft-tokens 3 `
. --draft-n-min 0 `
. --draft-p-min 0

Execute Log

Target model   : Qwen3.8-27B-Q4_K_M.gguf
Draft model    : Qwen3.8-27B-DFlash2-Q4_K_M.gguf
Algorithm      : dflash2
Prompt         : 'Write a detailed 1800-word technical article about speculative decoding. Use ten titled sections, include pseudocode, performance analysis, limitations, and a conclusion. Do not finish early.'
Measured runs  : 5 x 512 tokens per mode
Warmup         : 64 tokens per mode
Draft settings : min=0, max=3, p_min=0
Backend sample : enabled
Ignore EOG     : False
Limitations     : text-only, seq_id=0
[llama-cpp-python] loaded bundled OpenMP runtime: libomp140.x86_64.dll
[llama-cpp-python].find_library: loaded library from ggml-base.dll
[llama-cpp-python].find_library: loaded library from ggml.dll
[llama-cpp-python].find_library: loaded library from llama.dll

[ordinary]
  model load          : 6.664 s (not benchmarked)
  run 1              :  512 tokens / 13.709 s = 37.35 token/s
  run 2              :  512 tokens / 13.352 s = 38.35 token/s
  run 3              :  512 tokens / 13.751 s = 37.23 token/s
  run 4              :  512 tokens / 13.590 s = 37.68 token/s
  run 5              :  512 tokens / 13.424 s = 38.14 token/s
  aggregate           : 37.74 token/s
  median run          : 37.68 token/s

[DFlash2]
  model load          : 7.701 s (not benchmarked)
  run 1              :  512 tokens / 8.420 s = 60.81 token/s
  run 2              :  512 tokens / 8.609 s = 59.47 token/s
  run 3              :  512 tokens / 8.754 s = 58.49 token/s
  run 4              :  512 tokens / 8.756 s = 58.48 token/s
  run 5              :  512 tokens / 8.423 s = 60.78 token/s
  aggregate           : 59.59 token/s
  median run          : 59.47 token/s
  draft range         : 0..3, p_min=0
  block size          : 8
  selector top-k      : 16
  nextn output        : unmasked
  backend sampling    : requested=True, active=False
  M-RoPE              : True
  draft proposed      : 2850
  draft verified      : 2850 in 950 steps
  draft accepted      : 1610 (56.5%)
  draft rollbacks     : 605 (native: 605)
  target decode+sync  : 36.199 s
  draft phase         : 4.775 s
  process phase       : 1.051 s
  checkpoint          : 20.078 ms

[comparison: DFlash2 vs ordinary]
  output tokens match : False
  runs deterministic  : True
  first divergence    : generated token 46
  speedup             : 1.579x
  throughput change   : +57.9%

Summary

Performance was checked with unofficial dflash2 model so this improvement could be different then adviced.

draft-tokens: 3 and 4 were best with +55%, while 7 is +15% change.

[JamePeng]

Thank you for your test feedback.

[JamePeng]

The Dflash 2 NVFP4 fix I submitted to the underlying llama.cpp repository has been merged; it now supports proper inference output for dflash/dflash2 draft models using NVFP4-scale quantization.

ggml-org/llama.cpp#28000