🚀 [v0.3.49] Release Note: DFlash2 / DFlash / DSpark Speculative Decoding Support, and MTMD Video Support

[JamePeng]

llama-cpp-python 0.3.49

DFlash2 Support, Expanded Speculative Decoding, and MTMD Video Updates

Version 0.3.49 is another incremental release focused on expanding speculative decoding support, improving multimodal workflows, and tightening runtime behavior around caching, timing, and interrupted generation.

The biggest change in this release is the addition of DFlash2 support, together with broader support for DFlash and DSpark model-backed speculative decoding.

DFlash2 Support

I added support for DFlash2 selector-based speculative decoding, including:

• packed selector-lattice traversal
• predecessor-dependent transition scoring
• confidence filtering through draft_p_min
• checkpoint-aware rollback behavior
• unmasked NextN outputs for DFlash2
• M-RoPE-aware position handling
• support for DFlash2 NVFP4-scale draft models

The implementation also exposes DFlash2 selector metadata, automatically detects compatible draft models, and keeps the existing DFlash v1 and DSpark execution paths separate where required.

DFlash and DSpark Speculative Decoding

This release also adds text-only DFlash and DSpark decoding support.

I refactored the common model-backed speculative draft logic into a shared engine so MTP, DFlash, DFlash2, and DSpark can reuse the same core infrastructure for:

• model loading
• vocabulary validation
• context creation
• backend sampling
• candidate extraction
• feature injection
• checkpoint management
• rollback
• resource cleanup

DFlash and DSpark can now load external draft sidecar models, gather target-layer features, construct non-causal mask blocks, and use backend sampling where supported.

I also added a dedicated throughput benchmark so ordinary decoding can be compared directly against DFlash and DSpark drafting, including acceptance rate, phase timings, rollback statistics, and deterministic-output checks.

At this stage, MTP, DFlash, DFlash2, and DSpark speculative decoding are all text-only.

Multimodal speculative decoding is not supported yet. In particular, direct MTMD embedding batches are currently rejected because multimodal target-position mapping and draft-context integration are not implemented for these speculative paths.

Improved Speculative Recovery and Cache Handling

A large part of this release focuses on making speculative decoding safer when generation is interrupted.

The runtime now tracks speculative verification as an in-flight transaction and reconciles target and draft contexts when generation exits before the normal acceptance or rollback phase.

Depending on the model type, recovery can use:

• native rollback
• hybrid checkpoints
• replay of committed inputs
• direct transformer cache truncation
• full reset as a final fallback

Hybrid checkpoint handling now also tracks native memory positions so partial rollback can correctly remove the remaining attention-memory suffix.

MTP context sizing and verification limits were aligned with the initialized target context, and proposal lengths are now capped so verification batches remain atomic.

MTMD Video Support

MTMD helper bindings were updated with the latest video-related options.

The Python layer now exposes:

• video FPS configuration
• ffmpeg binary directory
• timestamp interval settings
• helper initialization options
• corrected video pointer and buffer ctypes signatures

The implementation also validates both ffmpeg and ffprobe and retains the encoded path for ctypes pointer lifetime safety.

I added a standalone MTMD video chat example and extended the fallback multimodal chat schema to accept video content.

https://github.com/JamePeng/llama-cpp-python/blob/main/examples/high_level_api/mtmd_video_chat.py

The documentation now includes guidance for:

• Gemma 4 video-compatible schemas
• ffmpeg setup
• FPS configuration
• context sizing
• resource considerations for long videos

Qwen3.8 has also been added to the documented multimodal model list, with GenericMTMDChatHandler as the recommended handler.

Better Multimodal and Context Performance Timing

Multimodal prompt evaluation timing is now reported separately from text generation.

Text, image, and audio prompt chunks are measured within the same request-scoped interval, with explicit synchronization only at the required phase boundaries.

I also refactored context performance APIs so completion and embedding counters are reset independently of logging verbosity, and no_perf is now respected consistently.

LlamaContext now provides cleaner wrappers for:

• backend sampler attachment
• sampled outputs
• performance metrics
• synchronization and buffer lifetime semantics

This also simplifies speculative decoding and embedding timing paths.

API Compatibility Improvements

The high-level completion APIs now expose:

ignore_eos=True

which allows generation to continue past EOG tokens when explicitly requested.

I also added presence_penalty as a compatibility alias for the existing present_penalty option across:

• completion
• callable API
• chat API
• OpenAI-compatible server request models

Internally, penalty_present remains the sampler-level canonical field.

There are apparently still users who prefer the presence_penalty naming choice, so this compatibility alias is now supported.

Cache and Sampling Fixes

Several smaller correctness fixes are included:

• corrected RAM accounting when replacing existing cache entries
• fixed "trie" server cache selection so it creates LlamaTrieCache
• aligned dry_penalty_last_n defaults with llama.cpp at 64
• preserved 0 as disabled and -1 as full-context behavior
• fixed a Jinja typo in Gemma4ChatHandler, reported by @craftingmod

The obsolete macOS Intel CI jobs have also been removed.

llama.cpp Synchronization

Updated llama.cpp to:

ggml-org/llama.cpp
commit 9723942adc518b43c4b95dc4dce6906903eb5e09

This update includes:

• load_mode support
• Qwen3.8-Flash-Next
• the DFlash2 NVFP4 fix patch (dflash: pass missing NVFP4 scales to attention operations ggml-org/llama.cpp#28000)

The llama, MTMD, and GGML API bindings were synchronized on:

2026-08-31

Overall, 0.3.49 mainly expands the speculative decoding stack with DFlash, DFlash2, and DSpark support, while also improving MTMD video handling, cache recovery, context timing, and runtime robustness.

For now, speculative decoding remains text-only across MTP, DFlash, DFlash2, and DSpark. Multimodal speculative decoding will require additional work around MTMD embedding and position handling.

Full comparison:

7562297...1966df5

JamePeng