benchmarks/stock: llama.cpp with its own DFlash2 drafter (and why it doesn't speed up here)

[nathansutton]

Closes #42 — and instead of only relabelling the llama.cpp row, measures llama.cpp with its own DFlash2 drafter.

What was out of date

The stock-engine table said llama.cpp's speculation was "none for this model", and docs/benchmarks.md said there was no DFlash2 head in GGUF form. Both were true for build 10470, the build measured, and are not now:

• llama.cpp gained DFlash2 in build 10658. One correction to the issue: #27342 merged into the xsn/dflash2 feature branch, not master (its commit is not an ancestor of any release). It reached master as spec : add DFlash2 support (local convolution + candidate selector) (#27342) ggml-org/llama.cpp#27816 (b10f9ca58c), and b10658 is the first release that contains it (b10657 is one commit behind).
• incoai/Qwen3.8-27B-DFlash2-GGUF publishes the drafter in BF16, Q8_0 and Q4_K_M.

The new measurement

stock.py llama-dflash runs llama-server (build 10917) on the same UD-Q3_K_XL GGUF twice — serial, then with the Q4_K_M drafter (-md … --spec-type draft-dflash --spec-draft-n-max 7, chad's draft width) — on the same 512-token filler chad-bench tiles (tokenized by the server itself), greedy, cache_prompt off, 3 reps each after a warmup. It also records a llama-bench verify-width probe. M4 Pro 24 GB, nothing else resident, run twice (the numbers below are the committed second run; the first agreed within 0.1 tok/s).

Engine	Prefill (512-tok prompt)	Decode (128 tok)	Speculative decoding
llama.cpp llama-bench (stock, build 10470)	102 tok/s	10.9 tok/s	off in this benchmark
llama.cpp llama-server (build 10917), serial	97 tok/s	11.3 tok/s	off
llama.cpp llama-server (build 10917)	95 tok/s	11.1 tok/s	DFlash2 drafter (Q4_K_M GGUF)
chad, serial (CHAD_NO_DFLASH=1)	99 tok/s	18.1 tok/s	off
chad, default	98 tok/s	62 tok/s	DFlash2 block drafter

The drafter works and the speedup doesn't arrive. 96.5% of drafted tokens are accepted (110 / 114 per run, ~7.5 committed per round), and decode is 11.1 tok/s against 11.3 serial. The verify pass is why: llama.cpp's Metal path runs a batch of 1 / 2 / 4 / 8 / 16 tokens at 11.2 / 12.8 / 13.5 / 14.0 / 45.5 tok/s on this GGUF, so the 8-token verify alone costs ~6.4 serial steps and no acceptance rate can pay for the round. The jump at 16 is a kernel switch the round sits just below. chad's whole round at the same width, drafter included, costs ~2.2 serial steps (mlx_dflash.BLOCK_ROUND_COSTS).

I checked it isn't a measurement fault before writing it down: the drafter loaded and engaged (common_speculative_impl_draft_dflash: n_max=7 … block_size=8, acceptance from the server's own counters), all 128 tokens were generated on every rep, and round time (~0.67 s) is the probe's 0.57 s verify plus ~0.1 s — not a CPU-bound drafter.

Scope, as the docs now say it

This is this GGUF on this Mac, not llama.cpp's DFlash2 in general — llama.cpp's PR reports ~1.8× on an M5 Pro with a Q4_K_M target, and a different quant or chip can land on the other side of that kernel switch. Both drafted rows are ceilings on tiled code; the real-session numbers for chad stay in the footnote. Ollama is now described as measured without speculative decoding, not as lacking it.

Changes

• README.md, docs/benchmarks.md: the two new rows, the relabelled build-10470 row, the Ollama wording, and the stale "no GGUF head" explanation replaced with the measurement.
• benchmarks/stock/stock.py: the llama-dflash arm, table() renders it and prints the llama-bench build, STOCK_LLAMA_BIN so a newer build runs from an unpacked release without replacing the brew install the matrix grid's llama.cpp rows are pinned to. Server logs go to the temp dir, not _runs/.
• benchmarks/stock/_runs/llama-dflash.json: the record (per-rep timings, probe, flags).
• benchmarks/stock/README.md: the new command and how to point it at a build.

The chad rows are unchanged (chad 2.0.0, same machine); nothing since touched the decode path. ruff clean on stock.py.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qsqne39Q9TQuEoKWukmSpT