spec : add ngram-mod - #19164
Conversation
|
Wow, that is wild. |
| if (!ngram_mod && params_base.speculative.type == COMMON_SPECULATIVE_TYPE_NGRAM_MAP_MOD) { | ||
| ngram_mod = std::make_unique<common_ngram_mod>(params_base.speculative.ngram_size_n, 1024*1024); | ||
|
|
||
| params_base.speculative.ngram_mod = ngram_mod.get(); |
There was a problem hiding this comment.
I guess you have to do this way because unique_ptr doesn't accept forward declared struct?
If that's the case, probably using std::shared_ptr or std::optional can be a better hack
| void add(const int32_t * tokens); | ||
| int32_t get(const int32_t * tokens, int32_t offs) const; // return -1 if not found | ||
|
|
||
| uint16_t n; // ngram size to hash |
There was a problem hiding this comment.
in multiple places in the code, we need to cast this to size_t, so I think it's probably better to use size_t
| uint16_t n; // ngram size to hash | |
| size_t n; // ngram size to hash |
Can EOG/EOS token a good criteria? ngram can be different between user message and assistant message |
EOG/EOS seems way too often. The hash container can store a lot of ngram hashes (hundred thousands with the current size) before collisions start to occur. I'm thinking more about logic such as: if more than |
7ef5b95 to
a9a076f
Compare
| std::vector<common_ngram_mod_ext_entry> entries; | ||
| }; | ||
|
|
||
| using common_ngram_mod_ext_ptr = std::unique_ptr<common_ngram_mod_ext>; |
| std::vector<entry_t> entries; | ||
| }; | ||
|
|
||
| using common_ngram_mod_ptr = std::unique_ptr<common_ngram_mod>; |
|
Looks like llama-bench doesn't know about --spec-type ngram-mod param. |
|
It seems this PR has an additional positive side effect: in the case of GPT-OSS in high mode, when the model falls into a reasoning loop, it can now recover much faster. Token generation jumps to around 200, and the model even produces a meaningful result. |
|
@MikeLP This does not affect @characharm Yes, I also noticed that. Overall, I think this speculator can become enabled by default in |
Is this example for a MoE or a dense model? I have no intuitive feel for what constitutes 'small n' or a 'long draft'. I assume the optimal value depends on both model, model architecture and task at hand. |
|
On the same prompt (just to repeat in verbatim 200 lines of given source code) I sometimes see a draft acceptance rate of 0, while on most other runs it's 0.90+ on gpt-oss-120b with Below logs of a bad case followed by a good case. (I also observed a good case right after starting llama-server, so it's not like that the first request is always "bad"). Log |
|
I’m experimenting with the n/min/max settings, but I don’t understand the balance yet. Does a large min–max range hurt us somehow? Qwen 30B and settings from the post: DetailsI see accept: |
|
Larger ngram size and larger drafts increase the chances that we will draft only when the LLM is repeating an existing text. Basically, we are trying to detect long repeating blocks without doing exhaustive searches. So unless your use case involves such repeating blocks of text, this method won't help. Yes, the |
Do I understand correctly that
means that the total "cost" of ngram_mod was only about 5 ms? My point is: should I try to increase that time by changing --spec-ngram-size-n / --draft-min / --draft-max, since even 500 ms still wouldn’t be noticeable? |
|
Just an idea to have a more consistent and sustained speedup behavior/avoid disadvantages of early low acceptance streaks in the current pruning mechanism: track for each ngram in the pool a capped score, initially set to 1 on insert. If an ngram was used successfully in a draft, count it up. If the draft was rejected count it down. On streaks remove all ngrams smaller or equal 0. Not sure if it's important to keep occupancy below a certain threshold. |
|
It works pretty well in OpenCode (GLM 4.7 Flash with thinking enabled), but I’m not sure if it’s real or placebo. I assume that a draft acceptance rate above 0.1 indicates some speedup. (I see also >0.5) |
|
To add to my message #19231 I think there still a problem. At start : and after regenrate : 
|
This is one of the ideas leading to the vector |
I see the same thing. |
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
|
Not a bug, just something I noticed. When the prompt contains an uploaded/pasted file with CRLF line endings, the ngrams often don't get accepted (even if task is repeating file verbatim) because models prefer LF endings. Running with: llama-server -m Devstral-2-123B-Instruct-2512-UD-Q5_K_XL-00001-of-00002.gguf --no-mmap --temp 0.15 --port 55553 --metrics --min-p 0.01 -c 32768 --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 32 --draft-max 48build: 7992 (612db61) with GNU 13.3.0 for Linux aarch64 Stats for CRLF file + prompt "Repeat verbatim." (temp set to 0 in UI) Stats for LF file + prompt "Repeat verbatim." (temp set to 0 in UI) |
|
As this isn't really clearly stated anywhere: |
…nch) Adds a self-contained struct modeled on llama.cpp's ngram_mod (PR ggml-org/llama.cpp#19164): - Configurable n (key = n-1 tokens, value = next token) - Open-addressing hash with overwrite-on-collision, 4 MB default - predict_chain() walks tail+predicted recursively until hash miss - Self-reset heuristics: occupancy > 25% OR 3 rounds < 50% accept - 4 unit tests covering observe, get, chain, collision, reset Microbench `chained_ngram_microbench` evaluates prediction accuracy against the existing bigram NgramCache on real DFlash-emitted token streams (load tokens.txt → train on first half → predict second). Results on /tmp/tokens-baseline.txt (LRU 1501-token loop): top-1 accuracy mean chain hit Δ vs bigram bigram: 80.0% n/a — chained n=3: 78.1% 3.7 / 16 -1.9% chained n=4: 90.4% 6.7 / 16 +10.4% chained n=6: 99.2% 15.0 / 16 +19.1% At n=6, 39 of 45 sampled chains hit all 16 tokens — the cache essentially memorizes structural loops. On non-loop content, bigram and chained both match the trained draft's predictions; the lift appears specifically on repetitive structures the bigram override misses (boilerplate, recurring variable names, structured-list tails). NOT YET integrated into spec_step_dflash. Drop-in into the existing override loop is ~30 lines (replace bigram (a,b) lookup with chained get(tail) where tail is the rolling (n-1)-token suffix), but deferring full plumbing pending workload-realistic measurements beyond LRU-loop content. References: - llama.cpp/common/ngram-mod.{cpp,h} — open-addressing hash impl - llama.cpp/common/speculative.cpp:642-756 — ngram_mod state machine - Existing engine/src/speculative.rs:1438 NgramCache (bigram, kept)
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
* spec : add ngram-mod * cont : simplify + keep track of occupancy * cont : cleanup * cont : move initialization to common/speculative * cont : cleanup * cont : cleanup * cont : fix
ngram-simple, ngram-map-k and ngram-map-k4v come from common/ngram-map.cpp (common_ngram_simple_draft, common_ngram_map_draft, common_ngram_map_accept; upstream PR ggml-org/llama.cpp#18471). ngram-mod comes from common/ngram-mod.cpp (common_ngram_mod's idx/add/get/reset) plus common/speculative.cpp's common_speculative_impl_ngram_mod, which owns its begin/draft_one/accept policy (PR ggml-org/llama.cpp#19164). spec::SelfSpeculator dispatches all of them, and the pre-existing ngram, behind one handle so the engine's decode path carries no per-variant branch. The ports keep upstream's index convention rather than tidying it. llama.cpp passes (tokens, sampled) — committed history plus the token just sampled, kept apart — where a speculator here owns one append-only history whose tail is that sampled token, so upstream's cur_len is history.len() - 1 and every tokens[x] below cur_len is history[x] unchanged. Index 0 keeps its double duty as "no match". A serving sequence never loses tokens, which makes common_ngram_map_begin's shrink-cleanup branches dead code here; only its size_last_begin and idx_last_check bookkeeping is ported, marked at the generation boundary by the first single-token observe. ngram-mod's table is shared by every sequence a worker serves, as upstream shares one common_ngram_mod across a context's sequences: what one request teaches the table, a concurrent request drafts from. That is why the dispatch splits into a per-worker SelfSpecFactory holding an Rc<RefCell> and a plain-data SelfSpecConfig that crosses the thread boundary — a worker's speculators are all created and driven on its single runtime thread. Both of upstream's souring defenses are ported: reset at 25% occupancy at a sequence's start, and reset after five consecutive rounds with under a quarter of the draft accepted. Three upstream behaviours are preserved deliberately, because changing them would change what is drafted: ngram-simple drops any draft shorter than its own size_n (so a size_m below size_n can never draft); a ngram-map-k key whose draft is fully rejected has its n_accepted pinned at 0 and goes quiet; and ngram-mod never verifies a table hit against the history, so a collision drafts a wrong-context token that the target model's verify pass then rejects. Verified against upstream directly rather than by reading it. llama.cpp's own ngram-map.cpp and ngram-mod.cpp were compiled against a shim (llama_tokens, no-op logging) and driven through the same schedule as these ports — observe the prompt, observe a token, draft, greedy-accept against the real stream, commit the accepted run — over 168 (variant, n, m, min_hits, alphabet, seed) combinations, including upstream's own 12/48 and 24/48/64 defaults over 6000-token histories. Every drafted token matched, with hundreds of non-empty drafts per variant (400 of 400 steps for ngram-mod at n_match 8) so the comparison is not of two silent drafters. The harness is scratchpad-only: it copies llama.cpp sources, so it is not committed here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKxiqVjnNJuh6VQuSkd7Xz
13 participants
cont #18471
Add basic ngram hasher for speculative decoding:
ntokens and pick the next token from the storageSome characteristics:
mis not fixed)Currently, a single hash pool is shared across all server slots, so different requests can benefit from each other.
Sample usage:
Applications:
Example:
spec-mod-0.mov
TODO: