As stated in the title, the logic of generate() method causes an error.
The following snippet is the simplest reproducible case;
from llama_cpp import Llama
llm = Llama(
model_path="Qwen3.5-4B-Q4_0.gguf", # any model with `is_hybrid==True`
n_ctx=2048,
ctx_checkpoints=2,
checkpoint_on_device=False,
)
# 9-token prompt -> first checkpoint at pos=8
output = llm("The quick brown fox jumps over the lazy dog", max_tokens=5)
# 8-token prompt -> raises RuntimeError
output2 = llm("The quick brown fox jumps over the lazy")As you know, generate () wanna evaluate at least one token to obtain logit. But in this case, The position of checkpoint in the first turn accidentally coincide with the length of the second prompt, which results in the error.
A workaround I come up with is to use the "second best" checkpoint instead of the best checkpoint.
With help of my AI agent, I made a patch to make it work;
llama_cache.py:
- def find_best_checkpoint(self, tokens: List[int], seq_id: int = 0) -> Optional[HybridCheckpoint]:
+ def find_best_checkpoint(
+ self, tokens: List[int], seq_id: int = 0, *, require_remainder: bool = False
+ ) -> Optional[HybridCheckpoint]:
...
for cp in self.checkpoints:
- if cp.seq_id != seq_id or cp.pos > len(tokens):
+ pos_limit = len(tokens) if not require_remainder else len(tokens) - 1
+ if cp.seq_id != seq_id or cp.pos > pos_limit:
continuegenerate() method in llama.py:
- best_ckpt = self._hybrid_cache_mgr.find_best_checkpoint(original_tokens, 0)
+ best_ckpt = self._hybrid_cache_mgr.find_best_checkpoint(
+ original_tokens, 0, require_remainder=True,
+ )
This keeps the model from reusing checkpoint whose position is equal to new prompt without any potential side effects, and it succeeds the next generation.
As stated in the title, the logic of
generate()method causes an error.The following snippet is the simplest reproducible case;
As you know,
generate ()wanna evaluate at least one token to obtain logit. But in this case, The position of checkpoint in the first turn accidentally coincide with the length of the second prompt, which results in the error.A workaround I come up with is to use the "second best" checkpoint instead of the best checkpoint.
With help of my AI agent, I made a patch to make it work;
llama_cache.py:generate()method inllama.py:This keeps the model from reusing checkpoint whose position is equal to new prompt without any potential side effects, and it succeeds the next generation.