Upstream: ggml-org#28527
An identical temperature:0, seed:0 request produces different text after its long prefix is reused from the server prompt cache.
Environment: current main 67672dc5b, Linux aarch64, NVIDIA GB10, CUDA 13.0, Meta-Llama-3-8B-Instruct Q4_0, 3,339-token prompt.
llama-server -m model.gguf -ngl 999 -c 4096 --cache-type-k iq4_nl --no-kv-offload
# Send the same request three times with temperature=0, seed=0, max_tokens=16.
Observed: response 1 differed; responses 2 and 3 matched each other. With --no-cache-prompt, all three responses were identical.
This is not specific to quantized K cache: --cache-type-k f16 also changed after the first response. Cached repeats back off one token and evaluate only that token, while the first request evaluates the full prompt; the two paths produce sufficiently different logits to change greedy decoding.
Upstream: ggml-org#28527
An identical
temperature:0, seed:0request produces different text after its long prefix is reused from the server prompt cache.Environment: current main
67672dc5b, Linux aarch64, NVIDIA GB10, CUDA 13.0, Meta-Llama-3-8B-Instruct Q4_0, 3,339-token prompt.llama-server -m model.gguf -ngl 999 -c 4096 --cache-type-k iq4_nl --no-kv-offload # Send the same request three times with temperature=0, seed=0, max_tokens=16.Observed: response 1 differed; responses 2 and 3 matched each other. With
--no-cache-prompt, all three responses were identical.This is not specific to quantized K cache:
--cache-type-k f16also changed after the first response. Cached repeats back off one token and evaluate only that token, while the first request evaluates the full prompt; the two paths produce sufficiently different logits to change greedy decoding.