Skip to content

server: id_slot pinned to an empty slot skips the prompt cache #34

Description

@am17an

Upstream: ggml-org#28139

Pinning a request to an empty slot bypasses an exact prompt-cache match and reprocesses the entire prompt.

Environment: current main 465e49b9, Linux aarch64, Gemma 4 12B Q4_K_M, --parallel 2 --slots --cache-ram 4096.

Reproduction:

  1. Send prompt A with cache_prompt:true,id_slot:0.
  2. Send prompt B with the same options, moving A to the prompt cache.
  3. Send A with cache_prompt:true,id_slot:1.

Observed on step 3: cache_n:0, prompt_n:290.

For an empty pinned slot, f_keep = 0 / 0 becomes NaN, so f_keep < 0.5f is false and the cache is not loaded. The host-memory prompt-cache path dates to d00cbea6.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingserver

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions