Skip to content

Set prompt_cache_key on the Grok and OpenAI Responses adapters - #543

Merged
TheGreatAxios merged 9 commits into
mainfrom
cl-6904-grokopenai-responses-adapters-omit-prompt_cache_key-xai
Aug 23, 2026
Merged

Set prompt_cache_key on the Grok and OpenAI Responses adapters#543
TheGreatAxios merged 9 commits into
mainfrom
cl-6904-grokopenai-responses-adapters-omit-prompt_cache_key-xai

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Collaborator

Problem

codex-responses-adapter.ts sets prompt_cache_key per session (~line 304). grok-responses-adapter.ts and openai-responses-adapter.ts never set it, and no session id reached the grok adapter at all. Grok threads were caching at 66-72% vs codex's 92%+, on roughly 340M grok input tokens per 4 days.

What the docs say

OpenAI (platform.openai.com/docs/guides/prompt-caching, now redirected to developers.openai.com/api/docs/guides/prompt-caching):

  • prompt_cache_key operates at request-routing granularity: requests are routed to a machine "based on prompt_cache_key, with a hash of the initial prefix of the prompt as a secondary key."
  • Reuse the same key across requests that share a long common prefix; a cache key improves routing but does not make different prompt prefixes match.
  • For GPT-5.6, OpenAI says prompt_cache_key must be set for reliable matching.
  • Keep traffic per key/prefix to roughly 15 req/min; partition higher-volume workloads across a stable set of keys.
  • No format/length constraint is documented beyond "session IDs and user IDs" as example values.
  • The guide never mentions store, so there's no documented interaction with store:false — the existing codex-adapter comment ("store:false has no other signal") is our own inference (no response-id chaining is possible with store:false, so prompt_cache_key is the only cache-locality lever), not a documented interaction.

xAI (docs.x.ai/developers/advanced-api-usage/prompt-caching, how-it-works/best-practices pages) — this was the critical check:

  • xAI's Responses API does honor prompt_cache_key — it is not OpenAI-only. Docs state: "Always set x-grok-conv-id (or prompt_cache_key for Responses API)" to route requests to the same server and maximize cache hits.
  • Caching itself is automatic (prefix match from the start of the messages array); prompt_cache_key/x-grok-conv-id only affects routing consistency, not whether caching happens.
  • Best practice: never edit/remove/reorder earlier messages (breaks the cache), front-load static content (system prompt, few-shot, reference docs), and monitor cached_tokens — persistent zeros mean the key or message ordering needs checking.
  • So shipping prompt_cache_key on the grok adapter is not a no-op; it's the documented mechanism.

What changed

  • grok-responses-adapter.ts: new GROK_SESSION_ID_OPTION providerOptions key; buildRequest sets body.prompt_cache_key from it when present.
  • openai-responses-adapter.ts: new OPENAI_SESSION_ID_OPTION providerOptions key; same pattern (used by OpenCode Go models on the Responses protocol, e.g. gpt-5.6-luna).
  • config/index.ts: buildXaiSource now requires sessionId and stashes it as GROK_SESSION_ID_OPTION; buildGoSource takes an optional sessionId and stashes it as OPENAI_SESSION_ID_OPTION when the endpoint speaks Responses.
  • config/inference-sources.ts: buildInferenceSourceForRef threads ctx.sessionId into both buildXaiSource and buildGoSource calls. The parent session gets its own real session id; each subagent thread already gets its own id via buildSubagentSources's existing sessionId ?? randomUUID() fallback (same pattern codex already used), so every thread's key is stable across turns and distinct from every other thread.
  • exec/runner.ts / tui/runner.ts: the initial xAI source build now passes the live sessionId through, matching the existing codex source build call right next to it.

Tests

  • tests/unit/grok-responses-adapter.test.ts: prompt_cache_key set and stable across two buildRequest calls for the same session id; distinct across two different session ids; absent from the body when no session id is present.
  • tests/unit/openai-responses-adapter.test.ts (new): same three cases for the OpenAI Responses adapter.
  • src/config.test.ts: buildXaiSource now requires sessionId in every call site; added a case asserting the session id lands in providerOptions.grokSessionId.

Gates

  • bunx prettier --check / bunx eslint on touched files: both report only pre-existing issues already present on main before this change (whole-file indentation drift, non-null assertions, type-vs-interface, unused vars) — none on the lines this PR adds. The two provider-adapter files and the new test file are fully clean on both. This is the known repo-wide prettier/eslint gap (see ci-split-prettier-eslint-required-checks), not something reformatted here.
  • bun run typecheck: clean.
  • bun run build: clean.
  • bun test ./src ./tests ./evals: 5121 pass, 1 pre-existing unrelated failure (src/agent/lsp-availability.test.ts fails identically on unmodified main in this worktree — missing typescript-language-server binary, not a code regression).

Fixes CL-6904

Codex's Responses adapter already keys prompt_cache_key by session id;
grok-responses-adapter.ts and openai-responses-adapter.ts never set it, so
xAI/OpenCode-Go Responses calls had no cache-routing signal beyond the
implicit prefix hash. Threads the session id (parent session's sessionId,
or a fresh id per subagent thread) through buildXaiSource/buildGoSource into
each adapter's providerOptions, and sets prompt_cache_key in the request body
when present. Per xAI's docs (docs.x.ai/developers/advanced-api-usage/prompt-caching),
the Responses API honors prompt_cache_key the same way OpenAI does; verified
this isn't a no-op before shipping it.
@linear-code

linear-code Bot commented Aug 23, 2026

Copy link
Copy Markdown

CL-6904

@TheGreatAxios
TheGreatAxios merged commit 4107330 into main Aug 23, 2026
5 checks passed
@TheGreatAxios TheGreatAxios mentioned this pull request Aug 23, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant