Skip to content

fix(server): bound the responses and conversation stores by bytes, not just entry count #1248

Description

@inureyes

The /v1/responses store and the conversation store cap only the number of entries, not their size. Each entry retains the complete request input, images included, so a client sending well-formed multimodal store: true requests can pin gigabytes of server memory inside the TTL window.

Evidence

src/server/responses_store.rs:43-46 stores the full response object plus the full request input per entry:

pub struct StoredResponse {
    pub response: ResponseObject,
    pub input_items: Vec<ResponseInputItem>,
}

src/server/responses_store.rs:56-67 caps the store by count only:

pub struct ResponsesStoreConfig {
    pub max_entries: usize,
    pub ttl: Duration,
}

impl Default for ResponsesStoreConfig {
    fn default() -> Self {
        Self {
            max_entries: 1024,
            ttl: Duration::from_secs(3600),
        }
    }
}

src/server/conversation_store.rs:53-63 is the same shape (max_entries: 256, 1-hour TTL), and its evict_to_capacity at conversation_store.rs:147-156 also evicts by count only. For a multimodal request the ResponseInputItem values carry the original base64 image data URLs, so 1,024 responses plus 256 conversation transcripts with no byte accounting is an unbounded memory footprint: entry count bounds nothing when a single entry can be tens of megabytes.

The repository already has the right pattern next door: src/server/prompt_cache/policy.rs enforces both a byte budget (capacity_bytes at policy.rs:40) and an entry cap, and prompt_cache_capacity_bytes is an operator knob.

Secondary: evict_to_capacity at src/server/responses_store.rs:237 does an O(n) linear min_by_key scan over the whole map to find the LRU victim on every insert (the comment itself flags it as "acceptable for Phase 1"), and conversation_store.rs:147 duplicates the same scan.

Suggested fix

  • Add byte accounting to both stores: track an approximate serialized size per entry at insert time and a running total, and add a max_bytes field to ResponsesStoreConfig and ConversationStoreConfig, mirroring the capacity_bytes pattern in src/server/prompt_cache/policy.rs.
  • Evict oldest-accessed entries until both the entry count and the byte total are under budget.
  • Expose the knob on both server binaries and document it in docs/environment-variables.md, following the naming convention of prompt_cache_capacity_bytes.
  • Replace the O(n) victim scan with a proper LRU structure (an access-ordered map or an intrusive list keyed by last_accessed) so eviction is O(log n) or amortized O(1) per insert.
  • Add over-budget eviction tests mirroring the existing lru_eviction_runs_when_capacity_exceeded test at src/server/responses_store.rs:320, including a test where a small number of large entries exceeds max_bytes while remaining far under max_entries.

Acceptance criteria

  • Both stores enforce a byte budget in addition to the entry cap, with tests proving eviction triggers on bytes alone
  • The byte-budget knob is exposed on both server binaries and documented in docs/environment-variables.md
  • LRU eviction no longer performs an O(n) scan per insert in either store
  • Existing store tests, including lru_eviction_runs_when_capacity_exceeded and the TTL tests, pass unchanged

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)priority:mediumMedium prioritystatus:readyReady to be worked ontype:bugBug fixes, error corrections, or issue resolutions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions