The /v1/responses store and the conversation store cap only the number of entries, not their size. Each entry retains the complete request input, images included, so a client sending well-formed multimodal store: true requests can pin gigabytes of server memory inside the TTL window.
Evidence
src/server/responses_store.rs:43-46 stores the full response object plus the full request input per entry:
pub struct StoredResponse {
pub response: ResponseObject,
pub input_items: Vec<ResponseInputItem>,
}
src/server/responses_store.rs:56-67 caps the store by count only:
pub struct ResponsesStoreConfig {
pub max_entries: usize,
pub ttl: Duration,
}
impl Default for ResponsesStoreConfig {
fn default() -> Self {
Self {
max_entries: 1024,
ttl: Duration::from_secs(3600),
}
}
}
src/server/conversation_store.rs:53-63 is the same shape (max_entries: 256, 1-hour TTL), and its evict_to_capacity at conversation_store.rs:147-156 also evicts by count only. For a multimodal request the ResponseInputItem values carry the original base64 image data URLs, so 1,024 responses plus 256 conversation transcripts with no byte accounting is an unbounded memory footprint: entry count bounds nothing when a single entry can be tens of megabytes.
The repository already has the right pattern next door: src/server/prompt_cache/policy.rs enforces both a byte budget (capacity_bytes at policy.rs:40) and an entry cap, and prompt_cache_capacity_bytes is an operator knob.
Secondary: evict_to_capacity at src/server/responses_store.rs:237 does an O(n) linear min_by_key scan over the whole map to find the LRU victim on every insert (the comment itself flags it as "acceptable for Phase 1"), and conversation_store.rs:147 duplicates the same scan.
Suggested fix
- Add byte accounting to both stores: track an approximate serialized size per entry at insert time and a running total, and add a
max_bytes field to ResponsesStoreConfig and ConversationStoreConfig, mirroring the capacity_bytes pattern in src/server/prompt_cache/policy.rs.
- Evict oldest-accessed entries until both the entry count and the byte total are under budget.
- Expose the knob on both server binaries and document it in docs/environment-variables.md, following the naming convention of
prompt_cache_capacity_bytes.
- Replace the O(n) victim scan with a proper LRU structure (an access-ordered map or an intrusive list keyed by
last_accessed) so eviction is O(log n) or amortized O(1) per insert.
- Add over-budget eviction tests mirroring the existing
lru_eviction_runs_when_capacity_exceeded test at src/server/responses_store.rs:320, including a test where a small number of large entries exceeds max_bytes while remaining far under max_entries.
Acceptance criteria
The
/v1/responsesstore and the conversation store cap only the number of entries, not their size. Each entry retains the complete request input, images included, so a client sending well-formed multimodalstore: truerequests can pin gigabytes of server memory inside the TTL window.Evidence
src/server/responses_store.rs:43-46 stores the full response object plus the full request input per entry:
src/server/responses_store.rs:56-67 caps the store by count only:
src/server/conversation_store.rs:53-63 is the same shape (
max_entries: 256, 1-hour TTL), and itsevict_to_capacityat conversation_store.rs:147-156 also evicts by count only. For a multimodal request theResponseInputItemvalues carry the original base64 image data URLs, so 1,024 responses plus 256 conversation transcripts with no byte accounting is an unbounded memory footprint: entry count bounds nothing when a single entry can be tens of megabytes.The repository already has the right pattern next door: src/server/prompt_cache/policy.rs enforces both a byte budget (
capacity_bytesat policy.rs:40) and an entry cap, andprompt_cache_capacity_bytesis an operator knob.Secondary:
evict_to_capacityat src/server/responses_store.rs:237 does an O(n) linearmin_by_keyscan over the whole map to find the LRU victim on every insert (the comment itself flags it as "acceptable for Phase 1"), and conversation_store.rs:147 duplicates the same scan.Suggested fix
max_bytesfield toResponsesStoreConfigandConversationStoreConfig, mirroring thecapacity_bytespattern in src/server/prompt_cache/policy.rs.prompt_cache_capacity_bytes.last_accessed) so eviction is O(log n) or amortized O(1) per insert.lru_eviction_runs_when_capacity_exceededtest at src/server/responses_store.rs:320, including a test where a small number of large entries exceedsmax_byteswhile remaining far undermax_entries.Acceptance criteria
lru_eviction_runs_when_capacity_exceededand the TTL tests, pass unchanged