Skip to content

refactor(server): give the four LRU eviction min_by_key calls a total order - #1299

Merged
inureyes merged 2 commits into
mainfrom
refactor/issue-1291-lru-eviction-total-order
Aug 22, 2026
Merged

refactor(server): give the four LRU eviction min_by_key calls a total order#1299
inureyes merged 2 commits into
mainfrom
refactor/issue-1291-lru-eviction-total-order

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Four LRU eviction sites picked their victim with min_by_key applied directly to a HashMap iterator: Inner::evict_oldest and Inner::evict_oldest_snapshot in src/server/prompt_cache/store.rs:315 and :336, and evict_to_capacity in both src/server/responses_store.rs:243 and src/server/conversation_store.rs:151. Iterator::min_by_key returns the FIRST minimum, so two entries tied on the timestamp key would be separated only by HashMap iteration order, which RandomState randomizes per map instance.

This is not a live bug, and it is not described as four bugs here. All four keys are std::time::Instant, and every writer (touch(), insert(), get(), append()) stamps a single entry with one Instant::now() call. At nanosecond resolution the tie is not reachable through the public API today. What this PR removes is a latent fragility: the pattern would become a real defect the moment one of these keys becomes coarser than a per-call Instant (a seconds-granularity timestamp, a logical clock, a value hoisted out of a loop), and nothing in the current code flags that risk.

What changed

  • src/server/prompt_cache/store.rs, evict_oldest (line ~315) and evict_oldest_snapshot (line ~336): both now key on (slot.entry.last_used(), *digest.as_bytes()). PromptCacheKeyDigest derives only Clone, Copy, PartialEq, Eq, Hash and is not Ord, so a bare tuple key would not compile. Per the issue's recommendation, I keyed on *digest.as_bytes() (&[u8; 32], which is Ord) rather than adding PartialOrd, Ord to the digest's derive, to avoid widening that type's public trait surface for one call site. last_used() is still called exactly once per element (the key-function closure runs once per item), so this does not turn the per-entry mutex lock into a double lock.
  • src/server/responses_store.rs, evict_to_capacity (line ~243): switched to min_by with a.last_accessed.cmp(&b.last_accessed).then_with(|| key_a.cmp(key_b)). The key is a String, so the comparator form avoids the clone a tuple key would force.
  • src/server/conversation_store.rs, evict_to_capacity (line ~151): same comparator form.
  • Each site carries a comment naming the tie-break component (digest bytes or map key) as what makes the order total, per docs/code-guidelines.md's "HashMap Iteration Order" section, so a later simplification pass does not read the component as redundant and strip it.

Scope matches the issue exactly: the four call sites plus a comment at each. No change to eviction policy, cap-enforcement loops, or what the stores are bounded by. #1248 (byte-bounding of the two stores) is open but still status:ready with no PR, i.e. not in flight, so this PR keeps all four sites rather than folding two into that work.

Regression test: decision and reasoning

No regression test was added. Reasoning:

  • The tie is not reachable through any production write path on any of the four sites (confirmed per site above), so a test would have to write an identical Instant directly into private state, bypassing touch()/insert()/get()/append() entirely.
  • Such a test would pin the comparator/tuple-key's generic sort behavior, not any reachable production state. It would not catch a real regression that a code reviewer plus the comment at each site wouldn't also catch.
  • The general defect class, its testing methodology (fresh map per iteration, 32+ iterations, an actual constructed tie), and precedent test suites are already recorded in docs/code-guidelines.md's "HashMap Iteration Order" section, including its explicit accounting that these exact four sites are "true positives for the lint and zero live bugs."
  • A test becomes worth adding the moment a future change makes one of these keys coarser than a per-call Instant (the actual risk this PR is guarding against) — and that change would introduce its own new code path, which is the natural place for its own review and its own test, not a synthetic test bolted onto code that cannot exercise the coarser key today.

This is a judgment call per the issue's acceptance criteria, which explicitly allow either outcome with reasoning stated. No existing test's expected outcome changed.

Test plan

  • cargo fmt --all -- --check — exit 0
  • cargo check --lib --tests --features cuda — exit 0
  • cargo clippy --lib --tests --features cuda -- -D warnings — exit 0
  • cargo test --profile test-fast --features cuda --lib server::prompt_cache — exit 0, 168 passed, 0 failed
  • cargo test --profile test-fast --features cuda --lib server::responses_store --no-run then run the built binary filtered to server::responses_store — exit 0, 8 passed, 0 failed
  • Same binary filtered to server::conversation_store — exit 0, 5 passed, 0 failed

make verify-test-cuda will be run separately against this branch.

Closes #1291

… order

Four LRU eviction sites picked their victim with min_by_key applied directly to a HashMap iterator: Inner::evict_oldest and Inner::evict_oldest_snapshot in src/server/prompt_cache/store.rs, and evict_to_capacity in both src/server/responses_store.rs and src/server/conversation_store.rs. Iterator::min_by_key returns the first minimum, so two entries tied on the timestamp key would be separated only by HashMap iteration order, which RandomState randomizes per map instance.

This is not a live bug in any of the four. All four keys are std::time::Instant, stamped once per call by a single writer (touch()/insert()/get()/append()), so at nanosecond resolution the tie is not reachable through the public API today. What this removes is the latent fragility: the pattern would become a real defect the moment a key becomes coarser than a per-call Instant, and nothing about the current code flags that risk.

Each site now folds a unique component into the comparison so the key is a total order and HashMap iteration order can no longer reach the outcome. The two prompt_cache sites key on PromptCacheKeyDigest, which is Copy but not Ord, so the tuple form uses *digest.as_bytes() rather than widening the type's derive for one call site. The two String-keyed stores use the comparator form (min_by with then_with) instead, since a tuple key would force a clone of the String. Each site carries a comment naming the tie-break component as what makes the order total, so a later simplification pass does not read it as redundant and strip it.

No regression test was added. Constructing the tie would require writing an identical Instant directly into private state, bypassing every real write path (touch/insert/get/append) that produces these timestamps; the resulting test would pin the comparator's generic sort behavior, not any reachable production state. The comment at each site plus docs/code-guidelines.md's HashMap Iteration Order section (which documents this exact defect class and testing methodology) are the guard against the pattern being simplified away; a test only becomes worth adding if a future change makes one of these keys coarser than a per-call Instant, and that change would carry its own tests.

Refs #1291
@inureyes inureyes added type:refactor Code restructuring without changing functionality priority:low Low priority area:architecture Architecture and code structure changes area:inference Generation, sampling, decoding (incl. speculative, DRY) status:review Under review labels Aug 22, 2026
@inureyes

Copy link
Copy Markdown
Member Author

make verify-test-cuda on GB10 (CUDA sm_121, Linux aarch64) at bd4bbadf, run with no other cargo process on the box: 8251 passed, 0 failed, 311 ignored, 101 suites, exit 0. The log was scanned for process-level aborts as well as failing tests, since a per-suite tally cannot see a teardown crash: no SIGABRT, no terminate called, no error: test failed.

No delta against an earlier baseline is quoted, because main moved during this work: #1292, #1295 and #1296 landed after the last full gate run here, and the latter two add tests. What is checkable is this branch's own contribution, which is zero: the diff adds no #[test] functions and removes none, so the total matching main's current count is the expected result rather than a coincidence.

cargo fmt --all -- --check, cargo check --lib --tests --features cuda and cargo clippy --lib --tests --features cuda -- -D warnings are all clean on the same tree.

Bilingual report for the LRU eviction total-order change, recorded before the
merge because this repository carries the `TECHNICAL_REPORTS/.keep-reports`
marker. Force-added since `TECHNICAL_REPORTS/` is gitignored.

Refs #1291
@inureyes inureyes added the status:done Completed label Aug 22, 2026
@inureyes
inureyes merged commit c9d5389 into main Aug 22, 2026
10 checks passed
@inureyes
inureyes deleted the refactor/issue-1291-lru-eviction-total-order branch August 22, 2026 07:25
inureyes added a commit that referenced this pull request Aug 22, 2026
## Summary

`BatchScheduler::select_eviction_victim` decides which in-flight sequence gets preempted when the batch is full and a higher-priority request is waiting. Both policy arms selected through a key that was not total, so tied sequences were separated only by `HashMap` iteration order: `LongestFirst` used `max_by_key(|seq| seq.generated_tokens.len())` and `LowestPriority` used `min_by` on priority then length, both over `ActiveBatch::iter_sequences`, which is `HashMap::values`. `RandomState` seeds each map instance, so the victim was not reproducible from the batch state.

Unlike the four latent sites fixed in #1291 / PR #1299, this tie is reachable through ordinary state. The `LongestFirst` key is `generated_tokens.len()`, a small integer that sequences admitted together and decoding in lockstep share routinely, and both `PreemptionPolicy::LongestFirst` and `RequestPriority::Normal` are the shipped defaults, so the reachable arm is the one that ships on. The harm is reproducibility rather than correctness: any tied-longest sequence does satisfy the stated policy, nothing is corrupted and nothing crashes, but two identically configured workers under identical load preempt different requests, and an operator investigating why a given request was preempted (and replayed with duplicate streamed tokens, per `try_evict_for_preemption`) cannot reproduce the choice.

## What changed

- **`src/server/batch/scheduler.rs`**: the policy is extracted into `select_eviction_victim_from(sequences, policy)`, a `pub(crate)` free function, and `BatchScheduler::select_eviction_victim` becomes a one-line wrapper over it. The `structured.is_none()` filter moves inside the function so no caller can lose it. The free-function form was chosen over a method on `ActiveBatch` for the reason the issue gives: `ActiveBatch` currently knows nothing about `PreemptionPolicy`, and a method would make `active.rs` depend on `crate::server::config`.
- **Both arms now append `seq_id`** as the component that makes the order total, using `as_u64()` because `SequenceId` derives only `Debug, Clone, Copy, PartialEq, Eq, Hash` and is not `Ord`. This is the same trap `PromptCacheKeyDigest` posed in #1291, and it is resolved the same way rather than by widening the type's public trait surface for two call sites. `LongestFirst` keys on `(generated_tokens.len(), Reverse(seq_id.as_u64()))`; `LowestPriority` appends `.then_with(|| a.seq_id.as_u64().cmp(&b.seq_id.as_u64()))`. The two directions differ because `max_by_key` returns the LAST maximum while `min_by` returns the FIRST minimum, so opposite-facing tie components are needed to break the same direction. Each arm carries a comment naming `seq_id` as the total-order component so a later simplification pass does not strip it.
- **`src/server/batch/scheduler_tests.rs`**: `eviction_selects_longest_first_by_default` and `eviction_selects_lowest_priority_then_longest` now call `select_eviction_victim_from` instead of reimplementing the selection expression. Neither expected victim changed, because neither test constructs a tie. Two determinism tests and an empty-batch test are added.
- **`docs/code-guidelines.md`**: this instance is added to the list in the "HashMap Iteration Order" section, which becomes eight. The `max_by_key` correction at the "order-sensitive consumers" bullet and the enforcement note were already in place from PR #1294 and are not redone; the only enforcement-section edits are to keep the measured catch rates arithmetically consistent with an eight-item list, scoped to what #1287 actually measured.

## Tie-break direction: smallest `seq_id`, anti-starvation argument accepted

The issue asked for an explicit decision and it is **smallest `seq_id` wins**, with the anti-starvation argument accepted as stated.

`SequenceId` is a `u64` newtype handed out by `CachePool` from an `AtomicU64` advanced with `fetch_add(1)` per allocation, so smallest-id-wins reads as "evict the sequence admitted earliest", which needs no new field to be intelligible.

The decisive part is what preemption does to the id. `try_evict_for_preemption` reallocates its victim under a fresh, therefore higher, id, so a large id marks a sequence that was preempted *recently*, not one that arrived late. Smallest-id-wins therefore rotates preemption onto sequences that have not been hit yet, whereas largest-id-wins would make the request that was just replayed the preferred next victim.

`SequenceInfo::created_at` was considered and rejected. It survives preemption (it is absent from the reset block) and so gives true arrival order, but for exactly that reason it points the wrong way for starvation: smallest-`created_at`-wins would keep selecting the same oldest request, because nothing about being preempted moves the field. It is also an `Instant` and not total on its own, so `seq_id` would have had to sit behind it anyway.

Both arms resolve a tie to the same direction, so the choice does not vary by policy.

## Regression test, and its failure against the unfixed code

Two tests, both built per `docs/code-guidelines.md` "Testing the fix": a fresh `ActiveBatch` inside each iteration (`RandomState` seeds per map instance, so probing one batch repeatedly measures nothing), 64 iterations, and an actual tie constructed on every axis the arm compares. Ids are inserted out of ascending order so a selector that returns the first- or last-inserted sequence cannot pass by coincidence. Each test records a mismatch per iteration rather than panicking on the first, so a failure reports the count out of 64.

The tests were written against the extracted-but-unfixed function and run before the tie-break was added:

```
running 5 tests
test server::batch::scheduler::tests::eviction_returns_none_for_an_empty_batch ... ok
test server::batch::scheduler::tests::eviction_selects_lowest_priority_then_longest ... ok
test server::batch::scheduler::tests::eviction_selects_longest_first_by_default ... ok
test server::batch::scheduler::tests::eviction_longest_first_tie_resolves_to_smallest_seq_id ... FAILED
test server::batch::scheduler::tests::eviction_lowest_priority_tie_resolves_to_smallest_seq_id ... FAILED

---- eviction_longest_first_tie_resolves_to_smallest_seq_id stdout ----
LongestFirst resolved a fully tied batch to something other than seq_id 2 on 58 of 64 freshly built batches (first Some((0, Some(5)))); the tie is falling through to HashMap order

---- eviction_lowest_priority_tie_resolves_to_smallest_seq_id stdout ----
LowestPriority resolved a batch tied on priority and length to something other than seq_id 2 on 39 of 64 freshly built batches (first Some((1, Some(14)))); the tie is falling through to HashMap order

test result: FAILED. 3 passed; 2 failed; 0 ignored; 0 measured; 5895 filtered out
```

So **58 of 64** for `LongestFirst` (first mismatch at iteration 0) and **39 of 64** for `LowestPriority` (first at iteration 1). Both are well under 100 percent, which is the reason the loop exists: a single-shot version of the `LowestPriority` test would have passed about 39 percent of the time. After the fix, all 64 iterations of both tests agree.

No `git stash` was used to produce this. The tests were run against the extraction commit before the tie-break was applied, and `git stash list` is empty.

## Notes on the issue

Everything the issue asserted checked out against the code, with two points worth recording:

- **Line numbers drifted.** The issue was verified at `a0e402ad`; on `c9d53891` (after PR #1299 and PR #1300 landed) `select_eviction_victim` is at `scheduler.rs:6826` rather than `:6803`, and the guideline list is at `docs/code-guidelines.md:184` rather than the quoted span. The code at those sites is otherwise identical to what the issue quotes.
- **One acceptance criterion was already satisfied on `main`.** "`docs/code-guidelines.md:137` no longer claims `max_by_key` returns the first extremum" was fixed by PR #1294; the line already states the two directions separately, so no change was needed there. The issue itself says as much in its "Proposed Solution" section, but the criterion was still listed unchecked.

The `structured.is_none()` filter is now in the tested code path, but no test constructs a `structured: Some(...)` sequence: `StructuredOutputConstraint` has no public test constructor, and building one through `build_json_schema_constraint` needs a real HuggingFace tokenizer with a vocab plus an llguidance grammar compilation, which does not belong in a scheduler policy unit test. Covering the exclusion directly would need a test-only constructor on that type, which is out of scope here.

## Test plan

- [x] `cargo fmt --all -- --check` (exit 0)
- [x] `cargo check --lib --tests --features cuda` (exit 0)
- [x] `cargo clippy --lib --tests --features cuda -- -D warnings` (exit 0)
- [x] `cargo test --profile test-fast --features cuda --lib server::batch::scheduler::tests::eviction` (exit 0, 5 passed, 0 failed)
- [x] `cargo test --profile test-fast --features cuda --lib server::batch::scheduler` (exit 0, 111 passed, 0 failed, 7 ignored)
- [x] `cargo test --profile test-fast --features cuda --lib server::batch` (exit 0, 373 passed, 0 failed, 7 ignored)
- [ ] `make verify-test-cuda` is run separately against this branch.

Closes #1293
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes area:inference Generation, sampling, decoding (incl. speculative, DRY) priority:low Low priority status:done Completed status:review Under review type:refactor Code restructuring without changing functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(server): give the four LRU eviction min_by_key calls a total order

1 participant