refactor(server): give the four LRU eviction min_by_key calls a total order - #1299
Conversation
… order Four LRU eviction sites picked their victim with min_by_key applied directly to a HashMap iterator: Inner::evict_oldest and Inner::evict_oldest_snapshot in src/server/prompt_cache/store.rs, and evict_to_capacity in both src/server/responses_store.rs and src/server/conversation_store.rs. Iterator::min_by_key returns the first minimum, so two entries tied on the timestamp key would be separated only by HashMap iteration order, which RandomState randomizes per map instance. This is not a live bug in any of the four. All four keys are std::time::Instant, stamped once per call by a single writer (touch()/insert()/get()/append()), so at nanosecond resolution the tie is not reachable through the public API today. What this removes is the latent fragility: the pattern would become a real defect the moment a key becomes coarser than a per-call Instant, and nothing about the current code flags that risk. Each site now folds a unique component into the comparison so the key is a total order and HashMap iteration order can no longer reach the outcome. The two prompt_cache sites key on PromptCacheKeyDigest, which is Copy but not Ord, so the tuple form uses *digest.as_bytes() rather than widening the type's derive for one call site. The two String-keyed stores use the comparator form (min_by with then_with) instead, since a tuple key would force a clone of the String. Each site carries a comment naming the tie-break component as what makes the order total, so a later simplification pass does not read it as redundant and strip it. No regression test was added. Constructing the tie would require writing an identical Instant directly into private state, bypassing every real write path (touch/insert/get/append) that produces these timestamps; the resulting test would pin the comparator's generic sort behavior, not any reachable production state. The comment at each site plus docs/code-guidelines.md's HashMap Iteration Order section (which documents this exact defect class and testing methodology) are the guard against the pattern being simplified away; a test only becomes worth adding if a future change makes one of these keys coarser than a per-call Instant, and that change would carry its own tests. Refs #1291
|
No delta against an earlier baseline is quoted, because
|
Bilingual report for the LRU eviction total-order change, recorded before the merge because this repository carries the `TECHNICAL_REPORTS/.keep-reports` marker. Force-added since `TECHNICAL_REPORTS/` is gitignored. Refs #1291
## Summary `BatchScheduler::select_eviction_victim` decides which in-flight sequence gets preempted when the batch is full and a higher-priority request is waiting. Both policy arms selected through a key that was not total, so tied sequences were separated only by `HashMap` iteration order: `LongestFirst` used `max_by_key(|seq| seq.generated_tokens.len())` and `LowestPriority` used `min_by` on priority then length, both over `ActiveBatch::iter_sequences`, which is `HashMap::values`. `RandomState` seeds each map instance, so the victim was not reproducible from the batch state. Unlike the four latent sites fixed in #1291 / PR #1299, this tie is reachable through ordinary state. The `LongestFirst` key is `generated_tokens.len()`, a small integer that sequences admitted together and decoding in lockstep share routinely, and both `PreemptionPolicy::LongestFirst` and `RequestPriority::Normal` are the shipped defaults, so the reachable arm is the one that ships on. The harm is reproducibility rather than correctness: any tied-longest sequence does satisfy the stated policy, nothing is corrupted and nothing crashes, but two identically configured workers under identical load preempt different requests, and an operator investigating why a given request was preempted (and replayed with duplicate streamed tokens, per `try_evict_for_preemption`) cannot reproduce the choice. ## What changed - **`src/server/batch/scheduler.rs`**: the policy is extracted into `select_eviction_victim_from(sequences, policy)`, a `pub(crate)` free function, and `BatchScheduler::select_eviction_victim` becomes a one-line wrapper over it. The `structured.is_none()` filter moves inside the function so no caller can lose it. The free-function form was chosen over a method on `ActiveBatch` for the reason the issue gives: `ActiveBatch` currently knows nothing about `PreemptionPolicy`, and a method would make `active.rs` depend on `crate::server::config`. - **Both arms now append `seq_id`** as the component that makes the order total, using `as_u64()` because `SequenceId` derives only `Debug, Clone, Copy, PartialEq, Eq, Hash` and is not `Ord`. This is the same trap `PromptCacheKeyDigest` posed in #1291, and it is resolved the same way rather than by widening the type's public trait surface for two call sites. `LongestFirst` keys on `(generated_tokens.len(), Reverse(seq_id.as_u64()))`; `LowestPriority` appends `.then_with(|| a.seq_id.as_u64().cmp(&b.seq_id.as_u64()))`. The two directions differ because `max_by_key` returns the LAST maximum while `min_by` returns the FIRST minimum, so opposite-facing tie components are needed to break the same direction. Each arm carries a comment naming `seq_id` as the total-order component so a later simplification pass does not strip it. - **`src/server/batch/scheduler_tests.rs`**: `eviction_selects_longest_first_by_default` and `eviction_selects_lowest_priority_then_longest` now call `select_eviction_victim_from` instead of reimplementing the selection expression. Neither expected victim changed, because neither test constructs a tie. Two determinism tests and an empty-batch test are added. - **`docs/code-guidelines.md`**: this instance is added to the list in the "HashMap Iteration Order" section, which becomes eight. The `max_by_key` correction at the "order-sensitive consumers" bullet and the enforcement note were already in place from PR #1294 and are not redone; the only enforcement-section edits are to keep the measured catch rates arithmetically consistent with an eight-item list, scoped to what #1287 actually measured. ## Tie-break direction: smallest `seq_id`, anti-starvation argument accepted The issue asked for an explicit decision and it is **smallest `seq_id` wins**, with the anti-starvation argument accepted as stated. `SequenceId` is a `u64` newtype handed out by `CachePool` from an `AtomicU64` advanced with `fetch_add(1)` per allocation, so smallest-id-wins reads as "evict the sequence admitted earliest", which needs no new field to be intelligible. The decisive part is what preemption does to the id. `try_evict_for_preemption` reallocates its victim under a fresh, therefore higher, id, so a large id marks a sequence that was preempted *recently*, not one that arrived late. Smallest-id-wins therefore rotates preemption onto sequences that have not been hit yet, whereas largest-id-wins would make the request that was just replayed the preferred next victim. `SequenceInfo::created_at` was considered and rejected. It survives preemption (it is absent from the reset block) and so gives true arrival order, but for exactly that reason it points the wrong way for starvation: smallest-`created_at`-wins would keep selecting the same oldest request, because nothing about being preempted moves the field. It is also an `Instant` and not total on its own, so `seq_id` would have had to sit behind it anyway. Both arms resolve a tie to the same direction, so the choice does not vary by policy. ## Regression test, and its failure against the unfixed code Two tests, both built per `docs/code-guidelines.md` "Testing the fix": a fresh `ActiveBatch` inside each iteration (`RandomState` seeds per map instance, so probing one batch repeatedly measures nothing), 64 iterations, and an actual tie constructed on every axis the arm compares. Ids are inserted out of ascending order so a selector that returns the first- or last-inserted sequence cannot pass by coincidence. Each test records a mismatch per iteration rather than panicking on the first, so a failure reports the count out of 64. The tests were written against the extracted-but-unfixed function and run before the tie-break was added: ``` running 5 tests test server::batch::scheduler::tests::eviction_returns_none_for_an_empty_batch ... ok test server::batch::scheduler::tests::eviction_selects_lowest_priority_then_longest ... ok test server::batch::scheduler::tests::eviction_selects_longest_first_by_default ... ok test server::batch::scheduler::tests::eviction_longest_first_tie_resolves_to_smallest_seq_id ... FAILED test server::batch::scheduler::tests::eviction_lowest_priority_tie_resolves_to_smallest_seq_id ... FAILED ---- eviction_longest_first_tie_resolves_to_smallest_seq_id stdout ---- LongestFirst resolved a fully tied batch to something other than seq_id 2 on 58 of 64 freshly built batches (first Some((0, Some(5)))); the tie is falling through to HashMap order ---- eviction_lowest_priority_tie_resolves_to_smallest_seq_id stdout ---- LowestPriority resolved a batch tied on priority and length to something other than seq_id 2 on 39 of 64 freshly built batches (first Some((1, Some(14)))); the tie is falling through to HashMap order test result: FAILED. 3 passed; 2 failed; 0 ignored; 0 measured; 5895 filtered out ``` So **58 of 64** for `LongestFirst` (first mismatch at iteration 0) and **39 of 64** for `LowestPriority` (first at iteration 1). Both are well under 100 percent, which is the reason the loop exists: a single-shot version of the `LowestPriority` test would have passed about 39 percent of the time. After the fix, all 64 iterations of both tests agree. No `git stash` was used to produce this. The tests were run against the extraction commit before the tie-break was applied, and `git stash list` is empty. ## Notes on the issue Everything the issue asserted checked out against the code, with two points worth recording: - **Line numbers drifted.** The issue was verified at `a0e402ad`; on `c9d53891` (after PR #1299 and PR #1300 landed) `select_eviction_victim` is at `scheduler.rs:6826` rather than `:6803`, and the guideline list is at `docs/code-guidelines.md:184` rather than the quoted span. The code at those sites is otherwise identical to what the issue quotes. - **One acceptance criterion was already satisfied on `main`.** "`docs/code-guidelines.md:137` no longer claims `max_by_key` returns the first extremum" was fixed by PR #1294; the line already states the two directions separately, so no change was needed there. The issue itself says as much in its "Proposed Solution" section, but the criterion was still listed unchecked. The `structured.is_none()` filter is now in the tested code path, but no test constructs a `structured: Some(...)` sequence: `StructuredOutputConstraint` has no public test constructor, and building one through `build_json_schema_constraint` needs a real HuggingFace tokenizer with a vocab plus an llguidance grammar compilation, which does not belong in a scheduler policy unit test. Covering the exclusion directly would need a test-only constructor on that type, which is out of scope here. ## Test plan - [x] `cargo fmt --all -- --check` (exit 0) - [x] `cargo check --lib --tests --features cuda` (exit 0) - [x] `cargo clippy --lib --tests --features cuda -- -D warnings` (exit 0) - [x] `cargo test --profile test-fast --features cuda --lib server::batch::scheduler::tests::eviction` (exit 0, 5 passed, 0 failed) - [x] `cargo test --profile test-fast --features cuda --lib server::batch::scheduler` (exit 0, 111 passed, 0 failed, 7 ignored) - [x] `cargo test --profile test-fast --features cuda --lib server::batch` (exit 0, 373 passed, 0 failed, 7 ignored) - [ ] `make verify-test-cuda` is run separately against this branch. Closes #1293
Summary
Four LRU eviction sites picked their victim with
min_by_keyapplied directly to aHashMapiterator:Inner::evict_oldestandInner::evict_oldest_snapshotinsrc/server/prompt_cache/store.rs:315and:336, andevict_to_capacityin bothsrc/server/responses_store.rs:243andsrc/server/conversation_store.rs:151.Iterator::min_by_keyreturns the FIRST minimum, so two entries tied on the timestamp key would be separated only byHashMapiteration order, whichRandomStaterandomizes per map instance.This is not a live bug, and it is not described as four bugs here. All four keys are
std::time::Instant, and every writer (touch(),insert(),get(),append()) stamps a single entry with oneInstant::now()call. At nanosecond resolution the tie is not reachable through the public API today. What this PR removes is a latent fragility: the pattern would become a real defect the moment one of these keys becomes coarser than a per-callInstant(a seconds-granularity timestamp, a logical clock, a value hoisted out of a loop), and nothing in the current code flags that risk.What changed
src/server/prompt_cache/store.rs,evict_oldest(line ~315) andevict_oldest_snapshot(line ~336): both now key on(slot.entry.last_used(), *digest.as_bytes()).PromptCacheKeyDigestderives onlyClone, Copy, PartialEq, Eq, Hashand is notOrd, so a bare tuple key would not compile. Per the issue's recommendation, I keyed on*digest.as_bytes()(&[u8; 32], which isOrd) rather than addingPartialOrd, Ordto the digest's derive, to avoid widening that type's public trait surface for one call site.last_used()is still called exactly once per element (the key-function closure runs once per item), so this does not turn the per-entry mutex lock into a double lock.src/server/responses_store.rs,evict_to_capacity(line ~243): switched tomin_bywitha.last_accessed.cmp(&b.last_accessed).then_with(|| key_a.cmp(key_b)). The key is aString, so the comparator form avoids the clone a tuple key would force.src/server/conversation_store.rs,evict_to_capacity(line ~151): same comparator form.docs/code-guidelines.md's "HashMap Iteration Order" section, so a later simplification pass does not read the component as redundant and strip it.Scope matches the issue exactly: the four call sites plus a comment at each. No change to eviction policy, cap-enforcement loops, or what the stores are bounded by.
#1248(byte-bounding of the two stores) is open but stillstatus:readywith no PR, i.e. not in flight, so this PR keeps all four sites rather than folding two into that work.Regression test: decision and reasoning
No regression test was added. Reasoning:
Instantdirectly into private state, bypassingtouch()/insert()/get()/append()entirely.docs/code-guidelines.md's "HashMap Iteration Order" section, including its explicit accounting that these exact four sites are "true positives for the lint and zero live bugs."Instant(the actual risk this PR is guarding against) — and that change would introduce its own new code path, which is the natural place for its own review and its own test, not a synthetic test bolted onto code that cannot exercise the coarser key today.This is a judgment call per the issue's acceptance criteria, which explicitly allow either outcome with reasoning stated. No existing test's expected outcome changed.
Test plan
cargo fmt --all -- --check— exit 0cargo check --lib --tests --features cuda— exit 0cargo clippy --lib --tests --features cuda -- -D warnings— exit 0cargo test --profile test-fast --features cuda --lib server::prompt_cache— exit 0, 168 passed, 0 failedcargo test --profile test-fast --features cuda --lib server::responses_store --no-runthen run the built binary filtered toserver::responses_store— exit 0, 8 passed, 0 failedserver::conversation_store— exit 0, 5 passed, 0 failedmake verify-test-cudawill be run separately against this branch.Closes #1291