docs(guidelines): record the HashMap iteration-order defect class

[inureyes]

Summary

One review sweep between 2026-08-20 and 2026-08-22 turned up seven independent instances of a single defect class across five modules, and no part of the toolchain flagged any of them. The class is a HashMap iteration result becoming ordered state, or feeding a consumer that is sensitive to order. This writes the class down in docs/code-guidelines.md, as a sibling to "JIT Kernel Cache Keys" and in the same shape, and records a measured decision to decline the static check that #1287 left open.

What changed

• docs/code-guidelines.md: new "HashMap Iteration Order" section, additive, with the existing three sections untouched. It states the rule with a wrong/right example drawn from the actual code, then a **Why this matters:** block, three subsections, and an **Enforcement:** block.
• docs/README.md: entry 20 extended so the docs index names the new section, matching how it already enumerates the file's other sections.

The section covers:

• Per-instance, not per-process. RandomState seeds each map instance, so a fixed binary on a fixed machine does not see a fixed order. Re-derived here with rustc -O 1.97.1: ten HashMap<&str, u32> built from the same five keys inside one process, over 200 processes, gave 10 distinct orders out of 10 in 127 processes, 9 in 62, and 8 in 11. Not one process in 200 had the ten maps agree. (The issue's figure was 133/52/15; the split moves between measurement runs because the experiment is itself random, and the invariant that the ten never agree does not.)
• What counts as an order-sensitive consumer. Positional indexing, min_by_key / max_by_key (both return the FIRST extremum, so they tie-break on input order), a prefix via take or an early break or return, a loop-carried accumulator such as a seed advanced once per element, and anything documented as a priority order.
• The stable-sort subtlety, which gets the most space. slice::sort_by_key and slice::sort_by are stable, so a sort does not make a HashMap-derived list deterministic unless the key is a total order over the elements. sort_unstable_by is not a remedy either; it substitutes a different arbitrary tie order. Three of the seven instances (fix(distributed): eviction candidate sorts are stable, so HashMap order still picks which tied sequence is evicted #1286, PR fix(distributed): break eviction sort ties on the unique id #1288) sorted and were still nondeterministic. Three correct total-order remedies are given, taken from the real fixes.
• The regression-testing rule, because every fix in this family needed it: a fresh map per iteration and at least 32 iterations; a deliberately constructed tie, since distinct keys already give a total order and such a test passes without the fix; and equal timestamps written in by hand, since Instant::now() never collides on its own at nanosecond resolution. PR fix(rt-detr-v2): make needs_sanitize independent of HashMap order #1281's pre-fix run failed on 27 of 64 freshly built maps, which is exactly why a single-shot test would have passed and shipped nothing.
• The BTreeMap tradeoff, and why fix(distributed): registry accessors return HashMap order, so node routing and failover re-routing are not reproducible #1277 kept its HashMap and sorted in the accessors instead.
• The seven instances as evidence: test: the klear prefill-causality parity test is nondeterministic run to run on CUDA, and TF32 amplifies it past its tolerance #1265 (PR fix(tests): seed synthetic model fixtures from a sorted key walk #1266), fix(lang-bias): the YAML bias map is a HashMap, so language priority is randomized per process #1267 (PR fix(lang-bias): preserve YAML bias: order so priority is deterministic #1269), fix(rt-detr-v2): needs_sanitize returns on the first marker found in HashMap order, so its verdict is order-dependent #1276 (PR fix(rt-detr-v2): make needs_sanitize independent of HashMap order #1281), fix(distributed): registry accessors return HashMap order, so node routing and failover re-routing are not reproducible #1277 (PR fix(distributed): give registry node accessors a defined order #1284), and the three in fix(distributed): eviction candidate sorts are stable, so HashMap order still picks which tied sequence is evicted #1286 (PR fix(distributed): break eviction sort ties on the unique id #1288).

Part 2: the static check is declined, with the numbers

Two candidates were prototyped in the shape of scripts/ci/check_kernel_dtype_keys.py and run over all 1,248 .rs files under src/, against the current tree and against the reconstructed pre-fix tree (git archive <fix>^ src) of each of the five fixes.

min_by_key / max_by_key on an unordered-map receiver. 4 flags on the current tree, 0 false positives: src/server/prompt_cache/store.rs:315 and :336, src/server/responses_store.rs:243, src/server/conversation_store.rs:151. Correctly excludes src/distributed/pipeline/partition_balance.rs:126 and the other slice receivers. But its catch rate against the seven known instances is 0 of 7: all five pre-fix trees produced those same four flags and nothing else, because none of the seven used min_by_key or max_by_key. Its four hits are also lint true positives and not live defects: every key is a std::time::Instant stamped once per call, so at nanosecond resolution the tie is not reachable. Gating on it would mean suppressing 4 of 4 findings on the day it landed, and a suppression comment does not re-arm on the change that would make the pattern real (the key becoming coarser), which is the only scenario in which it pays.

A Vec built from an unordered-map view and consumed order-sensitively. 22 flags on the current tree. It flags all three #1286 sites after PR #1288 fixed them, because the fix keeps the shape and changes only whether the sort key is total, which is a distinction no regex can draw. Its 3-of-7 catch rate held only against pre-fix trees; on a tree where the class has been fixed those three become permanent false positives, so it can never be a gate. It reaches none of the four separately filed instances, each for a different structural reason.

A tree-wide scan for type X = HashMap<...> does remove the type-alias blind spot the issue names as the biggest recall limit (it resolves WeightMap, Memo and DetachedMap), so that one limit is fixable. The other three are not at this level of analysis, and the general form is undecidable from source alone because the consumer is often in another module. Both numbers and the reasoning are recorded in the Enforcement block so the next person does not re-derive them.

Test plan

• make verify-fmt exits 0 (docs-only change; no Rust or build file is touched, and no script was added, so no other gate applies).
• grep -n '^## ' docs/code-guidelines.md shows a fourth section, HashMap Iteration Order at line 109, with the existing three unmoved.
• grep -n 'code-guidelines' docs/README.md CONTRIBUTING.md .github/PULL_REQUEST_TEMPLATE.md exits 0; the index entry at docs/README.md:36 now names the new section.
• The relative link ../scripts/ci/check_kernel_dtype_keys.py resolves, matching the JIT section's existing link.
• Every issue and PR number, accessor name, test-loop constant, Makefile:604-616 line range and the 27-of-64 figure cited in the section was checked against the tree or the PR body rather than copied from the issue.

Note on the issue body

The issue describes #1286 as open and its three instances as "still unfixed at main". They were fixed by PR #1288, merged after the body's last edit, and main at 9f4f6516 is that merge. The section is written against the fixed state, and that is what makes the second candidate rule's limitation visible.

Closes #1287