llama : add bounded routed expert store

[ajaxdude]

Overview

Add the bounded routed-expert SSD store/cache core for the DeepSeek V4.1 stack tracked by halo-box#48.

• add an explicit external tensor registration path that validates and consumes routed tensors without creating GGML/backend tensors
• exclude external extents from mmap prefetch and unmap their interior pages, while preserving mmap-backed loading for contexts whose ranges do not cross an external extent
• read native IQ2_XXS gate/up and Q2_K down expert planes with checked 64-bit offsets, aligned positional I/O, and strict short-read handling
• require direct I/O by default; buffered I/O is an explicit portability/test opt-in because page-cache bytes are outside cache_bytes
• report direct I/O unavailable on Windows until FILE_FLAG_NO_BUFFERING is implemented, so the production default fails closed instead of claiming bounded cached CRT reads
• enforce fixed cache byte and slot budgets with deterministic LRU eviction
• deduplicate multi-token requests, reserve capacity before mutation, remap original expert IDs to resident slot IDs, and return native payload pointers
• keep entries pinned through move-only leases so later asynchronous backend uploads cannot race eviction

This is stacked on halo-box#49 at current head 65fdc19831d30b12f06b981c3ca74dd18731f90e. The repaired base, normal-RoPE classification, and KV A norm correction were merged normally into this branch. This PR must be retargeted to master after ggml-org#49 merges.

Measurements

No throughput claim is made. This PR does not alter build_moe_ffn(), execute a model graph, upload experts to ROCm, or load the published 365 GB payload.

Published-GGUF header accounting is covered as a model-free test vector:

• gate plane: 3,041,280 bytes
• up plane: 3,041,280 bytes
• down plane: 3,870,720 bytes
• one expert across one layer: 9,953,280 bytes
• one resident expert slot across 40 layers: 398,131,200 bytes
• all 384 routed experts: 152,882,380,800 bytes
• 224 slots: 89,181,388,800 bytes
• 256 slots: 101,921,587,200 bytes
• non-routed/non-Engram extents: 10,067,427,328 bytes

The later runtime admission layer must derive these values from metadata. It must account for graph/state, process baseline, I/O buffers, and safety margin rather than selecting 256 slots from expert bytes alone.

Correctness:

• test-expert-store
• test-deepseek41-schema
• test-llama-archs
• test-gguf

All passed in a CPU-only local build.

After merging repaired schema head ea7793dba3497a95ab2516e7bf36f15846b5bd4b, the same targeted tests passed again. The full test-llama-archs sweep passed in a CPU-only build; the focused deepseek41 invocation also passed with the expected schema-only skips.

After merging the follow-up normal-RoPE classification at 6a473b7e942a2cc54f1e5bb2b2d5deacd247b888, test-expert-store, test-deepseek41-schema, test-gguf, the full CPU-only test-llama-archs sweep, and diff checks passed again.

At candidate head f8d537e3642eb03539c5e840a929e86a76dde134, the same packet passed after the KV A norm correction and boundedness fixes. This provenance-only descendant has the same source tree as fail-closed corrective commit 8805ae2d415e2ecc155a105659a9d49072550807. test-expert-store also passed with AddressSanitizer and UndefinedBehaviorSanitizer enabled. Independent review found five issues: unsafe contexts now load without mmap and discard copied source pages where supported, safe retained mmap spans preserve mlock without crossing punched holes, lease publication performs all throwing allocations before pin ownership is visible, files with external holes use random file advice with no positive prefetch ranges, and a failure to apply the strict random policy now fails before mmap instead of continuing with kernel-default readahead.

test-expert-store covers exact slice offsets and native bytes, published IQ2_XXS/Q2_K plane sizing, direct-I/O alignment, actual reads above 32-bit offsets, truncated files, invalid IDs/types/strides/bounds, byte and slot limits, deterministic eviction, repeated IDs, multi-token unions, slot remapping, pin/unpin behavior, payload validation, and the absence of full external-tensor allocation. It also includes a Linux regression for a final expert plane ending at an unaligned file tail: the aligned read stops as soon as the complete payload is present and must remain on the direct descriptor.

The focused Linux test also passed on the Strix host at exact head a8364460999b94ac2ef52cf28bd1d0d4c925eef2:

cmake --build /home/papa/ai/worktrees/expert-store-1572ea7fa/build-expert-store --target test-expert-store -j2
ctest --test-dir /home/papa/ai/worktrees/expert-store-1572ea7fa/build-expert-store --output-on-failure -R '^test-expert-store$'

Test #44: test-expert-store ................   Passed
100% tests passed, 0 tests failed out of 1

This ran on Fedora Linux 44, kernel 7.2.4-200.fc44.x86_64, without opening any model payload.

Additional information

The positional I/O and cache reservation design is adapted from ggml-org#25294, primarily commit 4260e460832f5e95c41d8b6d8b2aea98007df524. Lease lifetime and delayed publication safety are informed by ggml-org#27861 commit bccbacdb8945680f1cfc7e6bffd1e59014705750.

Fallback policy: production callers leave direct_io=true and allow_buffered_io=false, so an unavailable or failing direct descriptor returns an explicit error and cannot grow the page cache outside the configured expert-cache budget. Model-free and non-Linux tests may explicitly set allow_buffered_io=true; that mode logs the fallback and does not claim a host-wide memory bound.

Files containing external tensor holes are not prefetched. A backend context whose normal tensor span crosses a hole loads its resident tensors through bounded staging rather than copying from the mapping, and drops the copied source ranges from the filesystem cache where the platform supports POSIX_FADV_DONTNEED. Safe mmap-backed spans remain eligible for mlock; lazy mappings remain unlocked so they are not faulted into RAM.

On Linux, external-hole mappings use POSIX_FADV_RANDOM instead of file-wide POSIX_FADV_SEQUENTIAL. The strict external policy also produces no POSIX_MADV_WILLNEED or platform-equivalent positive prefetch ranges. The regression covers first, middle, adjacent, boundary-misaligned, and last excluded ranges plus repeated policy evaluation. Existing lazy-table prefetch behavior remains unchanged.

The Linux failure-path regression injects a posix_fadvise error into the real mapping constructor. Strict external mode throws before the constructor can continue to mmap, while the legacy sequential path remains warning-only and successfully maps the same model-free temporary file.

Independent exact-head review approved f8d537e3642eb03539c5e840a929e86a76dde134 with no blocking correctness, resource-bound, direct-I/O, mmap-residency, cache, pin/lease-lifetime, or cross-platform findings.

Windows currently uses cached CRT I/O in llama_file. has_direct_io() therefore reports false there, strict production construction fails explicitly, and the Windows regression verifies both that failure and the explicit non-bounded buffered opt-in. This PR does not add a FILE_FLAG_NO_BUFFERING implementation.

This PR intentionally does not port the upstream graph remap or waved-prefill implementation. Upstream testing retracted bit-exact parity for long multi-wave generations because changed accumulation order can diverge. A later integration must:

• have the DeepSeek V4.1 model register all routed gate/up/down tensors and own the resulting store
• derive the byte and slot budget from metadata plus host-wide admission
• hold each lease until the ROCm upload/event completes, then publish the slot remap
• reject or adapt a ubatch before graph execution when its selected union cannot fit a single resident pass
• prove exact output equivalence before introducing any wave partitioning

Engram disk rows remain owned by the separate Engram PR and are not changed here.

Requirements

• I have read and agree with the contributing guidelines
• This change is Strix Halo specific, or justified by measurements on Strix Halo. General llama.cpp improvements belong in halo-box/llama.cpp instead
• AI usage disclosure: AGENT-AUTHORED by GPT-5.6 Sol. The agent inspected the local loader and mmap paths, adapted the cited upstream designs, implemented the bounded core and model-free tests, and prepared this PR.
• What was NOT verified: no published-GGUF payload was opened, mapped, warmed, or executed; no ROCm upload, graph integration, inference, benchmark, memory watchdog, Engram path, or full-model load was run. The Linux O_DIRECT file-tail regression passed on the Strix host. The Windows fail-closed regression passed in hosted CI before the final boundedness-only follow-up.

[ajaxdude]

Strix Linux focused validation completed at exact head a8364460999b94ac2ef52cf28bd1d0d4c925eef2 without accessing any model payload.

Host: Fedora Linux 44, kernel 7.2.4-200.fc44.x86_64 (fedora2). The isolated worktree was on the home NVMe; no /mnt/bigspace path or ROCm package was touched.

cmake --build /home/papa/ai/worktrees/expert-store-1572ea7fa/build-expert-store --target test-expert-store -j2
ctest --test-dir /home/papa/ai/worktrees/expert-store-1572ea7fa/build-expert-store --output-on-failure -R "^test-expert-store$"

Test #44: test-expert-store ................   Passed    0.01 sec
100% tests passed, 0 tests failed out of 1
validated_head=a8364460999b94ac2ef52cf28bd1d0d4c925eef2

This executes the Linux-only final-expert/non-aligned-EOF O_DIRECT regression. The new Windows policy is fail closed: llama_file::has_direct_io() reports false until FILE_FLAG_NO_BUFFERING exists, so direct_io=true with allow_buffered_io=false fails explicitly. allow_buffered_io=true remains an explicit non-bounded portability/test opt-in.

[ajaxdude]

CI follow-up: the Windows job passed, including the new Windows fail-closed regression. Ubuntu was retried once and failed again in unrelated server tests: Hugging Face returned HTTP 429 for ggml-org/test-model-router-download, and unit/test_basic.py::test_no_ui returned 404. The retry summary was 2 failed, 243 passed, 3 skipped; neither failure touches the expert-store files or focused test. No further retry is queued.

[ajaxdude]

Stack repair completed with normal merge commit b7e4055d118c6116f9df40ca9880de94406ed8f4, merging repaired schema head ea7793dba3497a95ab2516e7bf36f15846b5bd4b into the expert-store branch. The only semantic conflict was src/CMakeLists.txt; resolution retained the repaired base core-source list and added llama-expert-store.cpp to it.

Post-merge validation:

• test-expert-store: passed
• test-deepseek41-schema: passed
• test-gguf: passed
• full CPU-only test-llama-archs: passed
• focused test-llama-archs -a deepseek41: passed with expected schema-only skips
• git diff --check and staged diff check: passed

No Strix job was restarted; all validation was local.

[ajaxdude]

Post-restack Windows CI failure is unrelated to this PR. Run 34728062210, job 103645675874, completed 99% of the server suite before one router test timed out:

FAILED unit/test_router.py::test_router_delete_model - requests.exceptions.ReadTimeout: HTTPConnectionPool(host="127.0.0.1", port=8110): Read timed out. (read timeout=600)
1 failed, 370 passed, 6 skipped in 816.25s (0:13:36)

No expert-store product or test code is implicated. Per review direction, no code change was made and no retry is queued.

[ajaxdude]

Merged the updated schema support branch normally. The named remote branch resolves to 6a473b7e942a2cc54f1e5bb2b2d5deacd247b888 (the supplied full SHA differed after the first nine characters); that commit is model : classify DeepSeek V4.1 rope type.

New expert-store head: 50e1b27d3a1287ec0b2c9bf19016317ff801d313.

Post-merge local validation passed:

• test-expert-store
• test-deepseek41-schema
• test-gguf
• full CPU-only test-llama-archs
• working-tree and staged git diff --check

No Strix work was performed.