engram : add bounded disk table and hash core

[ajaxdude]

Overview

Add the model-free DeepSeek V4.1 Engram core on top of the schema branch from halo-box#49.

• validate the exact published e4m3_e8m0_32_row264 metadata layout
• compute chunk-independent 4-token history hashes for two layers and 24 row IDs per token per layer
• decode 264-byte rows as 256 E4M3 values plus eight E8M0 scales with reference BF16 round-to-nearest-even and signed-zero behavior
• keep Engram tables outside mmap and the page cache through bounded aligned reads with explicit failures
• deduplicate repeated rows in bounded 2,048-token batches
• extract the aligned reader used by qwen4exp PLE and retain its cache and parallel gather behavior

This is the bounded disk-table, decoder, and token-history hash layer only. It does not add attention graph injection, routed-expert streaming, a memory watchdog, or real-model execution.

Depends on halo-box#49 and implements the Engram-core portion of halo-box#48. This fork PR is for stacked review and will be recreated or retargeted after ggml-org#49 merges.

Measurements

No performance claim is made. This PR is model-free infrastructure and was not run against the published model payload.

Device:     Apple Silicon development host
Memory:     N/A
Power:      N/A
BIOS:       N/A
Kernel:     macOS
Backend:    CPU-only model-free tests; no inference backend
Build:      -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_SERVER=OFF -DGGML_METAL=OFF
Baseline:   65fdc19831d30b12f06b981c3ca74dd18731f90e
Change:     68df4a972
Model:      none

Baseline: N/A - no throughput or memory claim.

After: N/A - no throughput or memory claim.

Correctness:

• pinned ds4 oracle bd66c402070042bf0a79ad6ece8242de4c93680c: make test-engram
• ctest --test-dir build-engram --output-on-failure -R '^(test-deepseek41-schema|test-engram|test-generate-models)$'
• ASan and UBSan build: ctest --test-dir build-engram-sanitize --output-on-failure -R '^test-engram$'
• corrected-schema head 68df4a97268c457a8b9a1e199864a79d8df03097: Python DeepSeek41 schema tests - 4/4 passed; ctest --test-dir build-engram --output-on-failure -R '^(test-deepseek41-schema|test-engram|test-generate-models)$' - 3/3 passed; ASan/UBSan schema and Engram tests - 2/2 passed
• independent code review of the exact eight-file diff at 68df4a97268c457a8b9a1e199864a79d8df03097: no significant issues found
• Linux Strix Halo host at 9a18bb4288547c093ea922a97efba3caef12e640: cmake -S . -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF, cmake --build build --target test-engram -j4, then ctest --test-dir build --output-on-failure -R '^test-engram$' - 1/1 passed, including the O_DIRECT EOF-tail regression

The focused test covers layout rejection, full-history and chunk equivalence, masks, invalid tokens without history mutation, large offsets, repeated and batched rows, all E4M3/E8M0 code combinations, BF16 rounding, signed zero, invalid row IDs, short reads, and arithmetic overflow. It also exercises qwen4exp PLE gathers through the extracted reader in buffered and uncached modes, verifies threaded read exceptions propagate without termination or deadlock, and includes a Linux O_DIRECT regression for a valid final row shorter than the aligned bounce length.

Additional information

The existing published GGUF remains the compatibility target. It uses engram_q_norm, engram_k_norm, and engram_kv, carries mapped deepseek41.engram.pad_id, stores primes as UINT32 and multipliers as UINT64, and omits explicit offsets because offsets are the cumulative prime sums.

The newer ggml-org#28696 converter instead writes engram_q, engram_k, and engram_wkv, plus explicit offsets and shape keys. This PR does not rename the published tensor contract or add unneeded aliases; a later model-loader integration can accept those names and metadata forms additively after validating that offsets equal the cumulative published layout.

Exact internal API added:

• llama_engram_hasher(layout) validates immutable metadata; hash(history, tokens, mask, count, rows) emits [token][layer][column]
• llama_engram_history::reset() establishes the dead/pad prefix state
• llama_engram_decode_row() decodes one native 264-byte row
• llama_engram_table(path, offset, rows) opens a strict uncached extent; read() and read_batch() return BF16-rounded values in F32 storage

Graph integration must keep one history per sequence, pass the token mask used by the reference, select the 24-row layer slice, stage only the returned packed rows, suppress Engram output at masked positions, and never create, map, prefetch, warm, or backend-admit the full table tensor.

Requirements

• I have read and agree with the contributing guidelines
• This change is Strix Halo specific, or justified by measurements on Strix Halo. General llama.cpp improvements belong in halo-box/llama.cpp instead
• AI usage disclosure: AGENT-AUTHORED by GPT-5.6 Sol. The agent inspected the pinned ds4 oracle and upstream converter work, implemented the core and tests, addressed independent review findings, ran validation, and prepared this PR.
• What was NOT verified: no DeepSeek V4.1 model payload was loaded; no attention graph, real-model inference, ROCm, Vulkan, expert streaming, memory-watchdog, or GPU performance validation was run.