perf(cpp): autotune dual-socket dense models to interleave + all cores

[Jackson57279]

Builds on #29.

PR #29 defaults dense dual-socket models to single-node with half the threads. Benchmarks on ai@192.168.1.132 show that models like Qwen2.5-32B-Q4_K_M (~18 GiB) decode ~68% faster when weights are interleaved across both sockets and all logical cores are used:

config	decode tps
--numa single --threads 48	1.33
--numa interleave --threads 96	2.26
--auto (after this change)	2.23

Changes:

• plan_cpu() now picks interleave + all logical cores for dual-socket dense models > 4 GiB; tiny models stay single-node.
• Add autotune_test coverage for the 32B-class dense dual-socket case.
• Fix a latent KV-cache indexing bug in forward_batched(): when KV cache is capped at load time, the batched prefill path was indexing with config_.context_size instead of the runtime kv_context_, causing out-of-bounds writes. This crash became visible once interleave + 96 threads exercised the capped-KV path.

Validation:

• autotune_test, gguf_mmap_policy_test, prefetch_layer_map_test pass locally and on ai@192.168.1.132.
• --auto benchmark on Qwen2.5-32B-Q4_K_M: decode 2.23 tok/s, prefill 6.17 tok/s.

---

Summary by cubic

Autotunes dense dual-socket models to use NUMA interleaving and all logical cores for faster decode on 32B‑class models. Also pulls in base-branch maintenance, including an anyhow 1.0.103 bump and mmap-policy test updates.

• Performance

  • plan_cpu() now selects --numa interleave and all logical cores for dual-socket dense models > 4 GiB; small models stay single-node.
  • Added autotune_test for a 32B-class dual-socket case; --auto on Qwen2.5‑32B‑Q4_K_M: 2.23 tok/s decode, 6.17 tok/s prefill.

• Bug Fixes

  • In forward_batched(), index KV cache with kv_context_ (not config_.context_size) to prevent OOB writes when the KV cache is capped.

^{Written for commit 079f7cb. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 3 files

_{Re-trigger cubic}

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[Jackson57279]

Triage note (maintainer pass): the +68% dual-socket decode result is compelling and the KV-cache OOB fix in forward_batched() (indexing with kv_context_ instead of config_.context_size) is a real correctness bug.

Two things before this can merge:

1. Full CI has never run here. Base is codex/async-layer-prefetch (perf(cpp): async layer-ahead prefetch + transparent hugepages for mmap'd GGUF #29's branch), not master, so only the cubic check fires — Build/test on the three OSes, CodeQL, udeps and Docker are all skipped. Once perf(cpp): async layer-ahead prefetch + transparent hugepages for mmap'd GGUF #29 lands, please rebase onto master so the full matrix validates it.
2. The OOB fix should not wait on the perf work. It's an out-of-bounds write on the capped-KV path — worth pulling into its own tiny PR against master so it can merge immediately regardless of what happens to the autotune change.

Also note the interleave/all-cores autotune here overlaps #32's NUMA replication logic; worth reconciling so plan_cpu() has one owner.

[cursor]

Review: what this is, how it works, what to steal

What it is. A 3-file follow-on stacked on #29 (codex/async-layer-prefetch), not on master. Changes dual-socket autotune and fixes a real KV OOB in batched prefill. Cannot land independently.

How it works.

• Autotune: for dual-socket dense models > 4 GiB, plan_cpu() picks --numa interleave and all logical cores (96 on the 5220R box). Tiny models stay single-node. Bench on Qwen2.5-32B-Q4_K_M: 1.33 → 2.23 tok/s decode (~+68%).
• Correctness: forward_batched() indexed KV with config_.context_size instead of the runtime cap kv_context_. Single-token forward() already uses kv_context_. The bug shows up when KV is capped (GLM-class huge context) and batched prefill writes past the allocation.

Do not steal the NUMA policy. master already settled on a different rule after this PR: dense ≤192 GiB → --numa single --threads 16; >192 GiB → --numa interleave --threads 48. The 4 GiB + all-logical-cores plan fights that on purpose (cross-node thrash vs bandwidth). Re-bench if we revisit; do not silently override.

Steal immediately (still a live bug on master):

oxidize-cpp/src/model_llama.cpp  forward_batched()
  pos % cfg.context_size  →  pos % kv_context_
  l * cfg.context_size    →  l * kv_context_

That 3-line fix is independent of #29 and of the autotune change. Cherry-pick it on its own; do not wait on this stack.

Closing: stacked on an unmerged base, NUMA policy superseded, no activity since the 2026-07-09 “needs CI against master” note.