CPU kernel parity: decode and tile kernels at llama.cpp speed or better on zen2, zen4, Intel Granite Rapids and M4 Pro

daslib/fio.das

@@ -692,8 +692,7 @@ def rmdir_rec_result(path : string) : fs_result_bool {
 
 def run_and_capture(args : array<string>; var output : string&; timeout_sec : float = 0.0) : int {
     //! Run an external command and capture its stdout+stderr (merged into one pipe by the underlying ``popen_argv``). Returns the process exit code;
-    //! -1 means the spawn itself failed. No shell is involved, but on Windows the program path (``args[0]``) must use backslashes —
-    //! CreateProcess does not resolve ``bin/daslang``-style forward-slash relative paths (probe-verified); ``replace(exe, "/", "\\")`` first.
+    //! -1 means the spawn itself failed. No shell is involved, and a forward-slash ``args[0]`` spawns on every platform (``popen_argv`` hands Windows the backslash spelling).
     var captured : string
     let exit_code = unsafe(popen_argv(args, timeout_sec, $(f) {
         if (f != null) {

modules/dasLLAMA/ARCHITECTURE.md

@@ -55,8 +55,11 @@ re-transcoding `$LCPP/src/unicode-data.cpp`).
   MoE region split.
 - `ARCHITECTURE_MEDIA.md` - sec.2.13-2.16: the padded tower GEMM widths, the family GPU hooks,
   the tower weight lane, and the plain-Model ASR decoders.
-- `ARCHITECTURE_MEASUREMENT.md` - sec.2.5, 2.10, 2.20: the benchmark rig, the tune gate, and the
-  sanctioned instrumentation rails.
+- `ARCHITECTURE_MEASUREMENT.md` - sec.2.5, 2.10, 2.20, 2.21, 2.26-2.27: the benchmark rig, the tune
+  gate, the sanctioned instrumentation rails, kernel-race fidelity, the gemv's own tune seat, and
+  the CPU kernel bench's fixture conditions.
+- `ARCHITECTURE_CPU_KERNELS.md` - sec.2.22-2.24: the sub-block-packed k3/k6 planes, the grid
+  formats' panel and row-group decodes, and the VBMI symbol lattice.
 
 ## 3. Inherited invariants
 

modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md

@@ -0,0 +1,45 @@
+# dasLLAMA architecture - the CPU kernel planes and decodes
+
+Companion of `ARCHITECTURE.md` (contract: `../../ARCHITECTURE_COMMON.md`). Section 2 here continues
+the mechanism numbering; each section is cited by the code that embodies it.
+
+## 2. Mechanisms
+
+### 2.22 The k3 and k6 planes are packed per sub-block {#kq-subblock-planes}
+
+A k6 grp<mr> plane's qh columns `2blk` and `2blk + 1` carry one sub-block's four `j` sites, each as
+a 2-bit field at bit `2j`; the disk byte `h*32 + half*16 + j*4 + t` feeds sub-blocks `4h..` at bit
+`2b`. k3 packs the same way - qs columns `2blk + half` carry the sub-block's four `j` sites at `2j`,
+and the hmask column `blk` carries its eight sites at bit `s` (lo at `j`, hi at `4 + j`). One
+sub-block's decode then costs two loads for k6 and three for k3, and nothing loaded lives past it;
+the row-interleaved disk order cost one load per `j`. The layout is CPU-flavor: the `.dlim` a box
+bakes is for the hardware that runs it, so a CPU plane owes nothing to the GPU tiers' shapes.
+`IMAGE_VERSION` 28 is this layout.
+
+### 2.23 A grid format's CPU gemv decodes as a panel or as row groups {#grid-decode-forms}
+
+Five formats (iq3s, iq3xxs, iq2s, iq2xs, iq2xxs) have two gemv decode forms. The PANEL form gathers
+a superblock into an alloca panel first and reads packed positions through one dword load per
+4-byte column, the four positions of a column sharing the load. The ROW-GROUP form composes a
+weight-width vector straight from the grid words - width/64 rows x 8 weights, one u64 grid entry
+per iq2 row - and reverts to byte loads, because the column dword read only pays inside the panel.
+The sdot lattice always takes row groups; on x86 the panel's latency chain does not scale with the
+core, so `x86-vnni512` takes row groups for iq2xxs and `x86-amx` for iq2xxs and iq3xxs, everything
+else the panel. A VBMI seat (sec.2.24) takes row groups unconditionally.
+
+### 2.24 The VBMI symbol lattice {#vbmi-lattice}
+
+On a zmm VBMI target a grid block decodes as row groups through a symbol lattice. Every grid byte
+comes from a tiny alphabet (three symbols for the iq2 family, eight for iq3), so the grid is baked
+as two compact code planes - entry `e`'s low and high half, four 2-bit symbols each for iq2, two
+3-bit for iq3 - plus the alphabet as a per-lane `vpshufb` table and `ksigns_iq2xs` whole (128 bytes:
+exactly the two registers one `VPERMI2B` indexes). Per format the block's index bytes gather into one
+64-lane vector per column (lane `r*4 + position`) and look up in the code planes - `VPERMI2B` per 128
+entries, index bit 7 blends the pairs, the 9th and 10th index bits arrive as lane masks (iq3s and iq2s
+from the row's qh byte, iq2xs from bit 0 of its u16 word's high byte). The row's sign bytes land in
+the same lane layout: the plane's own column for iq3s and iq2s, one ksigns `VPERMI2B` over the 7-bit
+codes for the rest. Per row group and weight octet a constant two-source shuffle places each row's
+code bytes in its qword, `VPMULTISHIFTQB` spreads the symbols into bytes, one `vpshufb` maps them to
+magnitudes, and the signs ride the activation copy as a mask `(x ^ m) - m`. The lattice row shares
+its tile body and planes with the 512/mr16 row, so only the gemv differs - what the gemv's own seat
+(`ARCHITECTURE_MEASUREMENT.md` sec.2.26) races.

modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md

@@ -160,3 +160,32 @@ no model, no tuner, every arm gated against a CPU plane-dequant oracle before it
 chain every dispatch through ONE shared output buffer on purpose - the serialized regime is
 the instrument's probe shape, imitating the reference tool it is compared against - and its
 numbers reach the engine only through a human porting decision, never a minted crown.
+
+### 2.26 The gemv takes its own tune seat {#gemv-seat}
+
+A kq family's manifest entry is its tile-best row, and the gemv gets a SECOND entry when a different
+row serves the streamed decode better. Only same-mr rows can differ, because the layout companion
+pins the plane's interleave; of those the two best by tile time race, the winner takes the gemv only
+by the margin over the tile winner's own gemv, and the incumbent keeps a tie. Every family's perm grid
+therefore carries a 256-wide `mr = 16` alternate beside its 512-wide tile crown. The seat is decided
+at the engine's decode shape - a DRAM-bound plane streamed by every lane through the engine's own
+splitter - because the engine's row length moves the answer (k3 on Granite Rapids: the 256 seat wins
+at n=2048 and loses at 14336 - `benchmarks/matmul/kq_kernel_bench.das`, tune mode, seats pinned, d=32768). The seat fixture is a 512-row build at the ffn width tiled 320 times,
+past the largest L3 a socket lends a slice of, and the seat takes the MEDIAN of seven rounds: a round
+that finds the plane in L3 must not crown it. In normal mode `llvm_tune` stamps a companion from its
+own manifest entry when one exists and is a perm this box can run, else from the tile's.
+
+### 2.27 The CPU kernel bench's fixture conditions {#cpu-kernel-bench-fixture}
+
+`benchmarks/matmul/kq_kernel_bench.das` times raw kernels on synthetic planes, and three fixture
+properties decide whether its numbers mean anything. Every plane of one format lives in ONE arena at
+fixed offsets, staggered so no two starts share their low 12 address bits: the heap places separate
+arrays at run-dependent relative addresses, and planes that alias in the L1/L2 set logic make a run's
+time depend on where the heap put them. Scale planes are filled with a byte that is a normal number in
+every scale form, never random bytes, because denormal math runs orders of magnitude slower. Each row
+is warmed before it is timed - three unmeasured rounds solo, six dispatches per row on the team arm -
+because a core ramps over several rounds and one warm call is not enough. The q8 row exists in two
+flavors: f32 group scales (the engine's own quantization) and `q8s16` over binary16 scales - the
+wscale_f16 rail a GGUF q8_0 tensor runs, and the like-for-like row against the reference's q8_0.
+Provenance for every figure in this section: `benchmarks/matmul/kq_kernel_bench.das` under
+`DAS_TUNE_MODE=tune`, one thread, its default `--fmt` / `-n` / `-d` shape.

modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md

@@ -260,6 +260,19 @@ loses cross-module inlining); the first run after a cache write pays one cold co
 deser re-key); QUIRK 21 still applies to emitter edits. Numbers, caveats and the
 invalidation ledger: `plans/jit_compile_time.md`.
 
+**Kernel-loop invocation (adopted 2026-09-01):** a kernel spelling is raced without a model in
+`benchmarks/matmul/kq_kernel_bench.das` - `DAS_TUNE_MODE=tune bin/Release/daslang.exe -jit
+modules/dasLLAMA/benchmarks/matmul/kq_kernel_bench.das -- --fmt <fmt> --perm <substr>` times every
+row of the tile's and gemv's `_variants()` registries at one thread on synthetic planes (seconds
+per try), and the reference row at the same shape is the reference exe's `test-backend-ops perf
+-o MUL_MAT -p "type_a=<type>,type_b=f32,m=4096,n=1,"` under `GGML_BENCH_THREADS=1`;
+`harness/kernel_ladder.sh` runs both sides for every format and prints the box's ratio table. The
+app run comes only after a spelling wins there. Procedure, fact base and work queue:
+`plans/kernel_parity_pass.md`.
+2026-09-01: the pass closed the M1 CPU at kernel level - all 32 ladder rows at or above the
+reference (the grid decodes 1.16-1.51x, from 0.51-0.93x) - so the per-format M1 CPU tg tails stamped
+below (0.51x-0.73x) predate the ARM row-group decode; re-stamp the vehicles before quoting them.
+
 A real file whose every tensor type is now loadable (the header census script in the session
 scratchpad, or `harness/gguf_dump.das`), through `examples/dasLLAMA/run.das` against
 `simple_ids.exe` from the llama.cpp reference build for the same prompt; then `test_model_image`
@@ -554,6 +567,9 @@ only the step-3 0.0267 top-2 flip vs llama.cpp. M1 benches: CPU das 897.4/56.3 v
 is CLOSED - and with it THE FORMAT LADDER: four-tier table zen2 2.86x/0.70x, vk 0.78x/0.70x,
 M1 CPU 6.41x/0.57x, Metal 0.93x/0.78x.
 
+2026-09-01 (kernel parity pass): zen2 16t (`benchmarks/lcpp_bench.das --for-debug-purposes`, debug-jit) vs the clean-cpu `llama-bench` on the IQ2_XXS-local 1B: pp512 506.9 vs
+181.2 (2.80x), tg128 88.6 vs 86.2 (1.03x, was 0.70x) - the sign column + u64 pair gemv.
+
 ### IQ2_XS Phase A (CPU, 2026-08-31) - the ksigns u64 tier
 
 Shape: 256-superblock grid format - each of the 32 u16 qs words carries a 9-bit index into
@@ -621,6 +637,9 @@ tiers. M1 16GB benches: CPU das 746.6/51.6 vs llama.cpp 144.8/101.1 (5.16x/0.51x
 (0.93x/0.86x - the iq2s pp class). The format is CLOSED on all four tiers; four-tier table:
 zen2 2.78x/0.70x, vk 0.77x/0.54x, M1 CPU 5.16x/0.51x, Metal 0.93x/0.86x.
 
+2026-09-01 (kernel parity pass): zen2 16t (`benchmarks/lcpp_bench.das --for-debug-purposes`, debug-jit) vs the clean-cpu `llama-bench` on the IQ2_XS-local 1B: pp512 480.7 vs
+174.6 (2.75x), tg128 93.8 vs 85.9 (1.09x, was 0.70x) - sign column + u64 pair + column dword read.
+
 ### IQ2_S Phase A (CPU, 2026-08-31) - the u64-grid tier
 
 Shape: 256-superblock grid format, the first with a u64 grid - a 10-bit index (qs byte |
@@ -699,6 +718,7 @@ the mradermacher i1 vehicle - IQ2_S attn x32 + IQ3_XXS/IQ3_S/Q4_K/Q5_K):
 | tier | pp512 das / llama.cpp | tg128 das / llama.cpp |
 |---|---|---|
 | zen2 CPU | 501.5 / 138.5 (3.62x) | 55.8 / 73.5 (0.76x) |
+| zen2 CPU, 2026-09-01 kernel parity pass | 494.4 / 138.1 (3.58x) | 71.9 / 74.5 (0.97x, +-4.2) - sign column + u64 pair |
 | 5060 Ti Vulkan | 12099.3 / 17377.5 (0.70x - the tier class) | 292.4 / 362.5 (0.81x) |
 | M1 CPU | 883.3 / 413.4 (2.14x) | 53.7 / 73.9 (0.73x) |
 | M1 Metal | 3170.5 / 3427.2 (0.93x) | 205.8 / 220.7 (0.93x) |
@@ -915,6 +935,7 @@ the local --tensor-type requant, iq3_xxs on attn_k/q + all ffn):
 | tier | pp512 das / llama.cpp | tg128 das / llama.cpp |
 |---|---|---|
 | zen2 CPU | 507.2 / 136.0 (3.73x) | 56.7 / 72.6 (0.78x) |
+| zen2 CPU, 2026-09-01 kernel parity pass | 501.0 / 135.8 (3.69x) | 74.9 / 74.3 (1.01x) - sign column + column dword read |
 | 5060 Ti Vulkan | 12225.7 / 17807.7 (0.69x - the tier class) | 372.1 / 389.9 (0.95x) |
 | M1 CPU | 906.0 / 410.5 (2.21x) | 53.5 / 74.0 (0.72x) |
 | M1 Metal | 3224.0 / 3429.9 (0.94x) | 213.5 / 227.3 (0.94x) |
@@ -1017,6 +1038,7 @@ embedding head is Q6_K - three formats share every decode step):
 | tier | pp512 das / llama.cpp | tg128 das / llama.cpp |
 |---|---|---|
 | zen2 CPU | 516.9 / 104.9 (4.93x) | 52.4 / 57.0 (0.92x) |
+| zen2 CPU, 2026-09-01 kernel parity pass | 526.1 / 104.7 (5.03x) | 69.9 / 56.5 (1.24x) - the vector sign column gemv |
 | 5060 Ti Vulkan | 12539.6 / 17865 (0.70x) | 288.1 / 324.2 (0.89x) |
 | M1 CPU | 886.2 / 433.6 (2.04x) | 57.4 / 66.6 (0.86x) |
 | M1 Metal | 3237.6 / 3344.3 (0.97x) | 199.4 / 209.0 (0.95x) |