CPU kernel parity: decode and tile kernels at llama.cpp speed or better on zen2, zen4, Intel Granite Rapids and M4 Pro by borisbat · Pull Request #3924 · GaijinEntertainment/daScript

borisbat · 2026-09-02T04:06:41Z

Behavior change: every .dlim image re-bakes on first load (IMAGE_VERSION 28) and every JIT cache re-mints (LLVM_JIT_CODEGEN_VERSION 0x68); popen_argv, spawn_argv and popen_argv_pipe now accept a forward-slash args[0] on Windows.

This is the CPU half of the kernel parity pass. The goal was every CPU decode and tile kernel at the speed of llama.cpp or better, measured kernel against kernel on four boxes: zen2 (3990X), zen4 (c7a EPYC 9R14), Intel Granite Rapids (c8i Xeon 6975P-C) and Apple M4 Pro. The goal is met on the ruled terms. Intel reads at or above 0.95 of the reference on every ladder row. zen4 does too, except a ruled set of streaming formats (iq4xs, k5, iq4nl, q40 at 0.91 to 0.99) that ledger entry 67 owns. M4 reads at or above 0.95 at the engine shape on every row. zen2 closed earlier in the arc.

The kernel work, in the order it landed: the grid formats' gemv applies signs as one masked negate per weight vector; the iq2 formats load their grid pair as one u64; ARM gets a row-pair decode plus a ksigns table for the five grid formats; k6 and k3 pack their high-bit planes per sub-block (the image version bump); k6 and k3 flush their scales fused into the accumulator; k4 and k5 share one fused form; the frame is realigned to 64 bytes at every kernel entry; and on AVX-512 VBMI the five grid decodes become a symbol lattice (one vpermi2b per 64 weights), which is where the zen4 and Intel grid rows crossed 1.0.

The tune framework grew what the work needed. A gemv now races its own seat, restricted to the tile's top two same-layout forms, and writes a companion entry. The seat race runs at the engine shape on a fixture past the L3 with a median of seven rounds, because an L3-resident fixture crowned the wrong Intel k3 seat. Class profiles are re-minted for x86-amx, x86-vnni512 and arm-i8mm. HOW_TO_GET_SIDECAR.md is the walk from a fresh rented box to a committed class profile.

Two defects found on the way are fixed at the source. The llvm_boost GEP wrapper set the inbounds flag on a folded constant GEP, which corrupted LLVM's heap and crashed cold starts; it now builds inbounds GEPs directly. And the board rig's parity pregate never ran on Windows because CreateProcess rejects a relative forward-slash args[0]; the spawn now respells it natively, with a test.

Where to look: modules/dasLLAMA/dasllama/dasllama_math_gen.das (the emitter, the lattice, the sign column), modules/dasLLVM/daslib/llvm_tune.das (the gemv-own seat and the companion write), modules/dasLLAMA/harness/ (the seat race fixture, the kernel ladder), src/builtin/module_builtin_fio.cpp (the spawn respelling).

Validation, claims, ledger

Validation

gen_tune_probe TEST mode: every generator variant bit-exact against the reference decode on zen2, zen4 (c7a), Intel (c8i Granite Rapids, AMX leg included) and M4 Pro.
The gemma-4-12B parity pregate: 40/40 token-for-token on zen2, standalone and under the board rig.
The board cell for gemma-4-12B Q4_K_M on zen2 was re-measured on the fixed rig (das cpu against the clean-cpu reference, reference unpinned as the 08-04 sweep ran it): pp512 1.18x and tg128 1.04x of the reference, against 1.19x and 1.07x on the 08-04 row. Both engines read about 5% lower than in August on this box; the ratio held. The row is not stored: the records gate rejects a run stamped with the Parsec daemon, which runs as a service on that box. Ledgered in the plan with the instrument; see Not done.
The goal ladders (v8 tables, plans/kernel_parity_pass.md section 2) come from harness/kernel_ladder.sh on the rented boxes; the boxes are terminated, the logs are archived with the plan.
Module suites run locally under -jit: modules/dasLLAMA/tests/test_kquant.das, test_dispatch_shaping.das, the full modules/dasLLVM/tests/ directory, tests/language/cpu_supports.das, tests/fio/popen_argv.das.
Forced-feature probe on zen2 (DAS_JIT_X64_FORCE_FEATURES with the AVX-512, VBMI and AMX rows, DAS_TUNE_MODE=test): the stamp plan arms the extra grid row for the five grid families (6 permutations against 5), then the run dies on the first AVX-512 instruction as a box without the silicon must. The gate's compile side is exercised here; its execution side was validated on the real boxes (Intel c8i, zen4 c7a).
Full local preflight ran once on the frozen tree.

Claims - stated, not tested

The VBMI gate rides x64_tier_feature exactly like the existing tiers, so write_exe sees no new branch; verified by reading the tier code, a break would be an exe that stamps a VBMI body on a non-VBMI host.
The Intel k6 16-lane mode gap (normal 1.44x slower than tune mode on the pre-rebase tree) vanished after the rebase onto master and was not bisected; if it returns, bisect llvm_tune.das between bbatkin/kernel-parity-pre-rebase and this tip.

Not done

followup_general 66: x86 hybrid core detection for the lane policy.
followup_general 67: per-format decode lane cap on SMT x86 (the ruled zen4 streaming set).
followup_general 68: iq2xs on arm-i8mm, 0.90 one thread; the cheaper 9-bit index read and an SMMLA lattice analog.
followup_general 69: a class profile carrying winners a class member cannot run (Cascade Lake).
The zen2 board row for gemma-4-12B Q4_K_M: measured (above), to be stored by a ten-minute rig run with Parsec stopped, the only way a row passes the records gate on that box.
Vulkan gap 1 (the budget split) is the next half of the arc.

🤖 Generated with Claude Code

https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

Copilot

🔵 Needs a closer look

It combines broad, performance-critical kernel/tuning changes across dasLLAMA + dasLLVM (plus cache/versioning and platform spawn behavior), which warrants final human review despite targeted tests.

Pull request overview

This PR advances dasLLAMA CPU kernel parity by adding/adjusting CPU feature gates, retuning/tune-framework behavior (including companion/“gemv seat” selection), and updating kernel/data-plane layouts—along with the necessary cache/version bumps and platform fixes (notably Windows argv[0] spawn path handling).

Changes:

Add avx512vbmi as a first-class x64 tier feature (cpuid + JIT target gates + tune feature lists) and incorporate it into JIT cache salting.
Extend the tune framework to allow a [tune_companion] to stamp from its own manifest entry when present/eligible, and add tests enforcing the new behavior.
Update Windows process spawning to accept forward-slash relative args[0], with corresponding tests and docs updates.

File summaries

File	Description
tests/README.md	Update test index entry to note forward-slash relative argv[0] coverage.
tests/language/cpu_supports.das	Add a constraint test that `avx512vbmi` implies `avx512f`.
tests/fio/popen_argv.das	Add a regression test for forward-slash relative argv[0] spawning.
tests/dastest/test_preflight_config.das	Remove Windows-only slash normalization now handled by spawn implementation.
tests/dastest/test_dasfmt_exclude_mask.das	Remove Windows-only slash normalization now handled by spawn implementation.
src/builtin/module_builtin_runtime.cpp	Add cpuid support for `avx512vbmi`.
src/builtin/module_builtin_fio.cpp	Normalize Windows argv[0] to native backslashes when building the CreateProcess command line.
skills/tune.md	Document companion-specific manifest entries and update CPU class naming notes.
skills/LAWS.md	Record the ruling that gemv companions get their own manifest entry (top-2 race).
skills/daslang/references/files-and-paths.md	Update docs to reflect that forward-slash args[0] is supported cross-platform.
plans/kernel_parity_research_zen4.md	Add zen4 grid-decode research memo (read-only plan artifact).
plans/kernel_parity_research_vk.md	Add Vulkan parity research memo (read-only plan artifact).
modules/dasLLVM/tests/llvm_tune_profiles.das	Strengthen class-chain test to ensure own class heads the ladder.
modules/dasLLVM/tests/llvm_tune_manifest.das	Add coverage for companion “own entry” stamping behavior.
modules/dasLLVM/tests/llvm_tune_manifest_client.das	Add a `write-own` path to set the companion’s manifest entry.
modules/dasLLVM/tests/llvm_gep_inbounds.das	Add an enforcement test banning `LLVMSetIsInBounds` outside bindings/.
modules/dasLLVM/REVIEW.md	Codify new review constraints for inbounds GEP and feature gating completeness.
modules/dasLLVM/daslib/llvm_tune.das	Implement companion “own manifest entry” stamping; add `avx512vbmi`; introduce `x86-amx` class.
modules/dasLLVM/daslib/llvm_jit_run.das	Bump codegen version/hash; incorporate VBMI/AMX into cache salt.
modules/dasLLVM/daslib/llvm_jit_common.das	Add `g_target_x64_avx512vbmi` and hook it into x64 tier gating.
modules/dasLLVM/daslib/llvm_boost.das	Fix GEP wrapper to build inbounds GEPs directly (avoid `LLVMSetIsInBounds`).
modules/dasLLVM/ARCHITECTURE.md	Document x64 tier-gate model and the constant-folded GEP hazard.
modules/dasLLAMA/tests/test_kquant.das	Add tests for VBMI lattice table round-trips and GPU gather invariants.
modules/dasLLAMA/tests/test_dispatch_shaping.das	Add a test pinning the default `gemv_chunks_per_lane`.
modules/dasLLAMA/REVIEW.md	Clarify checklist surfacing and add reserve-before-resize rule for dimension-scaled buffers.
modules/dasLLAMA/REVIEW_MEASUREMENT.md	Require board-row re-mints when optimizing already-served paths.
modules/dasLLAMA/performance/profile_common.das	Improve error logging for spawn failures; remove Windows slash normalization.
modules/dasLLAMA/performance/defaults/x86-vnni512.tune-defaults.json	Update vnni512 defaults to new winners/features (and new formatting style).
modules/dasLLAMA/performance/defaults/x86-amx.tune-defaults.json	Add a new defaults profile for the `x86-amx` class.
modules/dasLLAMA/performance/defaults/arm-i8mm.tune-defaults.json	Update arm-i8mm defaults profile (winners/provenance).
modules/dasLLAMA/HOW_TO_GET_SIDECAR.md	Add a full walkthrough for minting/exporting a CPU-class sidecar/profile.
modules/dasLLAMA/HOW_TO_ADD_A_FORMAT.md	Update kernel-parity procedure notes and add new measured rows.
modules/dasLLAMA/harness/README.md	Document `kernel_ladder.sh` purpose and requirements.
modules/dasLLAMA/harness/kernel_ladder.sh	Add a script that builds a per-box kernel ladder vs llama.cpp’s perf rows.
modules/dasLLAMA/harness/gen_tune_probe.das	Add gemv seat selection and a team/engine-shape gemv measurement path.
modules/dasLLAMA/followup_vulkan.md	Correct the Vulkan pp512 gap diagnosis and link to research memo.
modules/dasLLAMA/followup_general.md	Update CPU grid-format decode followups and extend the followup ledger.
modules/dasLLAMA/dasllama/dasllama_repack.das	Add reserves and change k3/k6 plane packing per sub-block.
modules/dasLLAMA/dasllama/dasllama_math.das	Update gemv-chunking comment to match current behavior/trigger conditions.
modules/dasLLAMA/dasllama/dasllama_math_gen.das	Add new tune perms/gates (incl. VBMI grid seats) and align scratch to 64 bytes.
modules/dasLLAMA/dasllama/dasllama_math_default.das	Update dequant paths to match new sub-block-packed plane layouts.
modules/dasLLAMA/dasllama/dasllama_layout.das	Update GPU gather logic to match new k3/k6 packed plane shapes.
modules/dasLLAMA/dasllama/dasllama_image.das	Bump `IMAGE_VERSION` to force re-bake under new k3/k6 plane packing.
modules/dasLLAMA/ARCHITECTURE.md	Extend architecture index to include new measurement/kernel-planes docs.
modules/dasLLAMA/ARCHITECTURE_MEASUREMENT.md	Document the gemv “own seat” mechanism and CPU kernel bench fixture constraints.
modules/dasLLAMA/ARCHITECTURE_CPU_KERNELS.md	Add CPU kernel plane/decode architecture companion doc (new).
daslib/fio.das	Update `run_and_capture` docstring to match new Windows argv[0] handling.

Review details

Files reviewed: 54/54 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…ows, and the two research memos kq_kernel_bench.das times every row of a kq kernel's tile and gemv registries at one thread on synthetic planes - the inner loop of kernel work, seconds per try, no model. The reference row is the reference exe's test-backend-ops perf at the same shape; the fact base it produced (the five grid formats at 48-51 ns per superblock against k4/k2's 8.6-8.8, i.e. 0.43x-0.91x of the reference decode kernel) lands in plans/kernel_parity_pass.md with the procedure and the work queue. Two memos (CPU grid vec_dot, Vulkan cm2 mul_mm) name what llama.cpp does that we do not, cited; followup_general 61 and followup_vulkan 34 are rewritten on the measured facts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…lumn negates whole vectors The panel holds raw grid words; per block one 32-byte load of the plane's sign column (the 8-weight sign bytes sit at row stride 4 for these two formats), a constant shuffle + and + cmpeq builds the byte mask, xor + sub negates the vector. One thread, m=4096 k=14336: iq3s decode 11581 -> 7732 us (the reference exe 10340), iq2s 12413 -> 7076 us (5074). Every variant bit-exact in gen_tune_probe TEST mode. The five gather emitters collapse into one gather over per-format decode functions. gather="reg" (compose the vector with insertelement, no panel) was measured first and dropped: 90.6 ns/superblock against the panel's 49.1, from 1025 insertelement chains and a 5x code body. kq_kernel_bench's tile arm hands the five grid formats the caller-unpacked panel form they read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…xs - the sign column synthesized from the 7-bit codes The 8th sign bit is the code's parity (ksigns[i] == i | parity(i) << 7), so the sign-byte column the masked negate wants is seven vector ops on the aux column (iq3xxs, iq2xxs) or on the u16 word column (iq2xs) - no plane layout change, no image version bump, nothing for the disk kernels or the GPU tiers. One thread, m=4096 k=14336: iq3xxs 11308 -> 7882 us (reference 6590), iq2xxs 10967 -> 7832 (5124), iq2xs 11711 -> 9197 (5386). All 20 sign_vec variants ok in gen_tune_probe TEST mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…te load iq2s, iq2xs and iq2xxs index the u64 grid as an even/odd dword pair; both halves now come from one load with the high half split off in registers (the h = 0 and h = 1 decodes share it through CSE). With sign=vec, one thread, m=4096 k=14336: iq2s 7076 -> 5152 us (reference 5074), iq2xxs 7832 -> 5209 (5124), iq2xs 9197 -> 6297 (5386). A single u16 load for iq2xs's word measured slower (6991) and stays two byte loads. 85 variants ok in gen_tune_probe TEST mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…' gemv path - the sign knob and its 20 seats go The tuner races the tile, where a gemv-only spelling is an identical body and ties, so --tune-only crowned the old seat; and on the M1's sdot lattice the vector column lost nowhere (iq3s 7383 -> 6105 us, iq2s 7161 -> 5012, iq2xxs 7602 -> 6293, iq2xs 7878 -> 7081, iq3xxs 7516 -> 7452, one thread). A win on every tier is not a knob: the gemv always negates per vector off the sign-byte column, the decode functions return raw grid words, the smask and ksigns globals go. Shipped profiles keep their seat names; LLVM_JIT_CODEGEN_VERSION 0x5a re-keys the JIT caches for the emission change. Plain seats on zen2: iq3s 7537, iq3xxs 7862, iq2s 5202, iq2xs 6501, iq2xxs 5121 us; 65 variants ok in TEST mode; test_kquant 265 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…tes and iq2xs's words One aligned dword load per 4-byte column with the byte split off in registers, the column shared through CSE. Measured per format at one thread, m=4096 k=14336: iq3xxs 7862 -> 6602 us (reference 6590) and iq2xs 6501 -> 4702 (5386) win; iq3s 7537 -> 8671, iq2s 5202 -> 6187 and iq2xxs 5121 -> 5574 lose and keep their byte loads. 65 variants ok in gen_tune_probe TEST mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…rms for q8, mxfp4 and q5_1, interleaved rounds, and harness/kernel_ladder.sh The bench times every format's registry rows at one thread on synthetic planes with the rows of a format interleaved round-robin (best and median reported); q8 and its mx4 companion ride the q8q8 grid's registries with the tokstep companion's tokens per call, q51 its per-32 planes, and the tile arm hands k5, k6 and the five grid formats the byte-expanded panel they read (kq_reads_packed_planes decides). kernel_ladder.sh runs the bench and the reference exe's test-backend-ops perf at the same shape for every format and prints the box's ratio table; it resolves its tree from its own path, never the global DASLANG. The zen2 table and the 16-thread vehicle rows land in plans/kernel_parity_pass.md and the HOW_TO sections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…r sub-block; k3's decode re-loads its shared columns The per-16 chains of k3 (4 x 1778) and k2 (4 x 762) stay inside i16 for the whole sub-block, so the two mid-block widens go: k2 decode 2061 -> 1286-1423 us (reference 1951-2121). k3's decode shape re-loads the 24 qs/hmask column vectors its eight sub-blocks share (volatile loads, L1 hits) instead of CSE holding them live against 16 ymm registers and spilling: 2828-3198 -> 2445 us (reference 2638-2773). The pin is decode-only - on the tile it forbids the reordering the token reuse lives on (k3 tile 776 -> 1109 ms when tried). 65 variants ok in TEST mode. The k-quant research memo joins plans/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…s; HOW_TO_GET_SIDECAR.md sections 1-5 as run on the zen4 box Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…ls emitters' comments say what they emit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…t-bound formats, the grid gather is the scalar bound on both ISAs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…t VNNI gate), first in the x86 chain An AMX box with no AMX profile adopts the vnni512 profile and races only the seats that class could not answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…medium/xhigh, the budget lever, the token penalty); 63 amended with the rollback cost Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…the compare, as run on the zen4 box Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…e ladder's k-quant rows; the how-to's box log Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…-byte load - k6 decode read 2x slow on Granite Rapids), and the q8 arm arms the AMX witness before racing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

… kq_kernel_bench: normal scale bytes (Intel's denormal penalty bent the q8/mx4 tile rows) and the AMX witness variants armed before racing The class profile differs from x86-vnni512 in 15 of 49 winners, all [tuned] loop-hint kernels preferring vec16 on the Xeon 6975P-C; every generator tile keeps the 512-bit VNNI seat - the AMX tiles raced and lost the q8q8 family. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…d the three box lessons (alignment, denormals, the AMX grant) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…o bench artifacts fixed and the open items Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

… - separate arrays landed at run-dependent relative addresses and k6 decode read 3.9-6.0 ms across identical runs; now 4.0-4.4 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…o rows x 8 weights per vector straight from the grid words The decode shape of the five grid formats on the NEON lattice composes each <16 x i8> weight vector as [rowA 8 weights | rowB 8 weights] from the existing per-dword decodes (one u64 grid entry per row for the iq2 formats), negates it with the sign column broadcast per row half, replicates the 8 activation bytes (ld1r) and feeds the non-indexed sdot; one pairwise add per row quad at block end yields the [A, B, C, D] accumulator the fold already takes - no panel, no per-dword sign math. Gated on DOT_SDOT, one token and width 128, so x86 emission is byte-identical. M1 one thread, us, before -> after (reference): iq2s 4932 -> 3256 (4917), iq3s 6089 -> 4937 (5682), iq2xs 6375 -> 4244 (3274), iq2xxs 6063 -> 4085 (3437), iq3xxs 6735 -> 5235 (4867). gen_tune_probe TEST 65/65 on the M1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…ace - every ladder row at or past 0.95 of llama.cpp 56 entries, provenance ok/ok. 16 lanes: q8 0.96, q8s16 1.02, k6 1.06, k3 0.95, the grids 1.17-2.35; one thread every row >= 1.15; tiles clean. The Intel half of the goal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…at race; the zen4 goal table 49 entries, provenance ok/ok; the five grid families' tile crowns are the vbmi rows (a tile tie with the 512 body they share), so their gemv companions carry the lattice. 16 lanes: every grid 1.19-1.51, k3/k6/q51/mx4/k2 >= 0.99, q8s16 0.96; the accepted streaming set (iq4xs, k5, iq4nl, q40) 0.91-0.99 across runs and the f32-scale q8 at its 0.944 byte cap. One thread every row >= 1.07. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…r-gates, cited from init_jit_target_flags), two REVIEW rules, the companion-seat rename The GEP wrapper's inbounds rule and the three-part tier-feature rule (cpuid table + TUNE_KNOWN_FEATURES + gate) land in REVIEW.md; the nine g_target_x64_* gates and how each is decided land as an anchored section; csuffix -> companionOwnSuffix, own -> stampedOwn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…es, the grid decode forms, the VBMI lattice) in ARCHITECTURE_CPU_KERNELS.md, two in ARCHITECTURE_MEASUREMENT.md (the gemv seat, the bench fixture), cited from every function that embodies them 189 added comments: 16 facts land as anchored sections with 11 [arch] citations across the emitter, the reference decodes, the repack, the layout gather, the harness and the bench; one REVIEW rule (reserve before a dimension-scaled resize); the undeclared DASLLAMA_GRID_ROWS_X86 knob is gone (the tune rows are the lab); frame_align64 is gone (an alignment experiment the measurements never confirmed); renames: TileEmit.gridPanel / gridGlobal, the bench's gemvKq / tileKq and gemvW_row / tileW_row; the by-value-table frame rule was rejected (its premise, the frame-size crash, was disproved by the GEP wrapper finding). Lint: zero warnings on the changed set, LINT026/027 clean over both modules. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…tries, TEST 103 ok); the M4 v2 table

…group decode (0.90/0.95; the ARM panel/row choice is the fix)

…e ARM standing - the old 1.46 does not reproduce with its own sources; the row form beats the panel on ARM everywhere

…, the GEP mechanism as its own anchored section, the tier rule as the invariant that holds, the emitter pin recomputed Eight gates, not nine; a generic target drops to forced-only truth rather than all-off. ARCHITECTURE.md#gep-constant-fold carries the ConstantExpr mechanism, cited from the llvm_boost wrapper; the REVIEW rule shrinks to the decidable ban (never LLVMSetIsInBounds). The three-part tier rule becomes "a name used in requires= or a gate has its cpuid line in the same diff" - five known names legitimately have no gate of their own; #x64-tier-gates gains the fingerprint citer. LLVM_JIT_EMITTER_HASH recomputed for the llvm_boost change (0x26d3a061cfe6fc27). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…default reverted to 8, two stale layout comments, provenance on ledgers 67/68 and sec.2.26, lcpp provenance on the two bring-up lines, the reserve rule sharpened, the board re-mint duty as its own rule, the tests/REVIEW.md routing said as routing The 8 -> 16 grain change is dropped: its evidence never ran both variants interleaved in one process and the later sweeps on zen4 and M4 read the difference as noise - the ruled-accepted rows do not depend on it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

…, rowForm/uses_grid_rows/uses_sign_column/accLo-Hi/teamUs renames, the prefetch knob spelled out, PF_* constants, qhbyte, SOLO_BURN, ladder header - 25 findings applied, 4 declined consciously

…icon-independent gates; the companion own-seat and class-chain scenarios; the GEP scanner; the cache key folds vbmi and amx; a retune always writes the gemv companion; the prefetch pointer drops inbounds; the ladder reference follows NTOK New gates: test_kquant round-trips every grid entry through the vbmi code planes and alphabet and pm1_of against a scalar expansion (pure host data - CI-provable with no VBMI silicon), and proves moe_gpu_gather_stack_kq's own invariant (disk-order and grp planes gather byte-identical device planes, k3 and k6 - the one sub-block reader with no dot oracle). llvm_tune_manifest gains the companion own-entry scenario (write-own -> KV 1 while the main keeps k2); llvm_tune_profiles asserts the box's class heads its ladder; llvm_gep_inbounds.das machine-enforces the GEP rule; test_dispatch_shaping pins the shipped grain default; cpu_supports gains the avx512vbmi implication row. jit_env_salt folds avx512vbmi and amx (tune_cpu_class selects grid decode forms, so both are emission inputs the cache key missed). The seat write is now unconditional - upsert-only merges could keep a stale gemv seat a re-race no longer wants. Emitter pin recomputed (0x6a0cebf74a395071). followup_general 69: a class profile can carry winners a member cannot run (the Cascade Lake case). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

… its module_path require; the grain-default pin moves into its own [test]

…ed a relative forward-slash exe, which is exactly what the board rig spawned The rig's parity pregate "flake" was a spawn that never happened: with a NULL application name, CreateProcess refuses "modules/.../dasllama-bench.exe" while "./x.exe", "../x.exe", absolute and backslash spellings all resolve. Every caller of popen_argv, spawn_argv and popen_argv_pipe now gets argv[0] respelled before the command line is built; the caller-side backslash workaround leaves the fio doc and the language skill; tests/fio/popen_argv.das spawns the offending shape; run_and_stream logs a failed spawn instead of pointing at output that never came. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

…fixed rig - pp 1.18x, tg 1.04x of the clean reference, the ratio the 08-04 row held The row rides the adopted x86-avx2 class profile (snapshot zen2.tune.3fe069fc370e); both engines read about 5% under the 08-04 sweep on this box and the ratio held. The plan ledger carries the spawn defect and the row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

…, the fio doc claims only what holds off Windows, the test names its launch dir das_version's probe-verified normalize and the two dastest twins were the workaround the respelling retires; "either slash style on every platform" over-claimed for POSIX, where a backslash is not a separator; the failed-spawn log takes the file's user-facing prefix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

…the salt's length escape rides its complexity one, and the zen2 12B row backs out until Parsec is stopped Lint: the five move-assigns into a fresh array become grid_words(kq) with move returns, the alphabet is const, the +-1 ternary casts once; jit_env_salt is one flat fold per emission input at 81 lines, so the STYLE038 escape joins the STYLE037 one with the same reason. The records gate rejects a run stamped with the parsecd daemon, which runs as a service on zen2, so the re-measured row (pp 1.18x, tg 1.04x) stays in the plan ledger with its instrument and the store keeps the 08-04 row until a rig run with Parsec stopped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

Copilot

🔵 Needs a closer look

It combines wide-ranging low-level kernel/tuning/ISA gating changes across multiple platforms, so it needs final human validation beyond automated review.

Review details

Files reviewed: 54/54 changed files
Comments generated: 3
Review effort level: Lite

…n of the banned name next to the wrapper is not a call site Copilot's point on the rebased tip: the substring scan would fail the gate on the very comment the REVIEW rule invites. The scanner now drops a trailing // comment from each line and looks for LLVMSetIsInBounds( only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

Copilot

🔵 Needs a closer look

It touches performance-critical kernel emitters/layouts plus cross-platform tuning/JIT/spawn plumbing across many subsystems, requiring careful human validation beyond what can be safely automated here.

Review details

Files reviewed: 54/54 changed files
Comments generated: 0 new
Review effort level: Lite

…nt: the registry test the linux lane runs was red since the lab landed The two get_env_variable("DAS_TUNE_MODE") reads become g_env_tune.tune_mode through llvm/daslib/llvm_env - the knob's one declared read site, as modules/dasLLAMA/REVIEW.md and tests/test_env_registry.das require. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014sNMfrttRvpmQypECWoV3x

Copilot

🔵 Needs a closer look

modules/dasLLAMA/harness/kernel_ladder.sh has confirmed macOS mktemp portability and argument-splitting issues that can break the ladder harness on intended hosts.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

modules/dasLLAMA/harness/kernel_ladder.sh:41

mktemp -d is not portable to macOS/BSD (it requires a template), but this harness is explicitly used on the M4 Pro path. This will fail early on macOS and prevent ladder generation.
modules/dasLLAMA/harness/kernel_ladder.sh:63
KL_MODULE_CACHE paths containing spaces will be split because MC_ARGS is built as a single string and expanded unquoted. This breaks -module-cache usage on such paths and can make the bench fail to start.

Files reviewed: 54/54 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot AI lite review requested due to automatic review settings September 2, 2026 04:06

Copilot started reviewing on behalf of borisbat September 2, 2026 04:07 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread modules/dasLLAMA/performance/defaults/arm-i8mm.tune-defaults.json

Comment thread modules/dasLLAMA/performance/defaults/x86-amx.tune-defaults.json

Comment thread modules/dasLLAMA/performance/defaults/x86-vnni512.tune-defaults.json

borisbat and others added 27 commits September 1, 2026 21:39

kq_kernel_bench: =name selects one registry row exactly

882c2e2

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the M1 ladder and the post-k3 zen2 rows in the plan

e45ac5c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

the k-quant memo through the Markdown ASCII gate

e0418cf

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

followup_general 63: the rollback-and-ban sampler for reasoning trace…

da989ab

…s; HOW_TO_GET_SIDECAR.md sections 1-5 as run on the zen4 box Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the ARM grid memo, its fix on the queue, and the globa…

c26e70a

…ls emitters' comments say what they emit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the zen4 ladder - 65/65 TEST, VNNI seats pay on the do…

ec2357a

…t-bound formats, the grid gather is the scalar bound on both ISAs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

the ARM memo through the Markdown ASCII gate

78c193c

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

HOW_TO_GET_SIDECAR.md: the mint, the provenance read, the export and …

4ccd7ea

…the compare, as run on the zen4 box Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

the Qwen 3.8 memo through the Markdown ASCII gate

1c0e29e

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the zen4 model-level stamp and the repack caveat on th…

17c44b2

…e ladder's k-quant rows; the how-to's box log Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

HOW_TO_GET_SIDECAR.md: the new-class walk as run on the Intel box, an…

612d7ac

…d the three box lessons (alignment, denormals, the AMX grant) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the Intel ladder (Granite Rapids, x86-amx) with its tw…

d99bce9

…o bench artifacts fixed and the open items Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

kernel parity: the zen2 ladder re-run with the fixed bench

cd2f427

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0136vDWNJ2GZFzxToEQxWj9i

borisbat and others added 16 commits September 1, 2026 21:39

tune profile arm-i8mm: re-minted on the rebased branch (M4 Pro; 49 en…

b94f2ef

…tries, TEST 103 ok); the M4 v2 table

followup_general 68: iq2xs on arm-i8mm regressed with the shared row-…

44cbfb2

…group decode (0.90/0.95; the ARM panel/row choice is the fix)

followup_general 68 corrected: M4 iq2xs 0.90/0.95 is the format's tru…

a467ee5

…e ARM standing - the old 1.46 does not reproduce with its own sources; the row form beats the panel on ARM everywhere

style-hygiene round: dead gather names, family docs onto their owners…

7b951e3

…, rowForm/uses_grid_rows/uses_sign_column/accLo-Hi/teamUs renames, the prefetch knob spelled out, PF_* constants, qhbyte, SOLO_BURN, ladder header - 25 findings applied, 4 declined consciously

test fixes from the first run: the GEP scanner self-excludes and gets…

a3cfd7d

… its module_path require; the grain-default pin moves into its own [test]

Copilot AI review requested due to automatic review settings September 2, 2026 04:50

borisbat force-pushed the bbatkin/kernel-parity branch from 07542cd to fb76369 Compare September 2, 2026 04:50

Copilot started reviewing on behalf of borisbat September 2, 2026 04:51 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread modules/dasLLVM/tests/llvm_tune_manifest.das

Comment thread tests/fio/popen_argv.das

Comment thread modules/dasLLVM/tests/llvm_gep_inbounds.das

Copilot AI review requested due to automatic review settings September 2, 2026 04:57

Copilot started reviewing on behalf of borisbat September 2, 2026 04:58 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 2, 2026 05:43

Copilot started reviewing on behalf of borisbat September 2, 2026 05:43 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

borisbat merged commit bfa5126 into master Sep 2, 2026
37 checks passed

borisbat deleted the bbatkin/kernel-parity branch September 2, 2026 07:50

Navigation Menu

Uh oh!

CPU kernel parity: decode and tile kernels at llama.cpp speed or better on zen2, zen4, Intel Granite Rapids and M4 Pro - #3924

CPU kernel parity: decode and tile kernels at llama.cpp speed or better on zen2, zen4, Intel Granite Rapids and M4 Pro#3924
borisbat merged 123 commits into
masterfrom
bbatkin/kernel-parity

borisbat commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

borisbat commented Sep 2, 2026

Validation

Claims - stated, not tested

Not done

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Suppressed comments (2)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants