Proposed gguf support

[Atomic-Germ]

As in #14 & #33 and also #9 tangentially

This isn't complete, but it's getting there

[vegah]

Haha, I had a branch almost ready, but let's keep it one branch :)

[vegah]

Added some comments just based on a Claude review (which again is based on my branch)

[Atomic-Germ]

I'm not sure what it will take to get the simplified flow working on windows also, but it probably already does. This wasn't really a deliberate out-of-scope addition, but I really did need it in order to work on the rest along the way:

• Repo-root workflow ( not src) with cmake --preset linux-default → cmake --build --preset linux-default → cmake --install build.
• Full-distribution note: the default Linux preset also compiles the open NPU kernel xclbins, requiring XRT (installed) with ironvenv and third_party/mlir-aie+third_party/Peano auto-created/cloned by the build.
• Presets: linux-default (full), linux-debug (engine-only, FLM_BUILD_KERNELS=OFF), linux-portable (bundled libs), windows-default.
• Kernel subsetting: -DFLM_KERNEL_SPECS=….
• Tests: ctest --preset linux-default (the flm list smoke test).
• Packaging: cpack --preset linux-package-tgz/-deb/-rpm (run from repo root).
• One-shot: cmake --workflow --preset linux-default.
• Bumped the CMake prerequisite to 3.25 (needed for the test/package/workflow presets).

I've also merged main back in since a lot happened since.

[Cyronius]

it's all good. I'll try windows soon and get a PR going if it's no bueno.

[vegah]

I am working on fixing the conflicts...

[vegah]

(Or, Claude is!)

[vegah]

Merged main into this branch and pushed the resolution (f3b9cda) — it was conflicting in six files.

main had moved under it four times: the FLM→OFLM rename and the preset build (#47), the GEMM-block prefill route (#39), the VLM / M-RoPE path (#45), and a local lm_head kernel TU. Each collided with the WeightFile seam.

The rule throughout: take main's side, then route it through the weights view. w_ is man_ for a .q4nx and the GGUF-derived manifest for a GGUF, so main's new code reading geometry, tensor names or the tail off man_ would read the wrong manifest the moment the model is a GGUF — silently, since both are valid manifests. 18 such sites now go through w_, and the GEMM-block embed moved from Q4nxFile::bf16_row to the WeightFile::embed_row virtual (which is what dequantizes a GGUF row).

Two that weren't mechanical:

• core.cpp keeps this branch's GGUF config derivation (a GGUF needs no config.json) beside main's image_token_id / mrope_section parsing, and its verbose embed check beside main's image-embedding path.
• lm_head_q4.py keeps main's locally generated kernel TU — designs/layer_x's copy is per-spec and gets removed again, which is what broke Gemma's lm_head build on 2026-09-08 — and gives it this branch's GEMV_Q4_PREFIX wrapping, so the f32-scale build still resolves gemv_q4s32_gy. The prefix is a compile flag, so one generated source serves both builds.

pools.cpp keeps the source-agnostic raw(const WeightFile&); main's raw(const Q4nxFile&) is subsumed. README.md and src/CMakeLists.txt take main's side wholesale — main had already absorbed this branch's build-from-source rewrite under the new name — except open_qwen36/gguf_file.cpp, re-added to the engine sources. build.cmd gets it too and now runs gguf_pack_test.

Verified on Windows/MSVC against the system XRT: manifest_test, pools_test (Q4_K transcode and the mixed pools included) and gguf_pack_test all PASS, and open_qwen36_cli links. Not run: anything on hardware, and the Linux kernel export.

— Claude, on behalf of @vegah

[vegah]

@Atomic-Germ Merge if it is okay, if not - just remove the latest commit.

[vegah]

Minor architectural worry: this makes the packing interpreter format-aware, and pools.cpp is the one file whose
byte-exactness is the safety property — mirrored in NumPy by recipes/pack.py and frozen against legacy_pools.py by
test_pack_plan.py. A new std_perm_gguf op plus a downcast means both interpreters now have to be kept in lockstep
across two layouts.

The alternative is to have the source hand back bytes already in the chunk layout, so std_perm stays one op and
pools.cpp never learns what a GGUF is. That's what I did on my branch and it kept pools.cpp to +21/−15 with no
arithmetic touched — happy to show it if useful, though it presumes the single-layout answer in comment 2.

[Atomic-Germ]

I'm definitely interested in that, at the very least we need to be looking at alternative methods of accomplishing this just in case there's an altogether better way we're missing.

[vegah]

Happy to. It is branch gguf-runtime off 0fe7922 (PR #21), four commits, never pushed — say the word and I will put it up.

The seam is one header, weight_source.hpp. Three methods — raw(name), embed_row(...), drop_pages() — and one contract:

a source hands back, per tensor, EXACTLY the bytes model.q4nx would have held for it, in the file's raster order (chunk f covers rows 32*(f/ncol), cols 256*(f%ncol)).

Q4nxFile satisfies that by pointing into its mapping; a GGUF source satisfies it by building the bytes. raw() returns a value rather than a pointer — TensorBytes, which is borrowed / solely owned / shared with a cache and looks the same to the caller — because bytes that had to be built need an owner and only the caller knows how long it needs them.

pools.cpp is then +21/−15, and all of it is two mechanical shapes. No op is added, std_perm stays one op, and no arithmetic is touched:

-const uint8_t* raw(const Q4nxFile& m, const std::string& name, size_t need, size_t* got = nullptr) {
-    size_t n = 0;
-    const uint8_t* p = m.raw(name, &n);
+TensorBytes raw(const WeightSource& m, const std::string& name, size_t need, size_t* got = nullptr) {
+    TensorBytes b = m.raw(name);
+    const size_t n = b.size();
     if (n < need) fail(name + " is " + std::to_string(n) + " B, the plan needs " + std::to_string(need));
     if (got) *got = n;
-    return p;
+    return b;
 }

-void apply(const PackOp& op, const Q4nxFile& m, int layer, uint8_t* dst, size_t dst_bytes, size_t ch) {
+void apply(const PackOp& op, const WeightSource& m, int layer, uint8_t* dst, size_t dst_bytes, size_t ch) {
     if (op.op == "std_perm") {
         ...
-        const uint8_t* src = raw(m, name, (op.chunk0 + op.nch) * ch) + op.chunk0 * ch;
+        const TensorBytes srcb = raw(m, name, (op.chunk0 + op.nch) * ch);
+        const uint8_t* src = srcb.data() + op.chunk0 * ch;
         auto perm = std_perm(op.nch, op.in_dim);
         for (size_t c = 0; c < op.nch; ++c) std::memcpy(dst + op.dst + c * ch, src + perm[c] * ch, ch);

The other five ops (expert_stripes, expert_down, put, lmhead_q8, conv_transpose) are each the same two lines, and pack_pool / pack_consts / pack_lmhead are signature-only. pools.hpp swaps its q4nx_file.hpp include for weight_source.hpp.

The format work does not disappear, it moves — gguf_repack.cpp (281 lines: the block formats to raster chunks, Q4_K included), gguf_source.cpp (214: the WeightSource), gguf_file.cpp (364: the container). That is more total lines than this PR's. The trade is only about which file they are in: none of them is the one recipes/pack.py mirrors and test_pack_plan.py freezes.

Two things I should be straight about:

• It is based on 0fe7922 and predates the OFLM rename (Cmake full build system #47), the GEMM-block prefill route (Open engine: batched prefill as a whole-array GEMM (1.95x TTFT) #39) and the VLM path (Open kernels: fast attention on every family, and images through the open engine (#16) #45), so it needs a rebase before it reads as a diff against today's main. I have not done that rebase.
• Its byte-exactness evidence is specs/open-engine/tests/test_gguf_pack.py. The test_law_* half is self-contained — random blocks, no model, no network — and pins the NumPy statement of the law against both open_kernels/q4_1_pack.py and gguf_repack.cpp, so a change to any one is caught by the other two. The test_model_* half needs a real GGUF and skips without one; it is not currently run anywhere, and I have not run it against the current tree.

And to be explicit about the dependency I flagged: this shape presumes comment 2's single-layout answer. If the f32-scale plane stays, the source has two output layouts to produce and the seam has to carry which one — still outside pools.cpp, but no longer free.

— Claude, on behalf of @vegah

[Atomic-Germ]

After way too much thought; you're definitely right.

[Cyronius]

I was trying to make sure my code was going to agree with this, and opus uncovered something:

I think the Q4_0 path is off by a constant.

A GGUF Q4_0 block stores nibbles 0-15 that stand for -8 to 7, so llama.cpp's
dequant is (q - 8) * d. The pack here uses q * d: the min plane is zeroed
for Q4_0 (pools.cpp, mf = has_min ? fp16_to_f32(um) : 0.f), embed_row
does d * (p[2 + j] & 0xF), and gguf_pool.py fills the min plane with
zeros. Every weight comes out 8*d too high - a block that should straddle zero
ends up entirely on one side of it.

The kernel already computes q * d + m, so writing m = -8 * d into the min
plane fixes it with no kernel change and no loss: -8 * d only moves an
exponent, exact in f32 and bf16 alike.

Why the tests don't catch it: gguf_pool's reference dequant uses the same
q * d + 0, so the packer and the checker agree with each other, and
gguf_pack_test only builds Q4_1 and Q8_0 tensors - there's no Q4_0 fixture at
all. We hit the same thing on the .q4nx side last week and it was invisible for
exactly that reason; what finally caught it was the model answering with noise
while every numerical check passed.

One other thing worth flagging while you're in here. "Q4_0" now means two
different byte layouts in this tree. GGUF's nibbles are offset by 8, so 0 means
-8. The .q4nx containers q4nx-build produces hold the same signed values as
two's complement, where 0 means 0 and 8 means -8 - unpack_q4_0 subtracts the
8, then model_converter packs the signed int8 straight back with & 0x0F.
Same numbers, different labels, and reading one as the other shifts everything
by 8*d. Probably worth separate names before the two paths meet.