Published *-Q2_0.gguf files (all four Bonsai rungs) declare type 42 with the group-128 layout and will not load on current prism HEAD

[ralphbutler]

Summary

Every *-Q2_0.gguf file currently published on the Hugging Face prism-ml repos declares ggml
tensor type 42 while storing the group-128 / 34-byte block layout. Current prism HEAD
reads type 42 as the official group-64 / 18-byte Q2_0, so it refuses to load them.

The loader already detects this precisely and prints an excellent diagnostic, so this is not a
report that something is silently broken — it is that the published reference models cannot be
loaded by a fresh build of the reference implementation, which is a confusing first experience.

Affected files

All four rungs, verified by reading each file's header over HTTP range requests and measuring the
block size from the gaps between consecutive tensor offsets (no full downloads):

repo	file	declared type	measured layout
prism-ml/Ternary-Bonsai-1.7B-gguf	Ternary-Bonsai-1.7B-Q2_0.gguf	42	group 128, 34 B/block
prism-ml/Ternary-Bonsai-4B-gguf	Ternary-Bonsai-4B-Q2_0.gguf	42	group 128, 34 B/block
prism-ml/Ternary-Bonsai-8B-gguf	Ternary-Bonsai-8B-Q2_0.gguf	42	group 128, 34 B/block
prism-ml/Ternary-Bonsai-27B-gguf	Ternary-Bonsai-27B-Q2_0.gguf	42	group 128, 34 B/block

The *-PQ2_0.gguf files load fine — those use type 142, which matches QK_PQ2_0 = 128.

Reproducer

git clone --depth 1 -b prism https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp && cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build -j --target llama-cli
./build/bin/llama-cli -m Ternary-Bonsai-1.7B-Q2_0.gguf -p "The capital of France is" -n 24 -ngl 0

gguf_init_from_reader: tensor 'blk.0.attn_k.weight' has offset 82516128, expected 87369536
gguf_init_from_reader: this file matches the legacy Prism Q2_0 layout (group size 128 stored as
  ggml type id 42), but this build reads Q2_0 as the official group-64 format
gguf_init_from_reader: you are probably using the wrong GGUF: use the PQ2_0 version of this model
  (ggml type id 142) or download the group-64 Q2_0 file
llama_model_load: error loading model: failed to load model

The 6% size difference is what makes the two layouts distinguishable: (n/128)*34 vs (n/64)*18.

Root cause

From ggml/src/ggml-common.h at HEAD:

#define QK2_0 64          // block_q2_0     = 2 + 64/4  = 18 bytes
#define QK_PQ2_0 128      // block_pq2_0    = 2 + 128/4 = 34 bytes

and ggml/include/ggml.h:

GGML_TYPE_Q2_0    = 42,
// Prism-private Q2_0 at group size 128 (upstream Q2_0 is group 64) ...
GGML_TYPE_PQ2_0 = 142,

So type 42 was renumbered to the upstream group-64 meaning after these model files were
published, and the group-128 layout moved to 142. The files were not re-tagged.

Possible resolutions

Any one of these would fix the first-run experience; listed in the order that seems least
disruptive:

1. Re-tag and re-upload the four *-Q2_0.gguf files with type 142, or simply retire them in
   favour of the already-correct *-PQ2_0.gguf files.
2. Note it in the model cards. Each card currently recommends the Q2_0 file — e.g. the 1.7B
   card lists Ternary-Bonsai-1.7B-Q2_0.gguf as "recommended" — which is the file that will
   not load. Pointing readers at PQ2_0 would be enough.
3. Accept the legacy layout on read. The loader already identifies it with certainty from the
   offset arithmetic, so it could remap type 42 to PQ2_0 for files that match, rather than
   erroring.

Happy to send a PR for (2) or (3) if either is welcome.

Environment

• PrismML-Eng/llama.cpp prism @ d8d96cf
• macOS 15 (Darwin 25.6.0), Apple M4 Max, 128 GB
• cmake 4.x, Release build, Metal + BLAS enabled

Aside

Found while writing an independent ternary inference engine that reads these files directly. That
reader derives the block layout from the tensor offsets instead of trusting the type id, so it
loads both variants — which is how the discrepancy surfaced. Both files decode to bit-identical
weights, so this really is only a labelling problem.

[khosravipasha]

Agent review: this reply was drafted and posted by the PrismML maintainers' coding agent from the maintainer account; the decisions in it are the team's.

Thanks for the careful report; the file-level analysis is exactly right, and the loader message you hit is doing its job.

What the files are. *-Q2_0.gguf on the older repos is the legacy group-128 pack from the prism-v5 line, which stored it under id 42. Current prism follows mainline, where id 42 is the official group-64 Q2_0, so the same id cannot mean both layouts and the loader refuses rather than guessing. Every repo already carries the two files a current build wants:

• *-PQ2_0.gguf (id 142): this fork, on Metal, CUDA, HIP and CPU. Same bytes as the legacy file, correct id.
• *-Q2_0_g64.gguf (27B: *-Q2_g64.gguf, id 42): mainline llama.cpp or any backend.

The README's top note lays this out.

What we will and won't change.

• We won't rename, re-tag or remove the published *-Q2_0.gguf files (option 1). Those names are referenced from a lot of places by now, and changing them is a breaking change for everyone still on prism-v5 or with the paths baked into scripts. They stay as the prism-v5 artifacts.
• We won't remap id 42 on read (option 3). A silent remap would make the same file load on this fork and fail on mainline, which is a worse first-run experience than a clear error pointing at the right file.
• We will fix the model cards (option 2): the 1.7B, 4B and 8B cards still recommend the legacy file and use it in every example, which is what sent you to the wrong file. They will be updated to match the 27B card: PQ2_0 for the fork, Q2_0_g64 for mainline, the legacy file marked as such. The cards live on Hugging Face, so we will handle that side; thanks for the offer.
• Future model releases ship both a proper group-64 Q2_0 and a PQ2_0 from day one, under those names, so this split does not recur.

Nice to hear the independent reader decodes both variants to identical weights; that matches what we see.

[khosravipasha]

Agent review: drafted and posted by the PrismML maintainers' coding agent from the maintainer account; the decisions are the team's.

A bit of history on why the two layouts exist, since it explains the naming:

• The original Prism Q2_0 was a group-128 format: one FP16 scale per 128 ternary weights, 2.125 bits/weight. When the format went to mainline llama.cpp, the maintainers there preferred group 64 to match the block size of the other 2-bit types, so mainline's Q2_0 (id 42) is group 64 at 2.25 bits/weight.
• We adopted mainline's definition for id 42 so that a Q2_0 file means the same thing everywhere, and kept the group-128 layout alive under its own id (142) and name, PQ2_0, for people who want the smaller footprint: about 6% less than group 64, same weights, and it is the faster of the two on this fork's kernels. That is the split you see in every repo: PQ2_0 for the fork, Q2_0_g64 for mainline.
• The legacy *-Q2_0.gguf files predate the renumbering and are what the older releases (prism-v5) read; that is the file the loader now refuses.

If footprint is what you care about, there is also PTQ1_0 (id 143): the same ternary weights packed in base 3 at 1.75 bits/weight, 17% smaller than PQ2_0, lossless with respect to it. It is in current prism on CPU, Metal and CUDA and is what newer model repos ship alongside PQ2_0. Decode speed is at or slightly above PQ2_0; prefill is behind it (roughly 0.6 to 0.9x depending on backend and size) because the base-3 unpack costs more than a shift. It is still in active development, so expect the prefill gap to move; the format itself is fixed.

[ralphbutler]

Thank you very much for the quick and thorough reply. You set me straight on two things. My suggestion to remap id 42 on read was the wrong instinct; I was thinking about the first-run experience and missed that a silent remap would load here and fail on mainline. A clear error pointing at the right file is the better failure. And your history explains the naming; I'd assumed the id collision was an oversight rather than a deliberate move to match mainline's definition of Q2_0. I appreciate the cards are getting updated; that was the part that actually cost me time. Thanks also for the PTQ1_0 pointer as I didn't know it existed. On Sep 8, 2026, at 8:33 PM, Pasha Khosravi ***@***.***> wrote: [https://avatars.githubusercontent.com/u/2780093?s=20&v=4]khosravipasha left a comment (PrismML-Eng/llama.cpp#167)<#167 (comment)> Agent review: drafted and posted by the PrismML maintainers' coding agent from the maintainer account; the decisions are the team's. A bit of history on why the two layouts exist, since it explains the naming: * The original Prism Q2_0 was a group-128 format: one FP16 scale per 128 ternary weights, 2.125 bits/weight. When the format went to mainline llama.cpp, the maintainers there preferred group 64 to match the block size of the other 2-bit types, so mainline's Q2_0 (id 42) is group 64 at 2.25 bits/weight. * We adopted mainline's definition for id 42 so that a Q2_0 file means the same thing everywhere, and kept the group-128 layout alive under its own id (142) and name, PQ2_0, for people who want the smaller footprint: about 6% less than group 64, same weights, and it is the faster of the two on this fork's kernels. That is the split you see in every repo: PQ2_0 for the fork, Q2_0_g64 for mainline. * The legacy *-Q2_0.gguf files predate the renumbering and are what the older releases (prism-v5) read; that is the file the loader now refuses. If footprint is what you care about, there is also PTQ1_0 (id 143): the same ternary weights packed in base 3 at 1.75 bits/weight, 17% smaller than PQ2_0, lossless with respect to it. It is in current prism on CPU, Metal and CUDA and is what newer model repos ship alongside PQ2_0. Decode speed is at or slightly above PQ2_0; prefill is behind it (roughly 0.6 to 0.9x depending on backend and size) because the base-3 unpack costs more than a shift. It is still in active development, so expect the prefill gap to move; the format itself is fixed. — Reply to this email directly, view it on GitHub<#167?email_source=notifications&email_token=BCDE3O3D274CZGHFAYXK3W35OCXQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJZGQ2DCMJTGA32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5594411307>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BCDE3O62ZETOQR4NZH55B5D5OCXQLAVCNFSNUABGKJSXA33TNF2G64TZHMYTCNRWHE3DKOJWGM5US43TOVSTWNJTHEZTKOBXGM3DBILWAI>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/BCDE3O6NTF3Q2TNODJ4D6TL5OCXQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJZGQ2DCMJTGA32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/BCDE3O7P6DXKOW23TMZIVWD5OCXQLA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJZGQ2DCMJTGA32M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[khosravipasha]

Thanks, yeah we added PTQ1_0 very recently so have not fully "released" it yet but it should be stable to use now. We will upload
Q2_0, PQ2_0, PTQ1_0 gguf formats for future ternary releases so people can choose based on their enviroment and speed vs memory tradeoff to see which ones to use.