feat(phi4): run Phi-4-mini-instruct Q8_0 GGUF on AIE4 via ryzenai-corelib

[chizamd]

Summary

Adds one catalog model, phi4-mini-it-aie4:4b, that FastFlowLM pulls as a Q8_0 GGUF and runs on AIE4 through AMD's ryzenai_corelib.dll. The weights are read straight from the GGUF — this branch does not generate, ship, or consume an ONNX model, an initializer manifest, or any converted weight artifact.

The existing Phi-4 NPU2/Q4NX model and its backend are untouched. The new path compiles only under -DFLM_ENABLE_CORELIB_AIE4=ON (Windows + XRT); with the option OFF, common/corelib/*.cpp is filtered out of the flm source glob and the binary contains no reference to it.

52 files, +7,738 / −385 vs 2d6c4838.

Draft because it is an alternative to #706 rather than an increment on it, and because packaging and CI are absent.

---

Design

GGUF is the only source of truth, read zero-copy

The file is memory-mapped and every tensor is handed out as a span — nothing is copied, converted, or cached to disk. RequireQ8/RequireF32 take a name and an expected shape, so all 161 tensors plus the rope factors are resolved and shape-checked before the first device object is created. Head counts, rope base and the short-factor table come from the file's own metadata; the architecture itself (32 layers, 3072 hidden, 24/8 heads, 200064 vocab) is a fixed C++ constant block that the GGUF metadata, config.json and tokenizer_config.json are all cross-checked against. A mismatched package fails at load with an exact diagnostic, never mid-generation.

Weights go straight from mapped bytes into corelib objects

Each projection calls matmul_weights_create_gguf_requantized on the Q8_0 bytes in place — no intermediate blob, no packed-weight file. Q8_0 is lossily requantized to group 64 during packing. The MLP is one fused SSMLP object per layer that absorbs the FFN norm scale, epsilon, and the next layer's attention norm scale, so the only host RMSNorm in the model is layer 0's. The LM head is built from token_embd.weight (tied embeddings), requantized the same way.

The packer is given a threads hint of 8 — corelib treats 0 as ONE deliberately, and this requantizing path is compute-bound and scales with the hint. The hint is per-create and the creates themselves are serialized. corelib's header records that loading a model with 8 concurrent creates on this entry point failed 2 of 10 with all-zero output, against 0 of 10 serialized, with attribution open; that configuration is deliberately not used, and a test pins one in-flight create.

One code path for prefill and decode, and it never allocates

Every device tensor — hidden/residual/skip, Q, K, attention, the 32 K and V caches, rope tables — is created once at load, sized to the maximum padded extents the shape plan interrogates out of corelib's own padding helpers. The plan is queried only at corelib's nine real execution buckets (1, 64, 128, 256, 512, 1024, 2048, 3072, 4096) and every logical row maps to its next bucket, so nothing re-enters the helpers on the hot path and nothing allocates per token.

Per layer the device does everything except the embedding gather: Q/K matmul, V matmul written directly into a tensor window of the V cache at the current position (no host-side KV scatter), flat_mha against the caches with the rope tables, O projection, fused SSMLP with a residual/skip ping-pong. The result is two synchronizes per token — one after all 32 layers, one after the head — not one per layer.

Explicit routing, no fallback, terminal failure

ResolveBackend() dispatches on model_info.details.execution_backend (corelib_aie4_gguf). Nothing is inferred from hardware, filename, or quantization level, and there is no fallback to CPU or NPU2 — if corelib is missing or the wrong version the model fails to load with a diagnostic. flm.exe never links ryzenai_corelib.lib; the DLL is resolved by absolute path (FLM_AIE4_CORELIB_PATH, else <exe_dir>/aie4/ryzenai_corelib.dll) and get_version is called first, so a mismatched runtime reports a version error rather than a missing symbol. An exact 0.3.0 match is required while corelib is pre-1.0.

Once work has reached the device, any throw drains with a synchronize and marks the engine poisoned; every later entry point refuses rather than continuing on indeterminate state.

Pinned multi-source pull

File	Source	Revision
Phi-4-mini-instruct.Q8_0.gguf	unsloth/Phi-4-mini-instruct-GGUF	78eb92a4
tokenizer.json, tokenizer_config.json, config.json	microsoft/Phi-4-mini-instruct	cfbefacb

model_list.json gains a file_sources map for per-file source/revision overrides. Existing catalog entries are not migrated — entries without file_sources keep their current behaviour, and test_model_downloader.cpp pins that compatibility. All four files are SHA-256 verified against model_info.json at pull time and on flm check. The run and serve paths check presence and version only; they do not re-hash a 4 GB file on every launch.

Capacity

The usable window is 4095 tokens (prompt + output), so the largest admissible prompt is 4094; over-capacity requests are rejected with HTTP 400 before submission. The 4096 ceiling is a correctness boundary, not a buffer size: it is Phi-4-mini's rope.scaling.original_context_length, LongRoPE selects factors by sequence length, and only the short branch is derived here — enforced at load, which fails unless the GGUF reports exactly 4096. The further step down to 4095 is this frontend's own conservatism so an admitted request can always finish.

Observability

Model loaded in N s is reported on both the CLI and server load paths. FLM_AIE4_PROFILE_LOAD=1 additionally breaks the AIE4 load into shape planning, GGUF resolution, host preparation, weight requantization and device allocation. Decode timing is recorded on the AIE4 path, so /status, /verbose and the Ollama-compatible eval_duration report real figures on this backend.

---

Tests

src/test/phi4_corelib_aie4/ is a standalone CMake project with 9 ctest targets built on a fake corelib implementing the 0.3.0 C ABI in-process, plus a deterministic GGUF v3 fixture builder that synthesizes valid and deliberately corrupt packages. Coverage: ABI/version/path/lifetime, parsing and corruption, host ops, shape plan, engine load and dispatch sequencing with object-leak and serialization checks, frontend routing and request lifecycle, a compile-gate proving the OFF build carries no corelib, and the downloader including legacy-entry compatibility.

9/9 pass on the AIE4 host with test_real_corelib executing against the real DLL, not skipped.

src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 drives the hardware matrix end to end and writes a machine-readable record.

---

Verified on hardware

Machine XCOMEDUSAD-43, AMD XDNA NPU, Windows 11 build 26100, corelib 3c35aebd ABI 0.3.0. Acceptance record passed: true, 0 failures:

• pull / check — four pinned files SHA-256 verified; the model directory contains exactly those four, with no manifest/onnx/converted/packed artifact.
• CLI — 10/10 fresh-process load-and-generate cycles exit 0; Backend: corelib_aie4_gguf and the loaded DLL path reported in every one.
• REST — /api/chat and /v1/chat/completions both 200, streaming and non-streaming.
• Cancellation — in-flight stream cancelled cleanly, next request 200 on the same server.
• Capacity — 4095 admitted; 4096 rejected with HTTP 400 before submission.
• No CPU or NPU2 fallback in the server log at any point.

Measured

Descriptive, no threshold claimed.

Metric	Value
Model load to serving	5.1 / 5.3 s (of which weight requantization 2.5 / 3.0 s)
Cold TTFT	4.21 s — first prompt in a fresh process, includes one-time kernel and ELF setup
Warm TTFT	65.0 ms
Decode, REST	21.3 tok/s
Decode, warm CLI session	35.8 tok/s

Ten fresh-process CLI cycles report 3.70–20.26 tok/s decode; those are not throughput, because each pays the one-time setup inside its own eight-token window. The 1.7× spread between the REST and warm-CLI decode figures is unexplained; treat single-run differences below roughly 2× as noise.

---

Known gaps and defects

1. One unexplained degenerate reply. A /api/chat answer to What is 2+2? came back as a truncated markdown image URL with done_reason: length. The same prompt answered correctly on every other occasion, so it reads as sampling nondeterminism rather than a routing fault — but it is a single observation and it is not understood.
2. The acceptance matrix was last run in full at 87721089. Later commits changed startup verification and the packer thread hint. After those, the hardware suite (9/9 with the real DLL) and output correctness were re-verified and load was re-measured, but the full matrix has not been re-run at the tip.
3. calculate_file_sha256 is ~8× slower than the work requires — ~28 s over 4 GB where Get-FileHash on the same machine takes 3.67 s, because it uses a portable pure-C++ SHA-256 with no hardware acceleration. This no longer sits on the startup path, but every flm pull and flm check still pays it, for every model. Not fixed here: it is shared pull code, unrelated to AIE4, and belongs in its own change.
4. No packaging. MSI, WiX and Inno are untouched; this produces an AIE4-enabled developer build and the runtime DLL is not staged or shipped.
5. No CI builds or runs any of this, and the release preset does not compile the AIE4 path.
6. Windows only, Q8_0 only — no Q4_0/Q4_K/Q6_K, no mixed quantization, no generic GGUF runtime, no other model family.
7. Usable context is 4095, against 128k for the model and 32k default for the existing tag.

Relationship to #706

#706 (feature/phi4-aie4-corelib) reaches the same backend through an ONNX manifest + packed weights pipeline, and carries packaging and tooling this branch does not. This is the independent GGUF-direct approach: far smaller surface, no manifest generator, no converted artifacts, no Python at runtime. They are alternatives, not increments; only one should land.