From df83bf6c2ff47dea0769cdd230cc1f476c079a2e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 01:47:13 -0700 Subject: [PATCH 01/37] docs: design Phi-4 GGUF AIE4 integration --- .../specs/2026-09-11-phi4-gguf-aie4-design.md | 371 ++++++++++++++++++ 1 file changed, 371 insertions(+) create mode 100644 docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md diff --git a/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md b/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md new file mode 100644 index 00000000..2cb3b9c5 --- /dev/null +++ b/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md @@ -0,0 +1,371 @@ +# Phi-4 Q8_0 GGUF on AIE4 Design + +## Summary + +Add one catalog model, `phi4-mini-it-aie4:4b`, that FastFlowLM can pull and run through `ryzenai-corelib` on AIE4. The implementation starts from FastFlowLM `main`, supports only the validated Phi-4 Mini Instruct Q8_0 GGUF, and uses corelib's explicit lossy Q8_0-to-group-64 requantization APIs. + +The model is read directly from GGUF. FastFlowLM will not generate or ship an ONNX initializer manifest, an ONNX model, or converted weight files. The Phi-4 architecture and tensor-name contract live in a model-specific C++ adapter, while shape, dtype, offset, and model metadata come from the GGUF file. + +This PR produces an AIE4-enabled developer build of the normal `flm.exe`. It does not package the AIE4 runtime in MSI, WiX, or Inno Setup. + +## Fixed inputs + +### Model + +- FLM tag: `phi4-mini-it-aie4:4b` +- GGUF repository: `unsloth/Phi-4-mini-instruct-GGUF` +- GGUF revision: `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` +- GGUF file: `Phi-4-mini-instruct.Q8_0.gguf` +- Supported quantization: GGML `Q8_0` only + +### Tokenizer and configuration + +- Repository: `microsoft/Phi-4-mini-instruct` +- Revision: `cfbefacb99257ffa30c83adab238a50856ac3083` +- Files: `tokenizer.json`, `tokenizer_config.json`, and `config.json` + +The tokenizer files come from a second repository because the selected Unsloth repository does not publish the files FastFlowLM's existing tokenizer frontend consumes. Model loading cross-checks the tokenizer/config contract against GGUF metadata rather than assuming the two fixed sources agree. + +### Corelib + +- Repository: `VitisAI/ryzenai-corelib` +- Commit: `3c35aebdefa3f0c2255668bab1be5648ece320f8` +- ABI version: `0.3.0` + +Because corelib remains pre-1.0, FastFlowLM requires an exact runtime version match: major, minor, and patch must all be `0.3.0`. + +## Goals + +1. `flm pull phi4-mini-it-aie4:4b` downloads the pinned GGUF and the pinned tokenizer/config files. +2. An AIE4-enabled `flm.exe` runs the model through corelib from the CLI and REST APIs. +3. The existing Phi-4 NPU2/Q4NX model continues to use its existing backend. +4. A build without AIE4 support, or an AIE4 build with no corelib DLL, still starts and runs non-AIE4 models. +5. The implementation validates the model and tokenizer contracts before creating device state. +6. No silent CPU or NPU2 fallback is possible for the AIE4 tag. +7. The completed implementation is exercised on real AIE4 hardware before the PR is considered complete. + +## Non-goals + +This PR does not add: + +- ONNX model loading for AIE4; +- a JSON tensor manifest or manifest generator; +- a generic GGUF runtime or arbitrary local GGUF support; +- Q4_0, Q4_K, Q6_K, or mixed-quantization support; +- another model family; +- a packed-weight disk cache; +- parallel Q8_0 weight creation; +- Python as a runtime dependency; +- automatic corelib/runtime download; +- MSI, WiX, or Inno Setup packaging; +- a new corelib API; +- CPU or NPU2 fallback for the AIE4 model. + +## Architecture + +```text +flm.exe + └── Phi4 frontend + ├── existing chat, tokenizer, sampling, and server integration + └── phi4_corelib_aie4 + ├── phi4_corelib_gguf GGUF v3 parsing and Phi-4 tensor mapping + ├── phi4_corelib_shape_plan corelib padding and buffer extents + ├── corelib_runtime DLL lifetime, version, and availability + └── corelib_api dynamically resolved C ABI + │ + └── ryzenai_corelib.dll → AIE4 +``` + +### Backend selection + +The new catalog entry sets `details.execution_backend` to `corelib_aie4_gguf`. `Phi4::load_model()` selects the new backend only for that explicit value. An absent backend field retains the current NPU2 behavior. An unknown value is an error. + +There is no automatic hardware or format detection and no fallback. This makes a request for the AIE4 tag observable and testable: it either runs through corelib or fails. + +### Component boundaries + +#### Phi-4 frontend + +`modeling_phi4.cpp` remains responsible for: + +- backend selection; +- tokenizer setup and chat-template application; +- sampling; +- request capacity checks; +- translating the existing `AutoModel` interface to the selected causal-LM engine. + +It does not parse GGUF or call individual corelib operators. + +#### Corelib API and runtime + +A small dynamic adapter resolves only the C ABI symbols used by this backend: + +- version, dependency self-test, device-context query, errors, and cleanup; +- object lifecycle; +- stream creation and synchronization; +- tensor creation, windows, reads, and writes; +- matmul padding, Q8_0 requantized weight creation, and dispatch; +- SSMLP padding, Q8_0 requantized weight creation, and dispatch; +- RMSNorm weight creation, padding, and dispatch; +- flat-MHA padding and dispatch. + +The adapter owns no model policy. It converts failed statuses into exceptions that retain the corelib status, call name, and thread-local detail message. RAII wrappers release every returned object. + +Corelib is not loaded during process startup. It is loaded when an AIE4 model is selected. Runtime lookup order is: + +1. the absolute DLL named by `FLM_AIE4_CORELIB_PATH`; +2. `/aie4/ryzenai_corelib.dll`. + +The loader does not search the current working directory. After loading, it resolves the version functions first, requires ABI `0.3.0`, resolves the remaining symbols, runs the dependency self-test, and verifies an AIE4 device context exists. + +#### GGUF package + +`Phi4GgufPackage` owns a read-only mapping of the single GGUF file and exposes validated, non-owning tensor views whose lifetime cannot exceed the mapping. It parses only the GGUF v3 facilities used by the pinned model: + +- little-endian header; +- metadata scalar, string, and array encodings; +- tensor names, dimensions, GGML types, and relative offsets; +- model alignment and tensor-data start. + +The parser performs checked arithmetic for every count, offset, alignment, and byte-length calculation. A truncated directory, duplicate tensor name, unsupported value type that cannot be skipped safely, out-of-range tensor, overlapping invalid range, or malformed string is a load error. + +The model-facing API is intentionally narrow: + +```cpp +TensorView RequireQ8(name, expected_shape); +FloatTensorView RequireF32(name, expected_shape); +ProjectionViews AttentionQkv(layer); +ProjectionViews GateUp(layer); +GgufPhi4Metadata Metadata(); +``` + +The adapter hardcodes Phi-4 Mini's expected tensor names and architecture. QKV and gate/up tensors are fused in this GGUF; the adapter splits each into row-aligned byte-range views without dequantizing or copying it. Q8_0 rows consist of complete 34-byte blocks for 32 weights, so every allowed split must fall on a complete-row boundary. + +#### Phi-4 AIE4 engine + +The execution engine selectively carries forward the hardware-validated structure from PR #706: + +- one corelib stream; +- padded tensors sized from corelib's helper APIs; +- fixed-size K/V caches; +- prefill and one-token decode; +- explicit synchronization at producer/consumer boundaries; +- host-side lazy embedding lookup; +- Phi-4 partial rotary tables; +- corelib matmul, fused SSMLP, standalone RMSNorm, and flat MHA dispatch; +- a maximum sequence length of 4096 and maximum usable decode window of 4095. + +The source path changes completely: no ONNX initializers and no manifest are accepted. + +For each quantized projection, the engine passes a raw Q8_0 block view to `ryzenai_corelib_*_weights_create_gguf_requantized` with group size 64. Weight objects are created serially. The corelib API documents an open, unattributed all-zero-output incident correlated with concurrent requantized creates; avoiding concurrency is the measured safe configuration and is required for this first implementation. + +GGUF stores norms as F32. The adapter converts only the required norm vectors and epsilon to BF16 at model load. Embedding rows are decoded lazily for requested token IDs instead of materializing the full 200064-by-3072 embedding. RoPE tables are derived once from the GGUF's Phi-3/Phi-4 rope metadata and uploaded as FP32. + +## Model contract validation + +Validation occurs before device weight creation wherever possible. The package must match all of these constraints: + +- the expected Phi-3/Phi-4 GGUF architecture identifier; +- 32 decoder layers; +- hidden size 3072; +- intermediate size 8192; +- 24 attention heads; +- 8 key/value heads; +- head size 128; +- vocabulary size 200064; +- partial rotary dimension 96; +- RMS epsilon `1e-5`; +- maximum sequence length 4096; +- `phi3.rope.dimension_count` equal to 96; +- finite, positive `phi3.rope.freq_base` and `phi3.rope.scaling.attn_factor`; +- `phi3.rope.scaling.original_context_length` equal to 4096; +- `rope_factors_short.weight`, when present, is F32 with exactly 48 elements; +- the long-rope branch is rejected because this backend supports only the original 4096-token window; +- every required projection present with the exact expected logical shape; +- every projection, tied embedding, and LM head source is Q8_0; +- every required norm present in the supported floating type; +- `output.weight` is absent and `token_embd.weight` is used for both embedding and LM head, as in the pinned model; +- tokenizer vocabulary size agrees with GGUF; +- EOS IDs include 200020 and 199999; +- BOS behavior agrees; +- the chat template contains the required Phi-4 user, end, and assistant markers. + +The error names the model field or tensor, its actual value, and the expected value. The loader does not repair, reinterpret, or silently accept a mismatch. + +## Pull and catalog design + +The existing catalog format assumes one base repository per model. Retain the existing string-only `files` array and add an optional `file_sources` object keyed by those file names. Each override contains `url` and `revision`; files without an override continue to use the model's base URL unchanged. `model_info.json` remains the source of expected remote size and content hash for `pull` and `check`. No existing catalog entry needs migration. + +For each file, pull: + +1. constructs a URL from that file's fixed repository and revision; +2. downloads or resumes into a temporary path; +3. validates expected size and SHA-256; +4. atomically renames the completed file into the model directory. + +A model is available only when all required files validate. `flm check` uses the same per-file records. No generated overlay is copied into the model directory. + +The final directory is: + +```text +models/phi4-mini-it-aie4/ +├── Phi-4-mini-instruct.Q8_0.gguf +├── tokenizer.json +├── tokenizer_config.json +└── config.json +``` + +## Build and runtime configuration + +The feature is disabled by default. An AIE4 developer build enables: + +```text +FLM_ENABLE_CORELIB_AIE4=ON +RYZENAI_CORELIB_INCLUDE_DIR= +``` + +The build consumes the public header from the pinned corelib commit but does not link its import library. Calls go through the dynamically resolved function table. The normal `flm.exe` is produced and supports `pull`, `run`, and `serve`. + +This PR does not copy runtime DLLs. The developer supplies `ryzenai_corelib.dll` and its DynamicDispatch, XRT, and RyzenMM dependency closure. `FLM_AIE4_CORELIB_PATH` or the executable-relative `aie4` directory identifies corelib itself; its dependent DLL directory must be available to the Windows loader. + +A default build has no corelib compile or runtime requirement. An AIE4-enabled build with a missing runtime still starts and can run ordinary models; selecting `phi4-mini-it-aie4:4b` reports the missing runtime. + +## Request lifecycle and error policy + +A process-wide AIE4 access manager serializes AIE4 generation. One model instance handles one active generation at a time, matching the stream and mutable KV-cache ownership model. + +Failures before any operation is submitted are recoverable model/request errors. Failures after submission, or during synchronization, leave device completion uncertain. The model instance is then marked poisoned, its conversational state is cleared, and subsequent requests are refused until the model is unloaded and recreated. FastFlowLM does not continue on potentially inconsistent KV state. + +A cancellation is checked before prefill and between decode steps. It never destroys a stream while work is outstanding; submitted work is synchronized before the request releases model state. + +Capacity checks happen before submission. They account for the rendered prompt and requested generation budget and enforce the AIE4 decode limit of 4095. Unbounded/sentinel generation requests are capped rather than allowed to reach an unsupported attention window. + +## Expected file changes + +### Existing files + +```text +src/CMakeLists.txt +src/CMakePresets.json +src/common/AutoModel/automodel.cpp +src/common/AutoModel/modeling_phi4.cpp +src/include/AutoModel/automodel.hpp +src/include/AutoModel/modeling_phi4.hpp +src/pull/model_downloader.cpp +src/pull/model_downloader.hpp +src/model_list.json +src/model_info.json +src/runner/runner.cpp +src/server/rest_handler.cpp +src/server/server.cpp +src/src/main.cpp +``` + +Only files proven necessary during implementation should be changed. In particular, downloader changes are limited to optional per-file sources, and shared frontend changes are limited to behavior the AIE4 route requires. + +### New product files + +```text +src/include/corelib/corelib_api.hpp +src/include/corelib/corelib_object.hpp +src/include/corelib/corelib_runtime.hpp +src/common/corelib/corelib_api.cpp +src/common/corelib/corelib_runtime.cpp +src/common/corelib/corelib_sources.cmake +src/include/models/phi4/phi4_corelib_aie4.hpp +src/include/models/phi4/phi4_corelib_constants.hpp +src/include/models/phi4/phi4_corelib_gguf.hpp +src/include/models/phi4/phi4_corelib_shape_plan.hpp +src/include/models/phi4/phi4_corelib_host.hpp +src/common/corelib/phi4_corelib_aie4.cpp +src/common/corelib/phi4_corelib_gguf.cpp +src/common/corelib/phi4_corelib_shape_plan.cpp +src/common/corelib/phi4_corelib_host.cpp +``` + +The host component owns lazy Q8_0 embedding-row decode, F32-to-BF16 norm conversion, and FP32 RoPE-table derivation. The engine owns only model state and operator sequencing. + +## Testing + +### Unit tests without AIE4 hardware + +Tests cover: + +- valid GGUF v3 metadata and tensor-directory parsing; +- truncation, arithmetic overflow, bad alignment, duplicate names, and out-of-file ranges; +- missing tensors and incorrect dtype, shape, or byte length; +- zero-copy QKV and gate/up splits at exact row boundaries; +- Phi-4 architecture validation; +- tokenizer/config/GGUF disagreement; +- missing corelib symbols and exact ABI mismatch; +- object release and cleanup through a fake corelib; +- corelib call descriptors, group size 64, Q8_0 type, sequencing, and synchronization; +- per-file pull URLs, revisions, hashes, resume behavior, and atomic completion; +- unchanged behavior for existing single-source catalog entries; +- frontend routing and the absence of fallback; +- request bounds, cancellation, and poisoned-instance behavior. + +### Runtime integration + +Against the real DLL, tests verify: + +- ABI `0.3.0`; +- every required symbol resolves; +- the dependency self-test succeeds; +- device-context reporting agrees with the test environment. + +### Required AIE4 acceptance run + +Before completion, run the produced `flm.exe` on a real AIE4 system: + +```powershell +flm pull phi4-mini-it-aie4:4b +flm check phi4-mini-it-aie4:4b +flm run phi4-mini-it-aie4:4b +flm serve phi4-mini-it-aie4:4b +``` + +The acceptance run must include: + +1. `What is 2+2?`, with a correct, self-terminated answer; +2. `What does AMD do?`, with a relevant answer; +3. at least ten prompts in one loaded process; +4. `/api/chat` and `/v1/chat/completions`; +5. request cancellation; +6. prompt and generation limit boundaries; +7. at least ten complete load-and-generate cycles, checking for empty or all-zero token output; +8. backend evidence proving corelib/AIE4 execution and no CPU/NPU2 fallback; +9. model load time, cold and warm TTFT, and decode tokens/second. + +The record identifies the machine, power mode, FastFlowLM commit, corelib commit, corelib ABI, GGUF revision, commands, and outcomes. Performance numbers are descriptive, not a pass/fail gate, unless a regression threshold is agreed separately. + +## Commit structure + +Keep the work in one PR with reviewable commits: + +1. `build: add optional dynamic corelib 0.3.0 runtime` +2. `feat: add validated Phi-4 Q8_0 GGUF reader` +3. `feat: add corelib-backed Phi-4 AIE4 engine` +4. `feat: route Phi-4 GGUF models through AIE4` +5. `feat: pull Phi-4 GGUF and tokenizer from pinned sources` +6. `test: validate Phi-4 GGUF AIE4 integration` +7. `docs: document developer setup and hardware results` + +Each of commits 1–5 must compile before the next product commit is added. The test and documentation commits may depend on the completed product path. No commit adds a generated tensor manifest. + +## Completion criteria + +The PR is complete only when: + +- default builds and existing model behavior remain unchanged; +- an AIE4-enabled build produces the normal `flm.exe`; +- ordinary models remain usable when corelib is absent; +- the new tag pulls and checks all files from their pinned sources; +- the installed model contains no manifest, ONNX model, or converted weights; +- the GGUF is mapped directly and Q8_0 projection views are passed to corelib's explicit requantized APIs; +- runtime ABI is exactly `0.3.0`; +- automated unit and fake-corelib tests pass; +- the real-DLL integration checks pass; +- the required real-AIE4 acceptance run passes; +- no CPU or NPU2 fallback exists; +- documentation states that Q8_0-to-group-64 conversion is lossy and records the tested revisions and hardware results. From 4948126b560c2fa095f2be20599d475046892323 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 02:11:56 -0700 Subject: [PATCH 02/37] docs: plan Phi-4 GGUF AIE4 integration --- .../plans/2026-09-11-phi4-gguf-aie4.md | 1300 +++++++++++++++++ .../specs/2026-09-11-phi4-gguf-aie4-design.md | 5 +- 2 files changed, 1303 insertions(+), 2 deletions(-) create mode 100644 docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md diff --git a/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md b/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md new file mode 100644 index 00000000..65706fa7 --- /dev/null +++ b/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md @@ -0,0 +1,1300 @@ +# Phi-4 Q8_0 GGUF on AIE4 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add the catalog model `phi4-mini-it-aie4:4b`, pull its pinned GGUF and tokenizer/config files, and run it through the dynamically loaded ryzenai-corelib 0.3.0 AIE4 backend from the normal FastFlowLM CLI and REST server. + +**Architecture:** Keep `Phi4` as the existing tokenizer/chat/sampling frontend and select a new `phi4_corelib_aie4` causal-LM engine only when `details.execution_backend == "corelib_aie4_gguf"`. The engine owns a validated, read-only GGUF mapping, derives host-only embedding/norm/RoPE data, creates all corelib weights serially through the explicit Q8_0-to-group-64 APIs, and executes one-stream prefill/decode with fixed KV caches. A feature-gated dynamic ABI layer keeps default builds and non-AIE4 models independent of the corelib DLL. + +**Tech Stack:** C++20, CMake 3.22+, Windows `LoadLibraryExW`/file mapping APIs, nlohmann/json, libcurl, existing FastFlowLM tokenizer/sampler/server, ryzenai-corelib C ABI 0.3.0, CTest, PowerShell for real-device acceptance. + +**Spec:** [`docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md`](../specs/2026-09-11-phi4-gguf-aie4-design.md) + +## Global Constraints + +- Work from current FastFlowLM `main`; use `origin/pr/706` (`0355fe4c4f3bf4bdb476ef5fd9c20f84411a162f`) only as a structural reference. Do not cherry-pick its ONNX manifest, overlay, packaging, process-termination policy, or stale corelib ABI. +- Corelib source is `VitisAI/ryzenai-corelib` commit `3c35aebdefa3f0c2255668bab1be5648ece320f8`; compile against its public `include/ryzenai/corelib.h` only and require runtime ABI exactly `0.3.0` (major, minor, and patch). +- The model is exactly `unsloth/Phi-4-mini-instruct-GGUF` revision `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80`, file `Phi-4-mini-instruct.Q8_0.gguf`, GGML type Q8_0. +- `tokenizer.json`, `tokenizer_config.json`, and `config.json` are exactly from `microsoft/Phi-4-mini-instruct` revision `cfbefacb99257ffa30c83adab238a50856ac3083`. +- Keep the existing `phi4-mini-it:4b` Q4NX/NPU2 path unchanged. An absent `details.execution_backend` means legacy NPU2; `corelib_aie4_gguf` means only AIE4; every other value is an error. +- Never infer a backend from hardware, filenames, or quantization. Never fall back from `corelib_aie4_gguf` to CPU or NPU2. +- `FLM_ENABLE_CORELIB_AIE4` defaults to `OFF`. A default build must not require corelib headers or DLLs. An enabled build uses `RYZENAI_CORELIB_INCLUDE_DIR`, does not link `ryzenai_corelib.lib`, and loads corelib only after an AIE4 model is selected. +- DLL lookup order is exactly: the absolute file named by `FLM_AIE4_CORELIB_PATH`, then `/aie4/ryzenai_corelib.dll`. Never search the current working directory. +- This PR does not copy or package corelib, DynamicDispatch, XRT, RyzenMM, or any other runtime DLL. Do not change MSI, WiX, Inno Setup, or installer inputs. +- Do not add ONNX loading, an ONNX initializer manifest, a JSON tensor manifest, converted weights, a packed-weight cache, arbitrary local-GGUF support, another model family, another quantization, or Python runtime code. +- GGUF is version 3, little-endian, directly memory-mapped read-only. All counts, products, offsets, alignments, and byte ranges use checked arithmetic before pointer/span creation. +- Validate the complete GGUF, config, and tokenizer contract before creating a stream, device tensor, or device weight. +- Fixed model contract: architecture `phi3`; 32 layers; hidden 3072; intermediate 8192; 24 query heads; 8 KV heads; head size 128; vocabulary 200064; partial rotary width 96; RMS epsilon `1e-5`; original/max supported context 4096. +- Require finite positive `phi3.rope.freq_base` and `phi3.rope.scaling.attn_factor`; require `phi3.rope.scaling.original_context_length == 4096`; reject the long-RoPE branch; accept optional `rope_factors_short.weight` only as F32 `[48]`. +- Require every projection, `token_embd.weight`, and tied LM-head source to be Q8_0; require `output.weight` to be absent; require every norm to be F32; split fused QKV and gate/up only on complete Q8_0 rows (34 bytes per 32 weights), without copy or dequantization. +- Require tokenizer vocabulary size 200064; require `tokenizer.json` to map `<|end|>` to 200020 and `<|endoftext|>` to 199999; require GGUF EOS 200020 and `config.json` EOS 199999; configure the frontend stop set as their union `{200020, 199999}`. Require `tokenizer_config.json` `add_bos_token == false` and a chat template containing `<|user|>`, `<|end|>`, and `<|assistant|>`. +- Every error for a model field or tensor names the field/tensor, actual value, and expected value. Do not repair, reinterpret, or silently accept mismatches. +- Every quantized weight uses `group_size = 64`, `ryzenai_corelib_gguf_quant_type_q8_0`, and the explicit `*_weights_create_gguf_requantized` entry point. The conversion is lossy by design. +- Create the 129 matmul weights (Q/K/V/O for 32 layers plus tied LM head), 32 SSMLP weights, and one RMSNorm weight serially with `threads = 0`; do not add concurrent creation. +- Keep one corelib stream, fixed K/V caches shaped `[8,4096,128]`, helper-derived padded extents, whole-prompt prefill, and one-token decode. The maximum usable total decode window is 4095. +- Check cancellation before prefill and between decode steps. Never release/destroy a stream with submitted work outstanding; synchronize submitted work before releasing request ownership. +- A failure before the first successful submission is recoverable. A failure after submission or during synchronization poisons that model instance, clears its conversation state, and makes later requests fail until unload/reload. +- Process-wide AIE4 request access and per-instance mutable state are serialized. Existing server NPU serialization may be reused, but every generation route must participate and exception/cancellation paths must release it exactly once. +- `model_info.json` remains authoritative for each file's exact byte size and content hash. A final model is available only when all four final files validate; `.part` files never count. +- Keep all work in one PR and use the seven commit messages fixed by the design. Each product commit 1–5 must compile and pass its focused tests before the next product commit begins. +- The PR is not complete until the real-DLL checks and the full real-AIE4 acceptance matrix pass and the hardware record is committed. Performance is descriptive unless a separate threshold is approved. + +## Assumptions + +- The AIE4 feature is built and exercised on Windows with MSVC; Linux/default builds remain feature-off and unchanged. +- `src/CMakePresets.json` remains the source of the current FastFlowLM/NPU version values; the new preset inherits them rather than duplicating them. +- The developer provides a corelib 0.3.0 installation and its dependency directories. This PR locates only `ryzenai_corelib.dll`; dependent DLL discovery remains the Windows loader's responsibility. +- The sibling `../ryzenai-corelib` working tree is currently on another branch, while `origin/main` points to the required commit. Verification and build commands must address commit `3c35aebdefa3f0c2255668bab1be5648ece320f8` explicitly and must not overwrite sibling work. +- Exact remote file sizes and SHA-256 values are immutable implementation data obtainable from the pinned URLs. Task 5 computes and independently verifies them before catalog edits; no unverified value may be committed. +- Real-AIE4 performance values are not known until Task 7 runs. The documentation commit is blocked until the acceptance script has emitted actual values and provenance. + +--- + +## Repository Findings and Reuse Boundaries + +- Current `src/common/AutoModel/modeling_phi4.cpp` always calls `_shared_load_model`, constructs `Q4NX`, and creates `phi4_npu`; backend selection must be added there, not in `get_auto_model`. +- Current `_shared_load_model` in `src/common/AutoModel/automodel.cpp` combines generic model/tokenizer state with NPU2 `npu_xclbin_manager` creation. Split those responsibilities so the AIE4 route never constructs the legacy backend. +- Current downloads write directly to the final path and use base-repository URLs. The per-file source and atomic-resume behavior therefore require coordinated changes in `model_downloader.*` and `download_model.*`. +- Current server request queuing already serializes most NPU endpoints, but `/v1/completions` is omitted and exception paths manually release the lock. Extend the existing mechanism rather than adding a second queue. +- `origin/pr/706` supplies useful shapes for the dynamic loader, move-only handles, shape plan, fake corelib, frontend routing, and engine sequencing. Its source contract is ONNX/manifest-based, its symbol list predates ABI 0.3.0, its environment variable is different, and its post-submit policy terminates the process; none of those parts are reusable unchanged. +- The pinned corelib header supplies tensor windows, matmul/SSMLP Q8_0 requantized creation, RMSNorm scale creation/dispatch, and flat-MHA. The implementation must resolve exactly those public functions and no testing-only symbols. + +## Locked Interfaces + +Use these names and signatures throughout the tasks so independently implemented pieces join without renaming: + +```cpp +namespace flm::corelib { + +struct CorelibVersion { + std::uint32_t major; + std::uint32_t minor; + std::uint32_t patch; +}; + +class CorelibError final : public std::runtime_error { +public: + CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text); + ryzenai_corelib_status status() const noexcept; + const std::string& call() const noexcept; + const std::string& detail() const noexcept; +}; + +struct CorelibFunctions { + decltype(&::ryzenai_corelib_get_version) get_version; + decltype(&::ryzenai_corelib_status_to_string) status_to_string; + decltype(&::ryzenai_corelib_get_last_error_message) get_last_error_message; + decltype(&::ryzenai_corelib_selftest_dependencies) selftest_dependencies; + decltype(&::ryzenai_corelib_has_device_context) has_device_context; + decltype(&::ryzenai_corelib_object_release) object_release; + decltype(&::ryzenai_corelib_create_stream) create_stream; + decltype(&::ryzenai_corelib_stream_synchronize) stream_synchronize; + decltype(&::ryzenai_corelib_create_device_tensor) create_device_tensor; + decltype(&::ryzenai_corelib_create_tensor_window) create_tensor_window; + decltype(&::ryzenai_corelib_tensor_write) tensor_write; + decltype(&::ryzenai_corelib_tensor_read) tensor_read; + decltype(&::ryzenai_corelib_tensor_get_byte_size) tensor_get_byte_size; + decltype(&::ryzenai_corelib_tensor_get_data_type) tensor_get_data_type; + decltype(&::ryzenai_corelib_matmul_bf16_pad_shape) matmul_pad_shape; + decltype(&::ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized) + matmul_weights_create_gguf_requantized; + decltype(&::ryzenai_corelib_matmul_bf16) matmul; + decltype(&::ryzenai_corelib_ssmlp_bf16_pad_rows) ssmlp_pad_rows; + decltype(&::ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) + ssmlp_weights_create_gguf_requantized; + decltype(&::ryzenai_corelib_ssmlp_bf16) ssmlp; + decltype(&::ryzenai_corelib_rmsnorm_bf16_weights_create_scale) + rmsnorm_weights_create_scale; + decltype(&::ryzenai_corelib_rmsnorm_bf16_pad_rows) rmsnorm_pad_rows; + decltype(&::ryzenai_corelib_rmsnorm_bf16) rmsnorm; + decltype(&::ryzenai_corelib_flat_mha_bf16_pad_rows) flat_mha_pad_rows; + decltype(&::ryzenai_corelib_flat_mha_bf16) flat_mha; + decltype(&::ryzenai_corelib_cleanup) cleanup; +}; + +class CorelibApi final { +public: + using Resolver = std::function; + static std::shared_ptr Load(const std::filesystem::path& dll); + static std::shared_ptr ResolveForTest(Resolver resolver); + static std::filesystem::path ResolveLibraryPath( + const std::filesystem::path& executable_dir); + const CorelibFunctions& functions() const noexcept; + CorelibVersion runtime_version() const noexcept; + void Check(ryzenai_corelib_status status, std::string_view call) const; + void RegisterObject() const noexcept; + void Release(void* object) const noexcept; + std::size_t live_object_count() const noexcept; +}; + +class CorelibRuntime final { +public: + static std::shared_ptr GetOrCreate( + const std::filesystem::path& executable_dir); + static std::shared_ptr CreateForTest( + std::shared_ptr api); + static void ShutdownProcess(); + std::unique_lock AcquireExecution(); + const std::shared_ptr& api() const noexcept; +}; + +} // namespace flm::corelib +``` + +`UniqueObject` is move-only and calls `CorelibApi::Release` exactly once. Define `UniqueStream`, `UniqueTensor`, `UniqueTensorWindow`, `UniqueMatMulWeights`, `UniqueSsMlpWeights`, and `UniqueRmsNormWeights`; each successful C create is wrapped immediately. + +```cpp +namespace flm::phi4 { + +inline constexpr std::int64_t kLayerCount = 32; +inline constexpr std::int64_t kHiddenSize = 3072; +inline constexpr std::int64_t kIntermediateSize = 8192; +inline constexpr std::int64_t kQueryHeadCount = 24; +inline constexpr std::int64_t kKvHeadCount = 8; +inline constexpr std::int64_t kHeadSize = 128; +inline constexpr std::int64_t kQueryDimension = 3072; +inline constexpr std::int64_t kKvDimension = 1024; +inline constexpr std::int64_t kVocabularySize = 200064; +inline constexpr std::int64_t kRopeDimension = 96; +inline constexpr std::int64_t kMaxSequenceLength = 4096; +inline constexpr std::int64_t kMaxDecodeWindow = 4095; +inline constexpr std::uint32_t kRequantizedGroupSize = 64; +inline constexpr float kRmsEpsilon = 1.0e-5f; + +struct TensorView { + std::string_view name; + std::span bytes; + std::vector logical_shape; + std::uint32_t ggml_type; +}; + +struct FloatTensorView { + std::string_view name; + std::span values; + std::vector logical_shape; +}; + +struct ProjectionViews { + // AttentionQkv: q, k, v in indices 0,1,2 and count == 3. + // GateUp: gate, up in indices 0,1 and count == 2. + std::array values; + std::size_t count; +}; + +struct GgufPhi4Metadata { + std::string architecture; + std::uint64_t layer_count; + std::uint64_t hidden_size; + std::uint64_t intermediate_size; + std::uint64_t attention_head_count; + std::uint64_t kv_head_count; + std::uint64_t context_length; + std::uint64_t rope_dimension_count; + double rope_frequency_base; + double rope_attention_factor; + std::uint64_t rope_original_context_length; + std::uint64_t tokenizer_vocabulary_size; + bool add_bos_token; +}; + +class Phi4GgufPackage final { +public: + static std::shared_ptr Open( + const std::filesystem::path& gguf_path); + TensorView RequireQ8( + std::string_view name, + std::span expected_shape) const; + FloatTensorView RequireF32( + std::string_view name, + std::span expected_shape) const; + ProjectionViews AttentionQkv(std::size_t layer) const; + ProjectionViews GateUp(std::size_t layer) const; + GgufPhi4Metadata Metadata() const; + void ValidatePhi4Contract( + const nlohmann::json& config, + const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const; +}; + +struct RopeTables { + std::vector cosine; + std::vector sine; +}; + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, + std::span token_ids); +std::vector ConvertF32ToBf16( + std::span values); +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors); + +struct Phi4RowExtents { + std::int64_t query_rows; + std::int64_t kv_rows; + std::int64_t output_rows; + std::int64_t ssmlp_rows; + std::int64_t rmsnorm_rows; + std::int64_t flat_mha_rows; +}; + +class Phi4ShapePlan final { +public: + static Phi4ShapePlan Build( + const std::shared_ptr& api); + const Phi4RowExtents& ForRows(std::size_t live_rows) const; + const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; + const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; +}; + +class phi4_corelib_aie4 final : public causal_lm { +public: + phi4_corelib_aie4( + LM_Config config, + std::shared_ptr package, + std::shared_ptr runtime, + std::uint32_t max_length = 4096); + buffer forward(int id) override; + buffer prefill(std::vector& ids, void* payload = nullptr) override; + void set_context_length(int length) override; + void load_weights(Q4NX&) override; + void update_max_length(std::uint32_t max_length) override; + void clear_context() override; + buffer get_k_cache(int layer, int index) override; + buffer get_v_cache(int layer, int index) override; + int get_current_context_length() override; + int checkpoint() override; + int restore() override; + bool poisoned() const noexcept; +}; + +} // namespace flm::phi4 +``` + +Frontend additions: + +```cpp +class ModelRequestError final : public std::runtime_error { +public: + ModelRequestError(int http_code, bool session_cleared, std::string message); + int http_code() const noexcept; + bool session_cleared() const noexcept; +}; + +struct lm_uniform_input_t { + // existing members remain unchanged + std::optional requested_max_new_tokens; +}; + +class AutoModel { +public: + virtual bool uses_corelib_aie4() const noexcept { return false; } +protected: + void _shared_initialize_model_state( + std::string model_path, json model_info, int context_length); + void _shared_initialize_legacy_npu(bool enable_preemption); +}; +``` + +Downloader additions: + +```cpp +namespace download_utils { +enum class HashAlgorithm { Sha256, GitBlobSha1 }; +struct DownloadRequest { + std::string url; + std::filesystem::path destination; + std::uint64_t expected_size; + HashAlgorithm hash_algorithm; + std::string expected_hash; +}; +bool download_file_atomic( + const DownloadRequest& request, + std::function progress_cb = nullptr); +} + +struct ModelFileSource { + std::string url; + std::string revision; +}; + +ModelFileSource resolve_file_source( + const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope); +``` + +## File Map + +### New production files + +- `src/include/corelib/corelib_api.hpp` — ABI 0.3.0 function table and typed errors. +- `src/include/corelib/corelib_object.hpp` — move-only ownership for every resolved object type. +- `src/include/corelib/corelib_runtime.hpp` — lazy process runtime, dependency/device validation, and execution mutex. +- `src/common/corelib/corelib_api.cpp` — safe DLL lookup, version-first symbol resolution, and error copying. +- `src/common/corelib/corelib_runtime.cpp` — singleton lifecycle and cleanup. +- `src/common/corelib/corelib_sources.cmake` — one source list shared by product and tests. +- `src/include/models/phi4/phi4_corelib_constants.hpp` — fixed validated architecture and group-64 constants. +- `src/include/models/phi4/phi4_corelib_gguf.hpp` — mapped GGUF views and model-contract API. +- `src/common/corelib/phi4_corelib_gguf.cpp` — checked GGUF v3 parser, tensor mapping/splitting, cross-source validation. +- `src/include/models/phi4/phi4_corelib_shape_plan.hpp` — helper-derived row extents and attention descriptor. +- `src/common/corelib/phi4_corelib_shape_plan.cpp` — queries and caches every required padded extent. +- `src/include/models/phi4/phi4_corelib_host.hpp` — lazy embedding decode, BF16 conversion, and RoPE derivation. +- `src/common/corelib/phi4_corelib_host.cpp` — bounds-safe host utilities only. +- `src/include/models/phi4/phi4_corelib_aie4.hpp` — causal-LM engine and poisoned-state surface. +- `src/common/corelib/phi4_corelib_aie4.cpp` — serial weight creation, persistent tensors/caches, prefill/decode sequencing. + +### New test/support files + +- `src/test/phi4_corelib_aie4/CMakeLists.txt` — standalone host/fake/real-DLL suite plus feature-on/off compile checks. +- `src/test/phi4_corelib_aie4/test_support.hpp` — `CHECK`, exception-message assertion, temporary-directory helpers. +- `src/test/phi4_corelib_aie4/gguf_fixture.hpp` — deterministic GGUF v3 byte builder with corruption controls. +- `src/test/phi4_corelib_aie4/fake_corelib.hpp` +- `src/test/phi4_corelib_aie4/fake_corelib.cpp` — complete fake of every resolved ABI 0.3.0 symbol and call recorder. +- `src/test/phi4_corelib_aie4/test_corelib_api.cpp` +- `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` +- `src/test/phi4_corelib_aie4/test_phi4_host.cpp` +- `src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp` +- `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` +- `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` +- `src/test/phi4_corelib_aie4/test_model_downloader.cpp` +- `src/test/phi4_corelib_aie4/test_real_corelib.cpp` +- `src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1` — repeatable CLI/REST/cancellation/boundary/load-cycle runner and JSON evidence writer. + +### Existing files to modify + +- `src/CMakeLists.txt` — optional corelib target; no import-library link and no packaging changes. +- `src/CMakePresets.json` — `windows-aie4` configure/build presets using the include-dir environment variable. +- `src/include/AutoModel/automodel.hpp` +- `src/common/AutoModel/automodel.cpp` — split generic model initialization from legacy NPU2 initialization; typed request errors and generation budget. +- `src/include/AutoModel/modeling_phi4.hpp` +- `src/common/AutoModel/modeling_phi4.cpp` — explicit routing, cross-source validation, capacity/cancellation/poison policy. +- `src/pull/download_model.hpp` +- `src/pull/download_model.cpp` — resumable `.part` transfer, size/hash verification, atomic promotion. +- `src/pull/model_downloader.hpp` +- `src/pull/model_downloader.cpp` — per-file source resolution and shared pull/check records. +- `src/model_list.json` — one new tag and `file_sources` schema instance. +- `src/model_info.json` — four immutable size/hash records. +- `src/runner/runner.cpp` — pass CLI generation budget and preserve poisoned-model errors. +- `src/server/rest_handler.cpp` — pass endpoint generation budgets/cancellation and emit typed 400/500 errors. +- `src/server/server.hpp` +- `src/server/server.cpp` — include all generation routes in the existing process-wide queue and release request ownership exactly once. +- `src/src/main.cpp` — feature-gated healthy corelib shutdown only; no startup load. +- `docs/docs/models/phi.md` — AIE4 tag, setup, limits, lossy conversion, and no-fallback behavior. +- `docs/docs/benchmarks/phi4_results.md` — pinned revisions and real-hardware acceptance results. + +## Task Ordering + +Tasks are strictly sequential. Tasks 1–5 are product commits and each must compile before the next starts. Task 6 is the integrated test commit. Task 7 runs real hardware acceptance and records documentation. Do not split this design into another PR. + +### Task 1: Optional Dynamic Corelib 0.3.0 Runtime + +**Files:** +- Create: `src/include/corelib/corelib_api.hpp` +- Create: `src/include/corelib/corelib_object.hpp` +- Create: `src/include/corelib/corelib_runtime.hpp` +- Create: `src/common/corelib/corelib_api.cpp` +- Create: `src/common/corelib/corelib_runtime.cpp` +- Create: `src/common/corelib/corelib_sources.cmake` +- Create: `src/test/phi4_corelib_aie4/CMakeLists.txt` +- Create: `src/test/phi4_corelib_aie4/test_support.hpp` +- Create: `src/test/phi4_corelib_aie4/fake_corelib.hpp` +- Create: `src/test/phi4_corelib_aie4/fake_corelib.cpp` +- Create: `src/test/phi4_corelib_aie4/test_corelib_api.cpp` +- Create: `src/test/phi4_corelib_aie4/test_real_corelib.cpp` +- Modify: `src/CMakeLists.txt` around options, source collection, and `flm` linkage +- Modify: `src/CMakePresets.json` configure/build preset arrays +- Modify: `src/src/main.cpp` include block and normal shutdown path + +**Interfaces:** +- Produces the `flm::corelib` interfaces in **Locked Interfaces**. +- Resolves only the 26 function pointers listed in `CorelibFunctions` above. +- `CorelibRuntime::AcquireExecution()` is the process-wide serialization primitive consumed by Task 3. +- `CorelibApi::ResolveForTest` and `CreateForTest` are test-only dependency injection; production always uses `Load`/`GetOrCreate`. + +- [ ] **Step 1: Write failing ABI, path, lifetime, and feature-gate tests** + +In `test_corelib_api.cpp`, define and invoke these named cases from `main()`: + +```cpp +TestVersionIsResolvedBeforeEveryOtherSymbol(); +TestExactlyVersion030IsAccepted(); +TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions(); +TestEveryRequiredSymbolIsResolvedExactlyOnce(); +TestMissingSymbolNamesTheSymbolAndUnloadsTheDll(); +TestCorelibErrorCopiesStatusCallAndThreadLocalDetail(); +TestEnvironmentPathMustBeAnAbsoluteDllPath(); +TestEnvironmentPathWinsOverExecutableRelativePath(); +TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory(); +TestEveryUniqueObjectReleasesExactlyOnceAfterMoves(); +TestRuntimeRunsDependencySelftestAndRequiresDeviceContext(); +TestExecutionLeaseSerializesTwoThreads(); +TestCleanupRunsAfterTheLastObjectAndOnlyOnce(); +``` + +The fake must export all 26 required symbols, let each status/detail/version/device result be injected, record resolution order, count live objects/releases, and record maximum simultaneous execution leases. In `test_real_corelib.cpp`, return CTest skip code 77 only when `FLM_AIE4_CORELIB_PATH` is unset; if it is set, assert ABI 0.3.0, every symbol, dependency self-test, and device context. + +Add two object-library compile guards in the test CMake project: one builds the production frontend/CMake source list without `FLM_ENABLE_CORELIB_AIE4` and no corelib include path; the other builds with the define and pinned include path. + +- [ ] **Step 2: Run RED checks** + +```powershell +cmake -S src/test/phi4_corelib_aie4 -B src/build/phi4-corelib-tests ` + -G "Visual Studio 17 2022" -A x64 ` + -DRYZENAI_CORELIB_INCLUDE_DIR=C:/Users/chiz/work/ryzenai-corelib/include +cmake --build src/build/phi4-corelib-tests --config Release --target test_corelib_api +``` + +Expected: configure or compile fails because the new adapter/runtime headers and sources do not exist. + +- [ ] **Step 3: Implement the minimal dynamic adapter and RAII layer** + +Implement version-first resolution: resolve and call `ryzenai_corelib_get_version`, reject anything other than `0.3.0`, then resolve the remaining 26 names. `Check` must copy `get_last_error_message()` before calling `status_to_string()`. Load with: + +```cpp +LoadLibraryExW(path.c_str(), nullptr, + LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); +``` + +`FLM_AIE4_CORELIB_PATH` must be an absolute file path. With it unset, return `absolute(executable_dir / "aie4" / "ryzenai_corelib.dll")`; do not call `LoadLibraryW` with a bare filename. Wrap every successful object immediately in the matching `UniqueObject`. + +`CorelibRuntime::GetOrCreate` must be lazy and process-wide. `CreateForTest` runs `selftest_dependencies`, then `has_device_context`, and rejects either failure before reporting ready. `ShutdownProcess` waits for the execution mutex, requires no live objects, calls `cleanup` once, then drops the API/module. + +- [ ] **Step 4: Integrate the feature-gated build** + +Add: + +```cmake +option(FLM_ENABLE_CORELIB_AIE4 + "Enable Phi-4 Q8_0 GGUF execution through ryzenai-corelib" OFF) +if(FLM_ENABLE_CORELIB_AIE4) + if(NOT WIN32) + message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows") + endif() + find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) + include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake") + add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES}) + target_include_directories(flm_corelib_aie4 PUBLIC + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") + target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1) + target_link_libraries(flm PRIVATE flm_corelib_aie4) +endif() +``` + +Do not add `ryzenai_corelib.lib`, runtime-copy commands, or installer rules. Add `windows-aie4` presets inheriting `windows-default`, using binary directory `${sourceDir}/build-aie4`, `FLM_ENABLE_CORELIB_AIE4=ON`, and `RYZENAI_CORELIB_INCLUDE_DIR=$env{RYZENAI_CORELIB_INCLUDE_DIR}`. + +Guard the `main.cpp` include and final `CorelibRuntime::ShutdownProcess()` call with `FLM_ENABLE_CORELIB_AIE4`; do not touch startup, `pull`, `list`, or non-AIE4 command paths. + +- [ ] **Step 5: Run GREEN checks and the default-build regression gate** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_corelib_api +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_corelib_api$" --output-on-failure +cmake --preset windows-default -S src +cmake --build --preset windows-default +$env:RYZENAI_CORELIB_INCLUDE_DIR='C:/Users/chiz/work/ryzenai-corelib/include' +cmake --preset windows-aie4 -S src +cmake --build --preset windows-aie4 +``` + +Expected: `test_corelib_api` passes; both builds produce the normal `flm.exe`; `dumpbin /imports src/build-aie4/Release/flm.exe | findstr /i ryzenai_corelib` prints no import. + +- [ ] **Step 6: Refactor only duplicated resolver/RAII mechanics and rerun Step 5** + +Keep symbol names in one constexpr table or one macro expansion so the function table, resolver, and fake cannot drift. Do not introduce a generic plugin framework. + +- [ ] **Step 7: Commit review gate** + +```powershell +git add src/CMakeLists.txt src/CMakePresets.json src/src/main.cpp ` + src/include/corelib src/common/corelib/corelib_api.cpp ` + src/common/corelib/corelib_runtime.cpp src/common/corelib/corelib_sources.cmake ` + src/test/phi4_corelib_aie4 +git commit -m "build: add optional dynamic corelib 0.3.0 runtime" +``` + +Expected: one build/runtime commit; no product model route exists yet. + +### Task 2: Validated Phi-4 GGUF v3 Package + +**Files:** +- Create: `src/include/models/phi4/phi4_corelib_constants.hpp` +- Create: `src/include/models/phi4/phi4_corelib_gguf.hpp` +- Create: `src/common/corelib/phi4_corelib_gguf.cpp` +- Create: `src/test/phi4_corelib_aie4/gguf_fixture.hpp` +- Create: `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` +- Modify: `src/common/corelib/corelib_sources.cmake` +- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` + +**Interfaces:** +- Produces `TensorView`, `FloatTensorView`, `ProjectionViews`, `GgufPhi4Metadata`, and `Phi4GgufPackage` exactly as declared in **Locked Interfaces**. +- `AttentionQkv(layer)` returns `count == 3`, ordered Q/K/V. `GateUp(layer)` returns `count == 2`, ordered gate/up. +- Task 3 consumes the returned byte spans directly in corelib GGUF component structs and retains the package for the engine lifetime. + +- [ ] **Step 1: Write a deterministic GGUF v3 fixture builder** + +`gguf_fixture.hpp` must write little-endian scalar/string/array metadata, tensor directory entries, configurable alignment, and aligned payloads. It must expose mutations for a truncated string/directory, count/product overflow, zero/non-power-of-two alignment, duplicate names, out-of-file ranges, overlapping ranges, unsupported metadata types, dtype mismatch, shape mismatch, and payload-length mismatch. + +Use exact fixture tensors: + +```cpp +"token_embd.weight" logical [200064, 3072] Q8_0 +"output_norm.weight" logical [3072] F32 +"blk.0.attn_norm.weight" logical [3072] F32 +"blk.0.ffn_norm.weight" logical [3072] F32 +"blk.0.attn_qkv.weight" logical [5120, 3072] Q8_0 +"blk.0.attn_output.weight" logical [3072, 3072] Q8_0 +"blk.0.ffn_up.weight" logical [16384, 3072] Q8_0 +"blk.0.ffn_down.weight" logical [3072, 8192] Q8_0 +"rope_factors_short.weight" logical [48] F32 +``` + +The fixture may use reduced payload backing for parser-only tests only when its declared dimensions are also reduced; contract tests use directory-only synthetic spans sized with checked Q8_0 arithmetic and a sparse temporary file. + +- [ ] **Step 2: Write failing parser and corruption tests** + +Define and invoke these cases: + +```cpp +TestValidV3HeaderMetadataDirectoryAndAlignment(); +TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely(); +TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail(); +TestCountProductAlignmentAndOffsetOverflowFail(); +TestZeroAndNonPowerOfTwoAlignmentFail(); +TestDuplicateTensorNamesFail(); +TestOutOfFileAndOverlappingTensorRangesFail(); +TestUnsupportedUnskippableMetadataTypeFails(); +TestRequireQ8AndRequireF32ReportNameActualAndExpected(); +TestAttentionQkvReturnsThreeZeroCopyWholeRowViews(); +TestGateUpReturnsTwoZeroCopyWholeRowViews(); +TestSplitRejectsNonIntegralQ8RowBoundary(); +TestViewsPointIntoTheReadOnlyMapping(); +``` + +For split checks, assert Q8_0 row bytes are `input_width / 32 * 34`; Q/K/V byte offsets are 0, `3072 * row_bytes`, and `4096 * row_bytes`; gate/up offsets are 0 and `8192 * row_bytes`. + +- [ ] **Step 3: Run RED** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_gguf +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_gguf$" --output-on-failure +``` + +Expected: compile fails because `Phi4GgufPackage` is not defined. + +- [ ] **Step 4: Implement checked mapping and parsing** + +Map with `CreateFileW(..., GENERIC_READ, FILE_SHARE_READ, ..., OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, ...)`, `CreateFileMappingW(..., PAGE_READONLY, ...)`, and `MapViewOfFile(..., FILE_MAP_READ, ...)`. Parse magic `0x46554747`, require version 3, and use helper functions equivalent to: + +```cpp +std::uint64_t CheckedAdd(std::uint64_t a, std::uint64_t b, std::string_view field); +std::uint64_t CheckedMultiply(std::uint64_t a, std::uint64_t b, std::string_view field); +std::uint64_t AlignUp(std::uint64_t value, std::uint64_t alignment); +std::span RequireRange( + std::span file, std::uint64_t offset, + std::uint64_t length, std::string_view field); +``` + +Retain only contract metadata while safely skipping every encoded metadata value used by the pinned file. The retained GGUF keys are `general.architecture`, `general.alignment`, `phi3.block_count`, `phi3.context_length`, `phi3.embedding_length`, `phi3.feed_forward_length`, `phi3.attention.head_count`, `phi3.attention.head_count_kv`, `phi3.attention.layer_norm_rms_epsilon`, `phi3.rope.dimension_count`, `phi3.rope.freq_base`, `phi3.rope.scaling.attn_factor`, `phi3.rope.scaling.original_context_length`, `tokenizer.ggml.tokens` (array count only), and `tokenizer.ggml.add_bos_token`. Compute Q8_0 bytes as `elements / 32 * 34` only after requiring divisibility by 32; compute F32 bytes as `elements * 4`. Reverse GGUF dimensions into logical row-major shapes at the model boundary. Sort absolute tensor ranges and reject overlap. + +- [ ] **Step 5: Write failing full-model contract tests** + +Define table-driven mutations for every fixed field and every required tensor across all 32 layer names. Each assertion must check the thrown text contains the field/tensor, the actual value, and the expected value. Include these independent cases: + +```cpp +TestAcceptsExactPhi3Phi4Contract(); +TestRejectsWrongArchitectureAndEveryDimension(); +TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole(); +TestRejectsMixedQuantizationAndOutputWeightPresence(); +TestRequiresTiedQ8TokenEmbeddingAsLmHead(); +TestRequiresOriginal4096WindowAndRejectsLongRopeBranch(); +TestValidatesOptionalShortRopeFactorsAsF32Length48(); +TestRejectsNonFiniteOrNonPositiveRopeValues(); +TestRejectsConfigDisagreement(); +TestDerivesStopSetFromGgufConfigAndTokenizerIds(); +TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement(); +TestValidationCreatesNoCorelibObjects(); +``` + +The config checks are `model_type == "phi3"`, `num_hidden_layers`, `hidden_size`, `intermediate_size`, `num_attention_heads`, `num_key_value_heads`, `head_dim`, `vocab_size`, `rms_norm_eps`, and `original_max_position_embeddings`. Determine tokenizer vocabulary size from `tokenizer.json`'s model vocabulary plus added-token IDs without assuming contiguous object iteration; compare the maximum assigned ID plus one and the distinct ID count to 200064. + +- [ ] **Step 6: Implement `ValidatePhi4Contract` and make all tests green** + +Perform intrinsic GGUF validation in `Open`; perform config/tokenizer/GGUF comparison in `ValidatePhi4Contract`. Validation must complete before any Task 3 engine constructor invokes a corelib create call. + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_gguf +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_gguf$" --output-on-failure +``` + +Expected: all parser, split, corruption, and contract cases pass. + +- [ ] **Step 7: Refactor parser helpers and rerun Step 6** + +Keep cursor advancement, checked arithmetic, range validation, and error formatting in single helpers; keep Phi-4 tensor names in the package adapter rather than a generic GGUF layer. Do not broaden accepted GGUF types or architectures. + +- [ ] **Step 8: Commit review gate** + +```powershell +git add src/include/models/phi4/phi4_corelib_constants.hpp ` + src/include/models/phi4/phi4_corelib_gguf.hpp ` + src/common/corelib/phi4_corelib_gguf.cpp ` + src/common/corelib/corelib_sources.cmake ` + src/test/phi4_corelib_aie4/gguf_fixture.hpp ` + src/test/phi4_corelib_aie4/test_phi4_gguf.cpp ` + src/test/phi4_corelib_aie4/CMakeLists.txt +git commit -m "feat: add validated Phi-4 Q8_0 GGUF reader" +``` + +Expected: the commit parses and validates but cannot execute a model. + +### Task 3: Corelib-Backed Phi-4 AIE4 Engine + +**Files:** +- Create: `src/include/models/phi4/phi4_corelib_shape_plan.hpp` +- Create: `src/common/corelib/phi4_corelib_shape_plan.cpp` +- Create: `src/include/models/phi4/phi4_corelib_host.hpp` +- Create: `src/common/corelib/phi4_corelib_host.cpp` +- Create: `src/include/models/phi4/phi4_corelib_aie4.hpp` +- Create: `src/common/corelib/phi4_corelib_aie4.cpp` +- Create: `src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp` +- Create: `src/test/phi4_corelib_aie4/test_phi4_host.cpp` +- Create: `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` +- Modify: `src/test/phi4_corelib_aie4/fake_corelib.hpp` +- Modify: `src/test/phi4_corelib_aie4/fake_corelib.cpp` +- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` +- Modify: `src/common/corelib/corelib_sources.cmake` + +**Interfaces:** +- Consumes `Phi4GgufPackage`, `CorelibRuntime`, the exact ABI function table, and the fixed constants. +- Produces `DecodeEmbeddingRowsQ8`, `ConvertF32ToBf16`, `BuildShortRopeTables`, `Phi4ShapePlan`, and `phi4_corelib_aie4` from **Locked Interfaces**. +- `Phi4ShapePlan::Build(std::shared_ptr)` caches helper results for live rows 1 through 4096 and rejects any padded K/N change. +- The engine stores the shared GGUF package so lazy embedding spans remain valid for its full lifetime. + +- [ ] **Step 1: Write failing host conversion tests** + +Use known Q8_0 blocks, including negative signed codes and FP16 scales, and define: + +```cpp +TestLazyEmbeddingDecodesOnlyRequestedRows(); +TestLazyEmbeddingPreservesRequestOrderAndDuplicates(); +TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds(); +TestF32ToBf16UsesRoundToNearestEven(); +TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs(); +TestRopeTablesApplyShortFactorsAndAttentionFactor(); +TestRopeTablesHaveShape4096By48(); +``` + +Assert sentinel bytes in unrequested embedding rows are never read by using a guarded fixture mapping. For RoPE, compare position 4095 against a double-precision scalar reference; a float-only implementation must fail the tolerance. + +- [ ] **Step 2: Implement only the host utilities and run them** + +Q8_0 row decode is `value = fp16_scale * int8_code` for each 34-byte block. Reject malformed row lengths before decoding. Build inverse frequencies as: + +```cpp +inv_freq[i] = 1.0 / + (std::pow(freq_base, (2.0 * i) / 96.0) * short_factor[i]); +cos[p * 48 + i] = static_cast(std::cos(p * inv_freq[i]) * attn_factor); +sin[p * 48 + i] = static_cast(std::sin(p * inv_freq[i]) * attn_factor); +``` + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_host +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_host$" --output-on-failure +``` + +Expected: all seven host tests pass without loading a DLL. + +- [ ] **Step 3: Write failing shape-plan tests** + +The fake helper API must record every argument and return configurable padded rows. Define: + +```cpp +TestShapePlanQueriesRows1Through4096AtGroup64(); +TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions(); +TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96(); +TestShapePlanRejectsPaddedKOrNChanges(); +TestShapePlanRejectsRowsOutsideCachedRange(); +TestShapePlanFailureNamesHelperAndLogicalShape(); +``` + +- [ ] **Step 4: Implement the shape plan and run it** + +Cache transition vectors for query projection `[M,3072]x[3072,3072]`, KV projection `[M,3072]x[3072,1024]`, output projection `[M,3072]x[3072,3072]`, SSMLP `(M,3072,8192,64)`, RMSNorm `(M,3072)`, flat-MHA descriptor `(24,8,128,4096,96)`, and LM head `[1,3072]x[3072,200064]`. Every allocation uses the maximum helper-returned extent, never a hand-rounded M. + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_shape_plan +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_shape_plan$" --output-on-failure +``` + +Expected: all shape-plan tests pass and every fake helper observation uses group 64. + +- [ ] **Step 5: Extend the fake corelib and write failing engine-load tests** + +The fake must implement tensors with shape/dtype/storage, tensor windows retaining parent storage, Q8_0 weight creation records, RMSNorm weight records, stream dispatch records, injected pre-submit/post-submit/synchronize failures, and an in-flight flag. Define: + +```cpp +TestEngineCreatesOneStreamAndPersistentHelperSizedTensors(); +TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight(); +TestEveryProjectionUsesQ8RequantizedGroup64Threads0(); +TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate(); +TestQkvAndGateUpPointersMatchExactMappedSubranges(); +TestNormsAndEpsilonReachCorelibAsBf16(); +TestEmbeddingMappingOutlivesAllLazyRowReads(); +TestNoDeviceObjectExistsWhenPackageValidationFails(); +``` + +Require all 162 weight creates to happen in deterministic layer order; this makes accidental parallelization visible and protects the documented all-zero-output mitigation. + +- [ ] **Step 6: Implement engine construction and serial weight creation** + +Create initial RMSNorm weights from `blk.0.attn_norm.weight`. For each layer create Q/K/V/O matmul weights and one SSMLP weight whose `norm0` is `blk.i.ffn_norm.weight` and whose `norm1` is `blk.(i+1).attn_norm.weight`, except layer 31 uses `output_norm.weight`. Create the LM-head matmul from `token_embd.weight`. Every GGUF component type is Q8_0 and every descriptor group is 64. + +Allocate once: hidden/residual/skip, Q, K, attention output, one-row LM input, logits, FP32 cosine/sine tables, and 32 K plus 32 V caches `[8,4096,128]`. Upload RoPE once. Do not materialize the embedding table. + +- [ ] **Step 7: Write failing prefill/decode/sequencing tests** + +Define and invoke: + +```cpp +TestPrefillDecodesEmbeddingRowsAndAdvancesPosition(); +TestDecodeUsesOneRowAndAdvancesPosition(); +TestVProjectionWritesWindowAtPositionTimes128(); +TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream(); +TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket(); +TestForwardSynchronizesBeforeHostReadAndLmHeadRead(); +TestKVCachesRemainFixedAt8By4096By128(); +TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken(); +TestTotalDecodeWindowStopsAt4095(); +TestClearContextResetsLogicalPositionWithoutRecreatingWeights(); +TestCheckpointRestoreChangesOnlyLogicalPosition(); +TestPreSubmitFailureIsRecoverable(); +TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState(); +TestSynchronizeFailurePoisonsAndClearsState(); +TestPoisonedInstanceRejectsEveryLaterEntryPoint(); +TestCancellationBoundaryLeavesNoOutstandingFakeWork(); +``` + +The fake call log must prove there is no CPU/NPU2 call and no lossless `*_create_gguf` call. The V-window assertion uses shape `[8,4096-position,128]` and element offset `position * 128`. + +- [ ] **Step 8: Implement the minimal execution state machine** + +At the beginning of each model step, reject a poisoned engine and validate IDs/capacity. Decode only requested embedding rows to FP32, write and zero helper-required input/residual extents, dispatch RMSNorm in place, then for each layer dispatch Q, K, V-to-window, flat-MHA, O, and SSMLP on the same stream. Swap residual/skip handles only after queueing SSMLP. Synchronize before reading the final hidden row, write it to the one-row LM input, dispatch LM head, synchronize, and read logits as BF16 into the existing `buffer` expected by `Sampler`. + +Track whether any submit succeeded. On a pre-submit failure, leave `poisoned_ == false`. On any later exception, best-effort synchronize, set `poisoned_ = true`, clear logical position/checkpoint, and throw an error that includes the failed corelib call. `clear_context()` must not clear `poisoned_`; only destroying/recreating the model does. + +- [ ] **Step 9: Run GREEN and leak/order checks** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target ` + test_phi4_host test_phi4_shape_plan test_phi4_engine +ctest --test-dir src/build/phi4-corelib-tests -C Release ` + -R "^test_phi4_(host|shape_plan|engine)$" --output-on-failure +``` + +Expected: all tests pass; fake live-object count returns to zero after engine destruction; maximum simultaneous requantized creates is one; no submitted work remains when a request/cancellation test returns. + +- [ ] **Step 10: Refactor repeated create/dispatch/error wrappers and rerun Step 9** + +Keep model policy in `phi4_corelib_aie4.cpp`; keep GGUF parsing out of the engine and corelib calls out of the GGUF package/host helpers. + +- [ ] **Step 11: Commit review gate** + +```powershell +git add src/include/models/phi4/phi4_corelib_shape_plan.hpp ` + src/include/models/phi4/phi4_corelib_host.hpp ` + src/include/models/phi4/phi4_corelib_aie4.hpp ` + src/common/corelib/phi4_corelib_shape_plan.cpp ` + src/common/corelib/phi4_corelib_host.cpp ` + src/common/corelib/phi4_corelib_aie4.cpp ` + src/common/corelib/corelib_sources.cmake ` + src/test/phi4_corelib_aie4 +git commit -m "feat: add corelib-backed Phi-4 AIE4 engine" +``` + +Expected: engine/fake tests pass; no CLI/catalog route selects it yet. + +### Task 4: Explicit Phi-4 Frontend Routing and Request Lifecycle + +**Files:** +- Modify: `src/include/AutoModel/automodel.hpp` +- Modify: `src/common/AutoModel/automodel.cpp` +- Modify: `src/include/AutoModel/modeling_phi4.hpp` +- Modify: `src/common/AutoModel/modeling_phi4.cpp` +- Modify: `src/runner/runner.cpp` +- Modify: `src/server/rest_handler.cpp` +- Modify: `src/server/server.hpp` +- Modify: `src/server/server.cpp` +- Create: `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` +- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` + +**Interfaces:** +- Consumes the package/runtime/engine from Tasks 1–3. +- Produces `ModelRequestError`, `lm_uniform_input_t::requested_max_new_tokens`, and `AutoModel::uses_corelib_aie4()` from **Locked Interfaces**. +- `Phi4::load_model` recognizes exactly `corelib_aie4_gguf`; no backend field remains NPU2. + +- [ ] **Step 1: Write failing routing and initialization tests** + +Inject an engine factory under `FLM_CORELIB_TESTING` and define: + +```cpp +TestAbsentBackendStillBuildsQ4nxPhi4Npu(); +TestCorelibAie4GgufBuildsOnlyTheCorelibEngine(); +TestUnknownAndNonStringBackendAreErrors(); +TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders(); +TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation(); +TestMissingCorelibFailsOnlyWhenAie4ModelLoads(); +TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure(); +TestPreemptionIsRejectedForTheAie4Route(); +``` + +Split `_shared_load_model` so the test can assert the AIE4 branch initializes config/tokenizer/sampler state without constructing `npu_xclbin_manager`; leave `_shared_load_model` behavior unchanged for every legacy caller. + +- [ ] **Step 2: Implement explicit routing and tokenizer contract checks** + +`Phi4::load_model` must: + +1. resolve the backend string; +2. for AIE4, reject preemption and context outside `1..4096`; +3. parse `config.json`, `tokenizer.json`, and `tokenizer_config.json` and open the single GGUF; +4. call `ValidatePhi4Contract` before `CorelibRuntime::GetOrCreate` or any engine/device creation; +5. initialize the existing `Tokenizer`, chat template, sampler, and EOS list `{200020, 199999}` after proving `tokenizer.json` maps `<|end|>`/`<|endoftext|>` to those IDs, GGUF declares EOS 200020, `config.json` declares EOS 199999, and `tokenizer_config.json` disables automatic BOS; +6. lazily acquire runtime and construct `phi4_corelib_aie4`; +7. set `uses_corelib_aie4_` only after all construction succeeds. + +Use `Phi-4-mini-instruct.Q8_0.gguf` as the only accepted model filename. The feature-off branch throws `This binary was built without Phi-4 AIE4 corelib support`; it must not attempt Q4NX. + +- [ ] **Step 3: Write failing budget, cancellation, poison, and generation tests** + +```cpp +TestRenderedPromptPlusExplicitBudgetMayEqual4095(); +TestRenderedPromptPlusExplicitBudgetAbove4095Is400(); +TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow(); +TestCancellationBeforePrefillSubmitsNothing(); +TestCancellationBetweenDecodeStepsStopsWithCancelReason(); +TestCancellationReturnsOnlyAfterSynchronize(); +TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned(); +TestPoisonedModelReturns500UntilReload(); +TestEosSelfTerminatesWithoutAnExtraDecode(); +TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics(); +``` + +Use `ModelRequestError(400, false, ...)` for admission failures and `ModelRequestError(500, true, ...)` for inference failures that clear state. A poisoned refusal is 500 with `session_cleared == true` and names that unload/reload is required. + +- [ ] **Step 4: Implement AIE4-specific insert/generate behavior** + +Pass `requested_max_new_tokens` into `lm_uniform_input_t` from CLI `generate_limit`, `/api/generate:max_tokens`, `/api/chat:options.num_predict`, `/v1/chat/completions:max_tokens|max_completion_tokens`, and `/v1/completions:max_tokens`. Normalize absent or non-positive sentinel limits to an unbounded request, then cap generation to `4095 - rendered_prompt_tokens`; do not pass the legacy default 4096 through as an explicit AIE4 budget. + +Check cancellation immediately before prefill and before every `forward` call. Since each engine call synchronizes before returning, a cancellation observed between calls has no outstanding work. Preserve existing tokenization, chat-template rendering, sampling settings, and output streams. + +- [ ] **Step 5: Make server serialization exception-safe and complete** + +Keep the existing process-wide NPU queue. Add `/v1/completions` to `requires_npu_access`, replace duplicated release calls with a move-only completion guard owned by each dequeued request, and prove exactly one release on success, JSON parse failure, model error, cancellation, and unknown exception. Do not create a second AIE4-only queue. + +Map `ModelRequestError::http_code()` to HTTP 400 or 500 for non-streaming responses. For OpenAI streaming after headers/data started, emit one structured error event followed by `[DONE]`; before streaming starts, return the normal JSON error response. Include `session_cleared` in the error body. + +- [ ] **Step 6: Run frontend and compile-gate tests** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target ` + test_phi4_frontend phi4_frontend_compile_off phi4_frontend_compile_on +ctest --test-dir src/build/phi4-corelib-tests -C Release ` + -R "^test_phi4_frontend$" --output-on-failure +cmake --build --preset windows-default +cmake --build --preset windows-aie4 +``` + +Expected: routing/lifecycle tests pass; feature-off production TUs compile without `RYZENAI_CORELIB_INCLUDE_DIR`; both full builds succeed; legacy `phi4-mini-it:4b` still routes to Q4NX/NPU2. + +- [ ] **Step 7: Refactor shared request-finalization logic and rerun Step 6** + +Centralize only typed-error JSON construction and exactly-once queue release; retain each endpoint's existing response schema and stream formatter. Do not refactor unrelated server routes. + +- [ ] **Step 8: Commit review gate** + +```powershell +git add src/include/AutoModel/automodel.hpp ` + src/common/AutoModel/automodel.cpp ` + src/include/AutoModel/modeling_phi4.hpp ` + src/common/AutoModel/modeling_phi4.cpp ` + src/runner/runner.cpp src/server/rest_handler.cpp ` + src/server/server.hpp src/server/server.cpp ` + src/test/phi4_corelib_aie4/test_phi4_frontend.cpp ` + src/test/phi4_corelib_aie4/CMakeLists.txt +git commit -m "feat: route Phi-4 GGUF models through AIE4" +``` + +Expected: the explicit route works with synthetic/fake inputs, and no catalog model exposes it yet. + +### Task 5: Pinned Multi-Source Pull and Catalog Entry + +**Files:** +- Modify: `src/pull/download_model.hpp` +- Modify: `src/pull/download_model.cpp` +- Modify: `src/pull/model_downloader.hpp` +- Modify: `src/pull/model_downloader.cpp` +- Modify: `src/model_list.json` +- Modify: `src/model_info.json` +- Create: `src/test/phi4_corelib_aie4/test_model_downloader.cpp` +- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` + +**Interfaces:** +- Produces `DownloadRequest`, `download_file_atomic`, `ModelFileSource`, and `resolve_file_source` from **Locked Interfaces**. +- Existing catalog entries without `file_sources` retain byte-for-byte URL construction and SHA-256-for-LFS/Git-blob-SHA1-for-ordinary-file checks. +- New records may carry explicit lowercase `sha256`; when present it is authoritative regardless of LFS status. + +- [ ] **Step 1: Freeze exact remote metadata before editing the catalog** + +Query each immutable revision's Hugging Face tree API. For LFS files, require `lfs.size` and the 64-hex `lfs.oid` (the content SHA-256); for ordinary files, download the small immutable file and calculate SHA-256. Record exactly: + +```json +{ + "total_size": 4100140571, + "records": [ + {"path":"Phi-4-mini-instruct.Q8_0.gguf","size":4084611040,"sha256":"26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42"}, + {"path":"tokenizer.json","size":15524095,"sha256":"382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea"}, + {"path":"tokenizer_config.json","size":2932,"sha256":"9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db"}, + {"path":"config.json","size":2504,"sha256":"ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b"} + ] +} +``` + +Expected: API revisions equal the pinned commits; both LFS records match their `lfs` metadata; fresh downloads of the two small regular files reproduce the listed hashes. The later real `flm pull` independently hashes the complete GGUF and tokenizer payloads before promotion, so this step must not download and discard 4 GB. + +- [ ] **Step 2: Write failing URL/catalog/backward-compatibility tests** + +Define: + +```cpp +TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName(); +TestGgufUrlContainsUnslothRevisionAndFilename(); +TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename(); +TestExistingSingleSourceEntryKeepsItsCurrentUrl(); +TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail(); +TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile(); +TestModelIsReadyOnlyWhenAllFourFinalFilesValidate(); +TestPartFileNeverMakesModelReady(); +TestResumeAppendsToPartThenAtomicallyPromotes(); +TestWrongSizeOrHashNeverReplacesAValidFinalFile(); +TestInterruptedTransferKeepsPartForNextResume(); +TestSuccessfulForceDownloadAtomicallyReplacesFinalFile(); +``` + +Use a `file://` URL and a small deterministic payload for transfer tests. Pre-create the first half at `request.destination.string() + ".part"`; assert the final bytes and hash match and the `.part` file disappears only after success. + +- [ ] **Step 3: Run RED** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_model_downloader +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_model_downloader$" --output-on-failure +``` + +Expected: tests fail because `file_sources`, resumable temporary downloads, explicit SHA-256, and the catalog entry do not exist. + +- [ ] **Step 4: Implement per-file source selection without migrating existing entries** + +For an override, require exactly non-empty string `url` and 40-character hexadecimal `revision`, then produce: + +```text +{url}/resolve/{revision}/{percent-encoded filename}?download=true +``` + +For no override, execute the existing base URL/ModelScope logic unchanged. Reject `--modelscope` for this new tag with a message that pinned Hugging Face per-file sources are required; never silently swap repositories or revisions. + +Add this fixed catalog identity, then add its numeric `size` from the verified Step 1 output as described immediately below: + +```json +"phi4-mini-it-aie4": { + "4b": { + "name": "phi4-mini-it-aie4", + "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "default_context_length": 4096, + "max_prefill_len": 4096, + "details": { + "family": "phi4", + "think": false, + "think_toggleable": false, + "parameter_size": "4B", + "quantization_level": "Q8_0 -> AIE4 group-64", + "execution_backend": "corelib_aie4_gguf" + }, + "flm_min_version": "1.0.3", + "vlm": false, + "files": [ + "Phi-4-mini-instruct.Q8_0.gguf", + "tokenizer.json", + "tokenizer_config.json", + "config.json" + ], + "file_sources": { + "tokenizer.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "tokenizer_config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + } + }, + "footprint": 4.1 + } +} +``` + +Add a numeric `size` field to that object using the exact `total_size` emitted in Step 1; `TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName` must compare it with the sum of the four committed records and reject zero or disagreement. Add the four emitted `{path,size,sha256}` records under `phi4-mini-it-aie4:4b` in `model_info.json`. + +- [ ] **Step 5: Implement resume, verification, and atomic promotion** + +Always transfer to `request.destination.string() + ".part"`. If that path exists and is smaller than expected, open append mode and set `CURLOPT_RESUME_FROM_LARGE` to its byte length. If it is larger, delete only that file and restart. Require the completed size and selected hash before promotion. On Windows promote with `MoveFileExW(part, destination, MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)`; on non-Windows use same-directory `std::filesystem::rename`, whose replacement is atomic. A transfer interruption keeps `.part`; a size/hash mismatch deletes `.part`; no failure mutates an already-valid final file. + +Update both `pull_model` and `check_model` to consume the same record resolver and integrity function. After download, return success only when all four final files pass. + +- [ ] **Step 6: Run GREEN plus real pull/check smoke** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release --target test_model_downloader +ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_model_downloader$" --output-on-failure +src/build/Release/flm.exe pull phi4-mini-it-aie4:4b +src/build/Release/flm.exe check phi4-mini-it-aie4:4b +Get-ChildItem (Join-Path $env:USERPROFILE 'Documents/flm/models/phi4-mini-it-aie4') | + Select-Object -ExpandProperty Name +``` + +Expected: tests pass; pull/check succeed; directory output is exactly the four names in the `files` array and contains no `.part`, manifest, ONNX, or converted-weight file. + +- [ ] **Step 7: Refactor source/hash selection and rerun Step 6** + +Use one resolved per-file record for download, check, and ready-state decisions. Preserve the legacy URL/hash branch intact and do not generalize the catalog beyond optional `file_sources` and optional explicit `sha256`. + +- [ ] **Step 8: Commit review gate** + +```powershell +git add src/pull/download_model.hpp src/pull/download_model.cpp ` + src/pull/model_downloader.hpp src/pull/model_downloader.cpp ` + src/model_list.json src/model_info.json ` + src/test/phi4_corelib_aie4/test_model_downloader.cpp ` + src/test/phi4_corelib_aie4/CMakeLists.txt +git commit -m "feat: pull Phi-4 GGUF and tokenizer from pinned sources" +``` + +Expected: existing catalog tests remain unchanged and the new pinned multi-source pull is complete. + +### Task 6: Integrated Offline, Build, and Real-DLL Verification + +**Files:** +- Modify: `src/test/phi4_corelib_aie4/fake_corelib.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_corelib_api.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_model_downloader.cpp` +- Modify: `src/test/phi4_corelib_aie4/test_real_corelib.cpp` +- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` + +**Interfaces:** +- Consumes all production interfaces; introduces no product API. +- `test_real_corelib` is skipped only when no real DLL path is configured, never for ABI/symbol/self-test/device failures. + +- [ ] **Step 1: Add cross-component regression cases before changing production code** + +Add these tests using the complete fake and synthetic GGUF package: + +```cpp +TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates(); +TestNoManifestOnnxConvertedWeightOrCachePathIsOpened(); +TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib(); +TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing(); +TestAie4SelectionWithMissingDllFailsWithoutChangingBackend(); +TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback(); +TestTwoConcurrentAie4RequestsNeverOverlapDispatch(); +TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits(); +TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable(); +``` + +The all-zero guard injects nonzero deterministic fake logits at every cycle and asserts every returned vector has at least one nonzero BF16 element. It complements, but does not replace, the real-hardware ten-cycle gate. + +- [ ] **Step 2: Run the complete offline suite and fix only integration defects** + +```powershell +cmake --build src/build/phi4-corelib-tests --config Release +ctest --test-dir src/build/phi4-corelib-tests -C Release --output-on-failure +``` + +Expected: every offline test passes; `test_real_corelib` is reported skipped when `FLM_AIE4_CORELIB_PATH` is absent. Do not weaken assertions or add production behavior not required by the spec. If an integration failure requires a product edit, return to Tasks 1–5, amend the owning product commit, rerun that task's focused gate, and then restart this task; do not hide product fixes in the test commit. + +- [ ] **Step 3: Run exact feature-off and feature-on product builds** + +```powershell +Remove-Item Env:FLM_AIE4_CORELIB_PATH -ErrorAction SilentlyContinue +cmake --preset windows-default -S src +cmake --build --preset windows-default +src/build/Release/flm.exe version + +$env:RYZENAI_CORELIB_INCLUDE_DIR='C:/Users/chiz/work/ryzenai-corelib/include' +cmake --preset windows-aie4 -S src +cmake --build --preset windows-aie4 +src/build-aie4/Release/flm.exe version +src/build-aie4/Release/flm.exe list +``` + +Expected: both binaries start; the AIE4-enabled binary runs `version` and `list` without loading corelib; the default binary contains no corelib import. + +- [ ] **Step 4: Run real-DLL integration against the pinned installation** + +First prove the sibling checkout and header are the requested revision rather than trusting its current branch name: + +```powershell +git -C ../ryzenai-corelib rev-parse origin/main +git -C ../ryzenai-corelib show 3c35aebdefa3f0c2255668bab1be5648ece320f8:include/ryzenai/corelib.h | + Select-String 'RYZENAI_CORELIB_VERSION_(MAJOR|MINOR|PATCH)' +``` + +Expected: first command prints `3c35aebdefa3f0c2255668bab1be5648ece320f8`; version lines are 0, 3, 0. Build/install that exact commit in an isolated corelib worktree or use an existing installation whose provenance records that commit; do not alter the sibling working tree if it contains other work. + +Then run: + +```powershell +$env:FLM_AIE4_CORELIB_PATH='C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll' +ctest --test-dir src/build/phi4-corelib-tests -C Release ` + -R "^test_real_corelib$" --output-on-failure +``` + +Expected: PASS, runtime reports exactly 0.3.0, all 26 symbols resolve, dependency self-test succeeds, and device context is true. A skip is not acceptance when the variable is set. + +- [ ] **Step 5: Commit review gate** + +```powershell +git add src/test/phi4_corelib_aie4 +git commit -m "test: validate Phi-4 GGUF AIE4 integration" +``` + +Expected: this commit contains tests/fake changes only; the complete offline suite and real-DLL test are green. + +### Task 7: Developer Documentation and Real-AIE4 Acceptance + +**Files:** +- Create: `src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1` +- Modify: `docs/docs/models/phi.md` +- Modify: `docs/docs/benchmarks/phi4_results.md` + +**Interfaces:** +- Consumes the normal `flm.exe`, the installed four-file model directory, and `FLM_AIE4_CORELIB_PATH`. +- Produces a machine-readable acceptance JSON beside the build output and a human-readable checked-in record in `phi4_results.md`. + +- [ ] **Step 1: Write the acceptance runner before using the device** + +The script parameters are concrete: + +```powershell +param( + [string]$FlmExe = 'src/build-aie4/Release/flm.exe', + [string]$Model = 'phi4-mini-it-aie4:4b', + [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', + [string]$Output = 'src/build-aie4/phi4-gguf-aie4-acceptance.json', + [int]$Port = 52625 +) +``` + +It must fail nonzero unless it records: machine/CPU/NPU identity, Windows build, power mode, FastFlowLM commit, corelib commit and ABI, GGUF/tokenizer revisions, DLL SHA-256, four model-file hashes, exact commands, exit codes, response text/token IDs, load time, cold/warm TTFT, decode tokens/s, cancellation result, boundary results, and backend evidence from `show_profile()` naming `corelib_aie4_gguf` plus the loaded DLL path. + +- [ ] **Step 2: Run the required model acquisition commands on the AIE4 host** + +```powershell +$env:FLM_AIE4_CORELIB_PATH='C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll' +src/build-aie4/Release/flm.exe pull phi4-mini-it-aie4:4b +src/build-aie4/Release/flm.exe check phi4-mini-it-aie4:4b +``` + +Expected: both succeed; all four files validate; the model directory contains only `Phi-4-mini-instruct.Q8_0.gguf`, `tokenizer.json`, `tokenizer_config.json`, and `config.json`. + +- [ ] **Step 3: Run CLI semantic and repeated-load acceptance** + +Use the script to run `flm run phi4-mini-it-aie4:4b` with `What is 2+2?` and `What does AMD do?`, then at least ten prompts in one loaded process. Require the first answer to contain the correct value 4 and self-terminate; require the second to be relevant to AMD's semiconductor/computing business and self-terminate. + +Run at least ten complete process/model load-and-generate cycles. Fail if a response is empty, every emitted token ID is zero, the profile omits the exact backend/DLL, or any cycle exits nonzero. + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass ` + -File src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 ` + -FlmExe src/build-aie4/Release/flm.exe ` + -CorelibDll $env:FLM_AIE4_CORELIB_PATH +``` + +Expected: script exits 0 and writes the complete JSON record. + +- [ ] **Step 4: Run REST, cancellation, and boundary acceptance** + +The same script starts: + +```powershell +src/build-aie4/Release/flm.exe serve phi4-mini-it-aie4:4b --port 52625 +``` + +It must issue both `POST /api/chat` and `POST /v1/chat/completions`, test streaming and non-streaming responses, cancel an active generation, then submit another request to prove the queue/model remains usable when cancellation occurred at a synchronized boundary. Test total rendered-prompt-plus-generation budgets at 4095 (accepted/capped) and 4096 (HTTP 400 before submission). Inject no fallback configuration; backend evidence must still name corelib/AIE4. + +Expected: both APIs return relevant nonempty text, cancellation completes without a process exit, the next request succeeds, boundary statuses match, and no CPU/NPU2 backend appears in logs/profile. + +- [ ] **Step 5: Record descriptive performance and documentation** + +Update `docs/docs/models/phi.md` with the exact tag, four-source pinning, Windows developer build flags, `FLM_AIE4_CORELIB_PATH` lookup/fallback, dependency-directory requirement, 4095 usable generation window, no fallback, no packaged runtime, and an explicit statement that Q8_0 is lossily requantized to group 64. + +Append the acceptance JSON's exact machine, power mode, commits/revisions, commands, pass/fail outcomes, load time, cold/warm TTFT, and decode tokens/s to `docs/docs/benchmarks/phi4_results.md`. Label performance descriptive and do not invent a pass threshold. + +- [ ] **Step 6: Run final repository verification** + +```powershell +ctest --test-dir src/build/phi4-corelib-tests -C Release --output-on-failure +cmake --build --preset windows-default +cmake --build --preset windows-aie4 +src/build-aie4/Release/flm.exe check phi4-mini-it-aie4:4b +Select-String -Path docs/docs/models/phi.md,docs/docs/benchmarks/phi4_results.md ` + -Pattern '3c35aebdefa3f0c2255668bab1be5648ece320f8','0.3.0','78eb92a46fc37e6b524df991ed9aca9bc6aa7b80','cfbefacb99257ffa30c83adab238a50856ac3083','lossy','group 64' +Get-ChildItem (Join-Path $env:USERPROFILE 'Documents/flm/models/phi4-mini-it-aie4') | + Where-Object { $_.Name -match '(manifest|onnx|converted|packed)' } +``` + +Expected: all configured tests pass (no real-DLL skip on the AIE4 host); both builds succeed; check succeeds; every required documentation string is found; the final `Get-ChildItem` command emits nothing. + +- [ ] **Step 7: Commit final review gate** + +```powershell +git add src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 ` + docs/docs/models/phi.md docs/docs/benchmarks/phi4_results.md +git commit -m "docs: document developer setup and hardware results" +``` + +Expected: the seventh commit contains the reproducible runner and actual observed record, with no unfilled values. + +## Final Review Checklist + +- [ ] `git diff origin/main...HEAD --name-only` contains only the approved product/test/doc files plus this plan and its design spec, and no installer/package files. +- [ ] After the planning commits, `git log --oneline origin/main..HEAD` shows the seven implementation commits in this order: + +```text +docs: document developer setup and hardware results +test: validate Phi-4 GGUF AIE4 integration +feat: pull Phi-4 GGUF and tokenizer from pinned sources +feat: route Phi-4 GGUF models through AIE4 +feat: add corelib-backed Phi-4 AIE4 engine +feat: add validated Phi-4 Q8_0 GGUF reader +build: add optional dynamic corelib 0.3.0 runtime +``` + +- [ ] Search the diff for `manifest`, `.onnx`, `weights_create_gguf(`, `ryzenai_corelib.lib`, and runtime-copy/install additions; only explanatory negative assertions may match. +- [ ] Confirm all 129 matmul and 32 SSMLP creations use the requantized Q8_0 group-64 entry points serially, and the single RMSNorm uses `weights_create_scale`. +- [ ] Confirm invalid GGUF/config/tokenizer input produces zero stream/tensor/weight creates. +- [ ] Confirm default build and legacy `phi4-mini-it:4b` behavior remain unchanged. +- [ ] Confirm missing corelib does not prevent process startup or ordinary-model execution. +- [ ] Confirm an AIE4 request never reaches Q4NX, `phi4_npu`, CPU fallback, or an alternate model file. +- [ ] Confirm cancellation, 4095/4096 bounds, and post-submit poison semantics in both fake tests and REST acceptance. +- [ ] Confirm the real-DLL integration and all real-AIE4 acceptance cases passed; skipped hardware tests do not satisfy completion. +- [ ] Confirm documentation contains actual measured values and exact provenance, not an empty table or promised follow-up. diff --git a/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md b/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md index 2cb3b9c5..0bfe5fb1 100644 --- a/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md +++ b/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md @@ -186,8 +186,9 @@ Validation occurs before device weight creation wherever possible. The package m - every required norm present in the supported floating type; - `output.weight` is absent and `token_embd.weight` is used for both embedding and LM head, as in the pinned model; - tokenizer vocabulary size agrees with GGUF; -- EOS IDs include 200020 and 199999; -- BOS behavior agrees; +- `tokenizer.json` maps `<|end|>` to 200020 and `<|endoftext|>` to 199999; +- GGUF identifies 200020 as its EOS token and `config.json` identifies 199999, so the frontend stop set is their explicit union `{200020, 199999}`; +- `tokenizer_config.json` has `add_bos_token == false` and the frontend does not prepend `config.json`'s BOS token; - the chat template contains the required Phi-4 user, end, and assistant markers. The error names the model field or tensor, its actual value, and the expected value. The loader does not repair, reinterpret, or silently accept a mismatch. From d1b0a05abe5473e23fdaea4e4cd24d0eed38f8af Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 02:31:23 -0700 Subject: [PATCH 03/37] build: add optional dynamic corelib 0.3.0 runtime --- src/CMakeLists.txt | 19 ++ src/CMakePresets.json | 19 ++ src/common/corelib/corelib_api.cpp | 164 ++++++++++++ src/common/corelib/corelib_runtime.cpp | 68 +++++ src/common/corelib/corelib_sources.cmake | 3 + src/include/corelib/corelib_api.hpp | 98 +++++++ src/include/corelib/corelib_object.hpp | 64 +++++ src/include/corelib/corelib_runtime.hpp | 30 +++ src/src/main.cpp | 6 + src/test/phi4_corelib_aie4/CMakeLists.txt | 46 ++++ src/test/phi4_corelib_aie4/compile_guard.cpp | 5 + src/test/phi4_corelib_aie4/fake_corelib.cpp | 105 ++++++++ src/test/phi4_corelib_aie4/fake_corelib.hpp | 39 +++ .../phi4_corelib_aie4/test_corelib_api.cpp | 252 ++++++++++++++++++ .../phi4_corelib_aie4/test_real_corelib.cpp | 33 +++ src/test/phi4_corelib_aie4/test_support.hpp | 43 +++ 16 files changed, 994 insertions(+) create mode 100644 src/common/corelib/corelib_api.cpp create mode 100644 src/common/corelib/corelib_runtime.cpp create mode 100644 src/common/corelib/corelib_sources.cmake create mode 100644 src/include/corelib/corelib_api.hpp create mode 100644 src/include/corelib/corelib_object.hpp create mode 100644 src/include/corelib/corelib_runtime.hpp create mode 100644 src/test/phi4_corelib_aie4/CMakeLists.txt create mode 100644 src/test/phi4_corelib_aie4/compile_guard.cpp create mode 100644 src/test/phi4_corelib_aie4/fake_corelib.cpp create mode 100644 src/test/phi4_corelib_aie4/fake_corelib.hpp create mode 100644 src/test/phi4_corelib_aie4/test_corelib_api.cpp create mode 100644 src/test/phi4_corelib_aie4/test_real_corelib.cpp create mode 100644 src/test/phi4_corelib_aie4/test_support.hpp diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index dd4d33fe..12b787d0 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -48,6 +48,15 @@ set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) # ——————————————————————————————————————————————— option(FLM_USE_HRX "Use the HRX amdxdna NPU runtime instead of XRT (0=XRT default, 1=HRX)" OFF) option(FLM_PORTABLE_BUILD "Build portable distribution with bundled runtime libraries" OFF) +option(FLM_ENABLE_CORELIB_AIE4 + "Enable Phi-4 Q8_0 GGUF execution through ryzenai-corelib" OFF) + +if(FLM_ENABLE_CORELIB_AIE4) + if(NOT WIN32) + message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows") + endif() + find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) +endif() if(FLM_USE_HRX) set(FLM_RUNTIME_NAME "hrx") @@ -237,6 +246,7 @@ add_subdirectory(${CMAKE_SOURCE_DIR}/../third_party/tokenizers-cpp # ——————————————————————————————————————————————— file(GLOB SOURCES "src/*.cpp" "runner/*.cpp" "common/*.cpp" "common/*/*.cpp" "server/*.cpp" "pull/*.cpp" ) file(GLOB HEADERS "include/*.hpp" "runner/*.hpp" "common/*.hpp" "common/*/*.hpp" "server/*.hpp" "pull/*.hpp") +list(FILTER SOURCES EXCLUDE REGEX ".*/common/corelib/.*\\.cpp$") # Exclude files that depend on missing libraries for Linux if(NOT WIN32) @@ -269,6 +279,15 @@ endif() add_executable(flm ${SOURCES} ${HEADERS}) +if(FLM_ENABLE_CORELIB_AIE4) + include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake") + add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES}) + target_include_directories(flm_corelib_aie4 PUBLIC + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") + target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1) + target_link_libraries(flm PRIVATE flm_corelib_aie4) +endif() + if(WIN32) if(VCPKG_TOOLCHAIN) # A vcpkg toolchain is active (e.g. the rocm-npu-staging dev.py build or diff --git a/src/CMakePresets.json b/src/CMakePresets.json index 39a07cdd..fbb1b1cb 100644 --- a/src/CMakePresets.json +++ b/src/CMakePresets.json @@ -53,6 +53,17 @@ "CMAKE_BUILD_TYPE": "Release" } }, + { + "name": "windows-aie4", + "displayName": "Windows AIE4", + "description": "Windows build with optional dynamically loaded ryzenai-corelib support", + "inherits": "windows-default", + "binaryDir": "${sourceDir}/build-aie4", + "cacheVariables": { + "FLM_ENABLE_CORELIB_AIE4": "ON", + "RYZENAI_CORELIB_INCLUDE_DIR": "$env{RYZENAI_CORELIB_INCLUDE_DIR}" + } + }, { "name": "windows-vs18", "displayName": "Windows VS18", @@ -85,6 +96,14 @@ "configurePreset": "windows-default", "configuration": "Release", "jobs": 4 + }, + { + "name": "windows-aie4", + "displayName": "Windows AIE4 Build", + "description": "Build the optional dynamically loaded AIE4 runtime in Release", + "configurePreset": "windows-aie4", + "configuration": "Release", + "jobs": 4 } ] } diff --git a/src/common/corelib/corelib_api.cpp b/src/common/corelib/corelib_api.cpp new file mode 100644 index 00000000..d2473b7f --- /dev/null +++ b/src/common/corelib/corelib_api.cpp @@ -0,0 +1,164 @@ +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include +#include + +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#endif + +namespace flm::corelib { +namespace { +constexpr CorelibVersion kRequiredVersion{0, 3, 0}; + +std::string VersionText(CorelibVersion version) { + return std::to_string(version.major) + "." + std::to_string(version.minor) + + "." + std::to_string(version.patch); +} + +std::string ErrorText(std::string_view call, + std::string_view status, + std::string_view detail) { + std::string result(call); + result += " failed: "; + result += status; + if (!detail.empty()) { + result += ": "; + result += detail; + } + return result; +} + +bool HasDllExtension(const std::filesystem::path& path) { + std::string extension = path.extension().string(); + std::transform(extension.begin(), extension.end(), extension.begin(), + [](unsigned char value) { + return static_cast(std::tolower(value)); + }); + return extension == ".dll"; +} +} // namespace + +CorelibError::CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text) + : std::runtime_error(ErrorText(call, status_text, detail)), + status_(status), + call_(std::move(call)), + detail_(std::move(detail)) {} + +ryzenai_corelib_status CorelibError::status() const noexcept { return status_; } +const std::string& CorelibError::call() const noexcept { return call_; } +const std::string& CorelibError::detail() const noexcept { return detail_; } + +CorelibApi::CorelibApi(Resolver resolver) : resolver_(std::move(resolver)) { + void* version_symbol = resolver_("ryzenai_corelib_get_version"); + if (!version_symbol) { + throw std::runtime_error("missing corelib symbol: ryzenai_corelib_get_version"); + } + functions_.get_version = + reinterpret_cast(version_symbol); + functions_.get_version(&runtime_version_.major, &runtime_version_.minor, + &runtime_version_.patch); + if (runtime_version_.major != kRequiredVersion.major || + runtime_version_.minor != kRequiredVersion.minor || + runtime_version_.patch != kRequiredVersion.patch) { + throw std::runtime_error("corelib ABI mismatch: runtime " + + VersionText(runtime_version_) + ", required " + + VersionText(kRequiredVersion)); + } + +#define FLM_RESOLVE_CORELIB_FUNCTION(member, symbol) \ + if constexpr (std::string_view(#symbol) != \ + std::string_view("ryzenai_corelib_get_version")) { \ + void* address = resolver_(#symbol); \ + if (!address) throw std::runtime_error("missing corelib symbol: " #symbol); \ + functions_.member = reinterpret_cast(address); \ + } + FLM_CORELIB_FUNCTIONS(FLM_RESOLVE_CORELIB_FUNCTION) +#undef FLM_RESOLVE_CORELIB_FUNCTION +} + +std::shared_ptr CorelibApi::ResolveForTest(Resolver resolver) { + if (!resolver) throw std::invalid_argument("corelib resolver is empty"); + return std::shared_ptr(new CorelibApi(std::move(resolver))); +} + +std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { +#ifndef _WIN32 + (void)dll; + throw std::runtime_error("ryzenai-corelib loading currently requires Windows"); +#else + const std::filesystem::path absolute_dll = std::filesystem::absolute(dll); + HMODULE raw_module = LoadLibraryExW( + absolute_dll.c_str(), nullptr, + LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); + if (!raw_module) { + throw std::runtime_error("failed to load corelib DLL '" + + absolute_dll.string() + "' (Windows error " + + std::to_string(GetLastError()) + ")"); + } + auto module = std::shared_ptr(raw_module, [](void* handle) { + FreeLibrary(static_cast(handle)); + }); + Resolver resolver = [module](std::string_view name) -> void* { + const std::string terminated(name); + return reinterpret_cast( + GetProcAddress(static_cast(module.get()), terminated.c_str())); + }; + return ResolveForTest(std::move(resolver)); +#endif +} + +std::filesystem::path CorelibApi::ResolveLibraryPath( + const std::filesystem::path& executable_dir) { + const char* configured = std::getenv("FLM_AIE4_CORELIB_PATH"); + if (configured && *configured) { + const std::filesystem::path path(configured); + if (!path.is_absolute()) { + throw std::runtime_error( + "FLM_AIE4_CORELIB_PATH must be an absolute .dll path"); + } + if (!path.has_filename() || !HasDllExtension(path)) { + throw std::runtime_error( + "FLM_AIE4_CORELIB_PATH must name an absolute .dll file"); + } + return path; + } + return std::filesystem::absolute(executable_dir / "aie4" / + "ryzenai_corelib.dll"); +} + +const CorelibFunctions& CorelibApi::functions() const noexcept { return functions_; } +CorelibVersion CorelibApi::runtime_version() const noexcept { return runtime_version_; } + +void CorelibApi::Check(ryzenai_corelib_status status, + std::string_view call) const { + if (status == ryzenai_corelib_status_success) return; + const char* detail_pointer = functions_.get_last_error_message(); + const std::string detail = detail_pointer ? detail_pointer : ""; + const char* status_pointer = functions_.status_to_string(status); + const std::string status_text = status_pointer ? status_pointer : "unknown"; + throw CorelibError(status, std::string(call), detail, status_text); +} + +void CorelibApi::RegisterObject() const noexcept { ++live_object_count_; } + +void CorelibApi::Release(void* object) const noexcept { + if (!object) return; + functions_.object_release(object); + --live_object_count_; +} + +std::size_t CorelibApi::live_object_count() const noexcept { + return live_object_count_.load(); +} + +} // namespace flm::corelib diff --git a/src/common/corelib/corelib_runtime.cpp b/src/common/corelib/corelib_runtime.cpp new file mode 100644 index 00000000..48c08ad2 --- /dev/null +++ b/src/common/corelib/corelib_runtime.cpp @@ -0,0 +1,68 @@ +#include "corelib/corelib_runtime.hpp" + +#include +#include + +namespace flm::corelib { +namespace { +std::mutex process_mutex; +std::shared_ptr process_runtime; +} + +CorelibRuntime::CorelibRuntime(std::shared_ptr api) + : api_(std::move(api)) {} + +std::shared_ptr CorelibRuntime::CreateReady( + std::shared_ptr api) { + if (!api) throw std::invalid_argument("corelib API is null"); + api->Check(api->functions().selftest_dependencies(), + "ryzenai_corelib_selftest_dependencies"); + if (!api->functions().has_device_context()) { + throw std::runtime_error("corelib has no AIE4 device context"); + } + return std::shared_ptr(new CorelibRuntime(std::move(api))); +} + +std::shared_ptr CorelibRuntime::GetOrCreate( + const std::filesystem::path& executable_dir) { + std::lock_guard lock(process_mutex); + if (!process_runtime) { + auto api = CorelibApi::Load(CorelibApi::ResolveLibraryPath(executable_dir)); + process_runtime = CreateReady(std::move(api)); + } + return process_runtime; +} + +std::shared_ptr CorelibRuntime::CreateForTest( + std::shared_ptr api) { + auto runtime = CreateReady(std::move(api)); + std::lock_guard lock(process_mutex); + if (process_runtime) { + throw std::runtime_error("corelib runtime already exists"); + } + process_runtime = runtime; + return runtime; +} + +void CorelibRuntime::ShutdownProcess() { + std::lock_guard process_lock(process_mutex); + if (!process_runtime) return; + + std::lock_guard execution_lock(process_runtime->execution_mutex_); + if (process_runtime->api_->live_object_count() != 0) { + throw std::runtime_error("cannot shut down with live corelib objects"); + } + process_runtime->api_->functions().cleanup(); + process_runtime->api_.reset(); + process_runtime.reset(); +} + +std::unique_lock CorelibRuntime::AcquireExecution() { + return std::unique_lock(execution_mutex_); +} + +const std::shared_ptr& CorelibRuntime::api() const noexcept { + return api_; +} + +} // namespace flm::corelib diff --git a/src/common/corelib/corelib_sources.cmake b/src/common/corelib/corelib_sources.cmake new file mode 100644 index 00000000..a2ba02f1 --- /dev/null +++ b/src/common/corelib/corelib_sources.cmake @@ -0,0 +1,3 @@ +set(FLM_CORELIB_AIE4_SOURCES + "${CMAKE_CURRENT_LIST_DIR}/corelib_api.cpp" + "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp") diff --git a/src/include/corelib/corelib_api.hpp b/src/include/corelib/corelib_api.hpp new file mode 100644 index 00000000..d68ae9fb --- /dev/null +++ b/src/include/corelib/corelib_api.hpp @@ -0,0 +1,98 @@ +#pragma once + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define FLM_CORELIB_FUNCTIONS(X) \ + X(get_version, ryzenai_corelib_get_version) \ + X(status_to_string, ryzenai_corelib_status_to_string) \ + X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ + X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ + X(has_device_context, ryzenai_corelib_has_device_context) \ + X(object_release, ryzenai_corelib_object_release) \ + X(create_stream, ryzenai_corelib_create_stream) \ + X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ + X(create_device_tensor, ryzenai_corelib_create_device_tensor) \ + X(create_tensor_window, ryzenai_corelib_create_tensor_window) \ + X(tensor_write, ryzenai_corelib_tensor_write) \ + X(tensor_read, ryzenai_corelib_tensor_read) \ + X(tensor_get_byte_size, ryzenai_corelib_tensor_get_byte_size) \ + X(tensor_get_data_type, ryzenai_corelib_tensor_get_data_type) \ + X(matmul_pad_shape, ryzenai_corelib_matmul_bf16_pad_shape) \ + X(matmul_weights_create_gguf_requantized, \ + ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized) \ + X(matmul, ryzenai_corelib_matmul_bf16) \ + X(ssmlp_pad_rows, ryzenai_corelib_ssmlp_bf16_pad_rows) \ + X(ssmlp_weights_create_gguf_requantized, \ + ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) \ + X(ssmlp, ryzenai_corelib_ssmlp_bf16) \ + X(rmsnorm_weights_create_scale, ryzenai_corelib_rmsnorm_bf16_weights_create_scale) \ + X(rmsnorm_pad_rows, ryzenai_corelib_rmsnorm_bf16_pad_rows) \ + X(rmsnorm, ryzenai_corelib_rmsnorm_bf16) \ + X(flat_mha_pad_rows, ryzenai_corelib_flat_mha_bf16_pad_rows) \ + X(flat_mha, ryzenai_corelib_flat_mha_bf16) \ + X(cleanup, ryzenai_corelib_cleanup) + +namespace flm::corelib { + +struct CorelibVersion { + std::uint32_t major; + std::uint32_t minor; + std::uint32_t patch; +}; + +class CorelibError final : public std::runtime_error { +public: + CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text); + ryzenai_corelib_status status() const noexcept; + const std::string& call() const noexcept; + const std::string& detail() const noexcept; + +private: + ryzenai_corelib_status status_; + std::string call_; + std::string detail_; +}; + +struct CorelibFunctions { +#define FLM_DECLARE_CORELIB_FUNCTION(member, symbol) decltype(&::symbol) member{}; + FLM_CORELIB_FUNCTIONS(FLM_DECLARE_CORELIB_FUNCTION) +#undef FLM_DECLARE_CORELIB_FUNCTION +}; + +class CorelibApi final { +public: + using Resolver = std::function; + static std::shared_ptr Load(const std::filesystem::path& dll); + static std::shared_ptr ResolveForTest(Resolver resolver); + static std::filesystem::path ResolveLibraryPath( + const std::filesystem::path& executable_dir); + const CorelibFunctions& functions() const noexcept; + CorelibVersion runtime_version() const noexcept; + void Check(ryzenai_corelib_status status, std::string_view call) const; + void RegisterObject() const noexcept; + void Release(void* object) const noexcept; + std::size_t live_object_count() const noexcept; + +private: + explicit CorelibApi(Resolver resolver); + + Resolver resolver_; + CorelibFunctions functions_{}; + CorelibVersion runtime_version_{}; + mutable std::atomic live_object_count_{0}; +}; + +} // namespace flm::corelib diff --git a/src/include/corelib/corelib_object.hpp b/src/include/corelib/corelib_object.hpp new file mode 100644 index 00000000..2dd106be --- /dev/null +++ b/src/include/corelib/corelib_object.hpp @@ -0,0 +1,64 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include + +namespace flm::corelib { + +struct StreamTag {}; +struct TensorTag {}; +struct TensorWindowTag {}; +struct MatMulWeightsTag {}; +struct SsMlpWeightsTag {}; +struct RmsNormWeightsTag {}; + +template +class UniqueObject final { +public: + UniqueObject() noexcept = default; + + UniqueObject(std::shared_ptr api, void* object) noexcept + : api_(std::move(api)), object_(object) { + if (object_) api_->RegisterObject(); + } + + ~UniqueObject() { reset(); } + + UniqueObject(const UniqueObject&) = delete; + UniqueObject& operator=(const UniqueObject&) = delete; + + UniqueObject(UniqueObject&& other) noexcept + : api_(std::move(other.api_)), object_(std::exchange(other.object_, nullptr)) {} + + UniqueObject& operator=(UniqueObject&& other) noexcept { + if (this != &other) { + reset(); + api_ = std::move(other.api_); + object_ = std::exchange(other.object_, nullptr); + } + return *this; + } + + void reset() noexcept { + if (object_) api_->Release(std::exchange(object_, nullptr)); + api_.reset(); + } + + void* get() const noexcept { return object_; } + explicit operator bool() const noexcept { return object_ != nullptr; } + +private: + std::shared_ptr api_; + void* object_{}; +}; + +using UniqueStream = UniqueObject; +using UniqueTensor = UniqueObject; +using UniqueTensorWindow = UniqueObject; +using UniqueMatMulWeights = UniqueObject; +using UniqueSsMlpWeights = UniqueObject; +using UniqueRmsNormWeights = UniqueObject; + +} // namespace flm::corelib diff --git a/src/include/corelib/corelib_runtime.hpp b/src/include/corelib/corelib_runtime.hpp new file mode 100644 index 00000000..e22ce1b4 --- /dev/null +++ b/src/include/corelib/corelib_runtime.hpp @@ -0,0 +1,30 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include + +namespace flm::corelib { + +class CorelibRuntime final { +public: + static std::shared_ptr GetOrCreate( + const std::filesystem::path& executable_dir); + static std::shared_ptr CreateForTest( + std::shared_ptr api); + static void ShutdownProcess(); + std::unique_lock AcquireExecution(); + const std::shared_ptr& api() const noexcept; + +private: + explicit CorelibRuntime(std::shared_ptr api); + static std::shared_ptr CreateReady( + std::shared_ptr api); + + std::shared_ptr api_; + std::mutex execution_mutex_; +}; + +} // namespace flm::corelib diff --git a/src/src/main.cpp b/src/src/main.cpp index 446b907b..983959d1 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -34,6 +34,9 @@ #include "utils/vm_args.hpp" #include #include "benchmarking.hpp" +#ifdef FLM_ENABLE_CORELIB_AIE4 +#include "corelib/corelib_runtime.hpp" +#endif #ifndef _WIN32 #include @@ -716,6 +719,9 @@ int main(int argc, char* argv[]) { return 1; } // Return 0 if the command is valid +#ifdef FLM_ENABLE_CORELIB_AIE4 + flm::corelib::CorelibRuntime::ShutdownProcess(); +#endif return 0; } catch (const std::exception& e) { // If an error occurs, this will be used to show the error diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt new file mode 100644 index 00000000..a996733b --- /dev/null +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -0,0 +1,46 @@ +cmake_minimum_required(VERSION 3.22) +project(phi4_corelib_aie4_tests LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(NOT WIN32) + message(FATAL_ERROR "Phi-4 corelib AIE4 tests currently require Windows") +endif() + +find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) +set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") +set(CORELIB_SOURCES + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/corelib_runtime.cpp") + +add_executable(test_corelib_api + test_corelib_api.cpp fake_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_corelib_api PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_corelib_api PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_real_corelib test_real_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_real_corelib PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) + +# Feature-off frontend code must compile without the pinned corelib include path. +add_library(phi4_frontend_compile_off OBJECT compile_guard.cpp) +target_include_directories(phi4_frontend_compile_off PRIVATE "${FLM_SOURCE_DIR}/include") + +# Feature-on frontend code sees both the feature define and pinned ABI headers. +add_library(phi4_frontend_compile_on OBJECT compile_guard.cpp) +target_include_directories(phi4_frontend_compile_on PRIVATE + "${FLM_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(phi4_frontend_compile_on PRIVATE + FLM_ENABLE_CORELIB_AIE4=1 RYZENAI_CORELIB_STATIC=1) + +include(CTest) +add_test(NAME test_corelib_api COMMAND test_corelib_api) +add_test(NAME test_real_corelib COMMAND test_real_corelib) +set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/compile_guard.cpp b/src/test/phi4_corelib_aie4/compile_guard.cpp new file mode 100644 index 00000000..da963ff5 --- /dev/null +++ b/src/test/phi4_corelib_aie4/compile_guard.cpp @@ -0,0 +1,5 @@ +#ifdef FLM_ENABLE_CORELIB_AIE4 +#include "corelib/corelib_runtime.hpp" +#endif + +void Phi4CorelibCompileGuard() {} diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp new file mode 100644 index 00000000..d6232647 --- /dev/null +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -0,0 +1,105 @@ +#include "fake_corelib.hpp" + +#include +#include + +namespace { +fake_corelib::State state; +thread_local std::string current_detail; + +void GetVersion(std::uint32_t* major, std::uint32_t* minor, std::uint32_t* patch) { + if (major) *major = state.version.major; + if (minor) *minor = state.version.minor; + if (patch) *patch = state.version.patch; +} + +const char* StatusToString(ryzenai_corelib_status) { + current_detail = "detail overwritten by status_to_string"; + return state.status_text.c_str(); +} + +const char* GetLastErrorMessage() { + current_detail = state.detail; + return current_detail.c_str(); +} + +ryzenai_corelib_status SelftestDependencies() { return state.selftest_status; } +bool HasDeviceContext() { return state.has_device_context; } +void ObjectRelease(void* object) { + if (object) { + delete static_cast(object); + --state.live_objects; + ++state.releases; + state.lifetime_events.emplace_back("release"); + } +} +void Cleanup() { + ++state.cleanup_calls; + state.lifetime_events.emplace_back("cleanup"); +} +void UncalledSymbol() {} + +void* FunctionFor(std::string_view name) { + if (name == "ryzenai_corelib_get_version") return reinterpret_cast(&GetVersion); + if (name == "ryzenai_corelib_status_to_string") return reinterpret_cast(&StatusToString); + if (name == "ryzenai_corelib_get_last_error_message") return reinterpret_cast(&GetLastErrorMessage); + if (name == "ryzenai_corelib_selftest_dependencies") return reinterpret_cast(&SelftestDependencies); + if (name == "ryzenai_corelib_has_device_context") return reinterpret_cast(&HasDeviceContext); + if (name == "ryzenai_corelib_object_release") return reinterpret_cast(&ObjectRelease); + if (name == "ryzenai_corelib_cleanup") return reinterpret_cast(&Cleanup); +#define FLM_FAKE_CORELIB_SYMBOL(member, symbol) \ + if (name == #symbol) return reinterpret_cast(&UncalledSymbol); + FLM_CORELIB_FUNCTIONS(FLM_FAKE_CORELIB_SYMBOL) +#undef FLM_FAKE_CORELIB_SYMBOL + return nullptr; +} +} // namespace + +namespace fake_corelib { + +State& GetState() { return state; } + +void Reset() { + state.version = {0, 3, 0}; + state.selftest_status = ryzenai_corelib_status_success; + state.has_device_context = true; + state.detail.clear(); + state.status_text = "success"; + state.missing_symbol.clear(); + state.resolution_order.clear(); + state.resolution_counts.clear(); + state.lifetime_events.clear(); + state.live_objects = 0; + state.releases = 0; + state.cleanup_calls = 0; + state.active_leases = 0; + state.maximum_active_leases = 0; +} + +flm::corelib::CorelibApi::Resolver Resolver() { + return [](std::string_view name) -> void* { + state.resolution_order.emplace_back(name); + ++state.resolution_counts[std::string(name)]; + if (name == state.missing_symbol) return nullptr; + return FunctionFor(name); + }; +} + +void* MakeObject() { + ++state.live_objects; + return new int(1); +} + +void EnterLease() { + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} +} + +void LeaveLease() { + --state.active_leases; + state.lifetime_events.emplace_back("lease_leave"); +} + +} // namespace fake_corelib diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp new file mode 100644 index 00000000..341880c3 --- /dev/null +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -0,0 +1,39 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include +#include +#include +#include + +namespace fake_corelib { + +struct State { + flm::corelib::CorelibVersion version{0, 3, 0}; + ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; + bool has_device_context{true}; + std::string detail; + std::string status_text{"success"}; + std::string missing_symbol; + std::vector resolution_order; + std::unordered_map resolution_counts; + std::vector lifetime_events; + std::atomic live_objects{0}; + std::atomic releases{0}; + std::atomic cleanup_calls{0}; + std::atomic active_leases{0}; + std::atomic maximum_active_leases{0}; +}; + +State& GetState(); +void Reset(); +flm::corelib::CorelibApi::Resolver Resolver(); +void* MakeObject(); +void EnterLease(); +void LeaveLease(); + +} // namespace fake_corelib diff --git a/src/test/phi4_corelib_aie4/test_corelib_api.cpp b/src/test/phi4_corelib_aie4/test_corelib_api.cpp new file mode 100644 index 00000000..87914810 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_corelib_api.cpp @@ -0,0 +1,252 @@ +#include "corelib/corelib_object.hpp" +#include "corelib/corelib_runtime.hpp" +#include "fake_corelib.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibError; +using flm::corelib::CorelibRuntime; +using flm::corelib::UniqueMatMulWeights; +using flm::corelib::UniqueRmsNormWeights; +using flm::corelib::UniqueSsMlpWeights; +using flm::corelib::UniqueStream; +using flm::corelib::UniqueTensor; +using flm::corelib::UniqueTensorWindow; + +void SetCorelibPath(const char* value) { +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", value ? value : ""); +#else + if (value) setenv("FLM_AIE4_CORELIB_PATH", value, 1); + else unsetenv("FLM_AIE4_CORELIB_PATH"); +#endif +} + +std::shared_ptr ValidApi() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +void TestVersionIsResolvedBeforeEveryOtherSymbol() { + fake_corelib::Reset(); + ValidApi(); + const auto& order = fake_corelib::GetState().resolution_order; + TEST_REQUIRE(order.size() == 26); + TEST_REQUIRE(order.front() == "ryzenai_corelib_get_version"); +} + +void TestExactlyVersion030IsAccepted() { + fake_corelib::Reset(); + const auto api = ValidApi(); + const auto version = api->runtime_version(); + TEST_REQUIRE(version.major == 0); + TEST_REQUIRE(version.minor == 3); + TEST_REQUIRE(version.patch == 0); +} + +void TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions() { + for (const auto version : {flm::corelib::CorelibVersion{1, 3, 0}, + flm::corelib::CorelibVersion{0, 4, 0}, + flm::corelib::CorelibVersion{0, 3, 1}}) { + fake_corelib::Reset(); + fake_corelib::GetState().version = version; + const std::string error = RequireThrows([&] { ValidApi(); }); + RequireContains(error, "0.3.0"); + RequireContains(error, std::to_string(version.major) + "." + + std::to_string(version.minor) + "." + + std::to_string(version.patch)); + TEST_REQUIRE(fake_corelib::GetState().resolution_order.size() == 1); + } +} + +void TestEveryRequiredSymbolIsResolvedExactlyOnce() { + fake_corelib::Reset(); + ValidApi(); + TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 26); + for (const auto& [name, count] : fake_corelib::GetState().resolution_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } +} + +void TestMissingSymbolNamesTheSymbolAndUnloadsTheDll() { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = "ryzenai_corelib_create_stream"; + std::weak_ptr unloaded; + std::string error; + { + auto module_lifetime = std::make_shared(1); + unloaded = module_lifetime; + auto base = fake_corelib::Resolver(); + CorelibApi::Resolver resolver = + [module_lifetime, base](std::string_view name) { return base(name); }; + module_lifetime.reset(); + error = RequireThrows([&] { CorelibApi::ResolveForTest(std::move(resolver)); }); + } + RequireContains(error, "ryzenai_corelib_create_stream"); + TEST_REQUIRE(unloaded.expired()); +} + +void TestCorelibErrorCopiesStatusCallAndThreadLocalDetail() { + fake_corelib::Reset(); + fake_corelib::GetState().detail = "invalid tensor row count"; + fake_corelib::GetState().status_text = "bad argument"; + const auto api = ValidApi(); + try { + api->Check(ryzenai_corelib_status_bad_argument, "tensor_write"); + TEST_REQUIRE(false); + } catch (const CorelibError& error) { + TEST_REQUIRE(error.status() == ryzenai_corelib_status_bad_argument); + TEST_REQUIRE(error.call() == "tensor_write"); + TEST_REQUIRE(error.detail() == "invalid tensor row count"); + RequireContains(error.what(), "bad argument"); + } +} + +void TestEnvironmentPathMustBeAnAbsoluteDllPath() { + SetCorelibPath("relative/ryzenai_corelib.dll"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + "absolute"); + SetCorelibPath("C:/apps/flm/aie4"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + ".dll"); + SetCorelibPath(nullptr); +} + +void TestEnvironmentPathWinsOverExecutableRelativePath() { + SetCorelibPath("C:/corelib/custom.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == + std::filesystem::path("C:/corelib/custom.dll")); + SetCorelibPath(nullptr); +} + +void TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory() { + SetCorelibPath(nullptr); + const auto expected = std::filesystem::absolute( + std::filesystem::path("C:/apps/flm") / "aie4" / "ryzenai_corelib.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == expected); +} + +void TestEveryUniqueObjectReleasesExactlyOnceAfterMoves() { + fake_corelib::Reset(); + const auto api = ValidApi(); + { + UniqueTensor first(api, fake_corelib::MakeObject()); + UniqueTensor moved(std::move(first)); + UniqueTensor assigned; + assigned = std::move(moved); + UniqueStream stream(api, fake_corelib::MakeObject()); + UniqueTensorWindow window(api, fake_corelib::MakeObject()); + UniqueMatMulWeights matmul(api, fake_corelib::MakeObject()); + UniqueSsMlpWeights ssmlp(api, fake_corelib::MakeObject()); + UniqueRmsNormWeights rmsnorm(api, fake_corelib::MakeObject()); + TEST_REQUIRE(!first && !moved && assigned); + TEST_REQUIRE(api->live_object_count() == 6); + TEST_REQUIRE(fake_corelib::GetState().releases == 0); + } + TEST_REQUIRE(fake_corelib::GetState().releases == 6); + TEST_REQUIRE(api->live_object_count() == 0); +} + +void TestRuntimeRunsDependencySelftestAndRequiresDeviceContext() { + fake_corelib::Reset(); + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "selftest_dependencies"); + + fake_corelib::Reset(); + fake_corelib::GetState().has_device_context = false; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "device context"); + + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + TEST_REQUIRE(runtime->api() != nullptr); + CorelibRuntime::ShutdownProcess(); +} + +void TestExecutionLeaseSerializesTwoThreads() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + std::atomic ready{0}; + auto worker = [&] { + ++ready; + while (ready.load() != 2) std::this_thread::yield(); + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }; + std::thread first(worker); + std::thread second(worker); + first.join(); + second.join(); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + CorelibRuntime::ShutdownProcess(); +} + +void TestCleanupRunsAfterTheLastObjectAndOnlyOnce() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + auto object = std::make_unique(runtime->api(), + fake_corelib::MakeObject()); + RequireContains(RequireThrows([] { CorelibRuntime::ShutdownProcess(); }), + "live corelib object"); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 0); + object.reset(); + + std::atomic lease_acquired{false}; + std::thread holder([&] { + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + lease_acquired = true; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }); + while (!lease_acquired.load()) std::this_thread::yield(); + CorelibRuntime::ShutdownProcess(); + holder.join(); + CorelibRuntime::ShutdownProcess(); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 1); + TEST_REQUIRE(fake_corelib::GetState().releases == 1); + TEST_REQUIRE(fake_corelib::GetState().lifetime_events == + std::vector({"release", "lease_leave", "cleanup"})); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestVersionIsResolvedBeforeEveryOtherSymbol); + RUN_TEST(TestExactlyVersion030IsAccepted); + RUN_TEST(TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions); + RUN_TEST(TestEveryRequiredSymbolIsResolvedExactlyOnce); + RUN_TEST(TestMissingSymbolNamesTheSymbolAndUnloadsTheDll); + RUN_TEST(TestCorelibErrorCopiesStatusCallAndThreadLocalDetail); + RUN_TEST(TestEnvironmentPathMustBeAnAbsoluteDllPath); + RUN_TEST(TestEnvironmentPathWinsOverExecutableRelativePath); + RUN_TEST(TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory); + RUN_TEST(TestEveryUniqueObjectReleasesExactlyOnceAfterMoves); + RUN_TEST(TestRuntimeRunsDependencySelftestAndRequiresDeviceContext); + RUN_TEST(TestExecutionLeaseSerializesTwoThreads); + RUN_TEST(TestCleanupRunsAfterTheLastObjectAndOnlyOnce); +#undef RUN_TEST + return 0; +} diff --git a/src/test/phi4_corelib_aie4/test_real_corelib.cpp b/src/test/phi4_corelib_aie4/test_real_corelib.cpp new file mode 100644 index 00000000..1febec71 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_real_corelib.cpp @@ -0,0 +1,33 @@ +#include "corelib/corelib_api.hpp" +#include "corelib/corelib_runtime.hpp" +#include "test_support.hpp" + +#include +#include +#include + +int main() { + const char* configured = std::getenv("FLM_AIE4_CORELIB_PATH"); + if (configured == nullptr || *configured == '\0') { + std::cout << "SKIP: FLM_AIE4_CORELIB_PATH is unset\n"; + return 77; + } + + try { + const auto api = flm::corelib::CorelibApi::Load( + flm::corelib::CorelibApi::ResolveLibraryPath( + std::filesystem::current_path())); + const auto version = api->runtime_version(); + TEST_REQUIRE(version.major == 0 && version.minor == 3 && version.patch == 0); +#define FLM_ASSERT_CORELIB_SYMBOL(member, symbol) TEST_REQUIRE(api->functions().member != nullptr); + FLM_CORELIB_FUNCTIONS(FLM_ASSERT_CORELIB_SYMBOL) +#undef FLM_ASSERT_CORELIB_SYMBOL + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(api); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } + return 0; +} diff --git a/src/test/phi4_corelib_aie4/test_support.hpp b/src/test/phi4_corelib_aie4/test_support.hpp new file mode 100644 index 00000000..1065e6ea --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_support.hpp @@ -0,0 +1,43 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#define TEST_REQUIRE(condition) \ + do { \ + if (!(condition)) { \ + throw std::runtime_error(std::string("requirement failed: ") + \ + #condition); \ + } \ + } while (false) + +inline void RequireContains(std::string_view text, std::string_view expected) { + if (text.find(expected) == std::string_view::npos) { + throw std::runtime_error("expected '" + std::string(text) + + "' to contain '" + std::string(expected) + "'"); + } +} + +template +std::string RequireThrows(Callable&& callable) { + try { + callable(); + } catch (const Exception& error) { + return error.what(); + } + throw std::runtime_error("expected exception was not thrown"); +} + +inline void RunTest(void (*test)(), const char* name) { + try { + test(); + std::cout << "PASS " << name << '\n'; + } catch (const std::exception& error) { + std::cerr << "FAIL " << name << ": " << error.what() << '\n'; + std::exit(1); + } +} From 1f67adb756fd4addf72072d7bb9ed1345f9d166f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 02:55:18 -0700 Subject: [PATCH 04/37] test: strengthen corelib integration guards --- src/include/corelib/corelib_api.hpp | 5 + src/test/phi4_corelib_aie4/CMakeLists.txt | 86 ++++++++++-- src/test/phi4_corelib_aie4/compile_guard.cpp | 5 - src/test/phi4_corelib_aie4/fake_corelib.cpp | 131 ++++++++++++------ src/test/phi4_corelib_aie4/fake_corelib.hpp | 5 + .../phi4_corelib_aie4/test_corelib_api.cpp | 27 ++++ 6 files changed, 205 insertions(+), 54 deletions(-) delete mode 100644 src/test/phi4_corelib_aie4/compile_guard.cpp diff --git a/src/include/corelib/corelib_api.hpp b/src/include/corelib/corelib_api.hpp index d68ae9fb..c94e5474 100644 --- a/src/include/corelib/corelib_api.hpp +++ b/src/include/corelib/corelib_api.hpp @@ -2,6 +2,11 @@ #include +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 3 || \ + RYZENAI_CORELIB_VERSION_PATCH != 0 +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.3.0" +#endif + #include #include #include diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index a996733b..cde9b4b9 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -9,11 +9,42 @@ if(NOT WIN32) endif() find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) +find_path(BOOST_INCLUDE_DIR NAMES boost/program_options.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_path(XRT_INCLUDE_DIR NAMES xrt/xrt_bo.h + HINTS "$ENV{XRT_INCLUDE_DIR}" + "${CMAKE_CURRENT_LIST_DIR}/../../../../xrt_package/xrt/include" + "C:/dev/XRT/src/runtime_src/core/include" REQUIRED) set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") set(CORELIB_SOURCES "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" "${FLM_SOURCE_DIR}/common/corelib/corelib_runtime.cpp") +# A synthetic mismatched copy proves that the adapter rejects header drift at +# compile time. The caller-provided header remains untouched. +file(READ "${RYZENAI_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" CORELIB_HEADER_TEXT) +string(REGEX REPLACE + "#define RYZENAI_CORELIB_VERSION_PATCH[ \t]+0" + "#define RYZENAI_CORELIB_VERSION_PATCH 1" + WRONG_CORELIB_HEADER_TEXT "${CORELIB_HEADER_TEXT}") +set(WRONG_CORELIB_INCLUDE_DIR + "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/include") +file(MAKE_DIRECTORY "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai") +file(WRITE "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" + "${WRONG_CORELIB_HEADER_TEXT}") +file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + "#define RYZENAI_CORELIB_STATIC 1\n#include \"corelib/corelib_api.hpp\"\n") +try_compile(CORELIB_WRONG_VERSION_ACCEPTED + SOURCES "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + CMAKE_FLAGS + "-DCMAKE_CXX_STANDARD=20" + "-DCMAKE_CXX_FLAGS=/I${FLM_SOURCE_DIR}/include /I${WRONG_CORELIB_INCLUDE_DIR}" + OUTPUT_VARIABLE WRONG_CORELIB_COMPILE_OUTPUT) +if(CORELIB_WRONG_VERSION_ACCEPTED) + message(FATAL_ERROR "Corelib adapter accepted a non-0.3.0 header") +endif() + add_executable(test_corelib_api test_corelib_api.cpp fake_corelib.cpp ${CORELIB_SOURCES}) target_include_directories(test_corelib_api PRIVATE @@ -29,16 +60,53 @@ target_include_directories(test_real_corelib PRIVATE "${RYZENAI_CORELIB_INCLUDE_DIR}") target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) -# Feature-off frontend code must compile without the pinned corelib include path. -add_library(phi4_frontend_compile_off OBJECT compile_guard.cpp) -target_include_directories(phi4_frontend_compile_off PRIVATE "${FLM_SOURCE_DIR}/include") +# Compile the actual production frontend translation unit in both feature modes. +# Empty declaration-only FFmpeg headers isolate this compile check from an +# unrelated optional SDK that is absent on the standalone test host. +set(FRONTEND_STUB_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/frontend-compile-stubs") +foreach(STUB_HEADER IN ITEMS + libavcodec/avcodec.h + libswscale/swscale.h + libavutil/imgutils.h + libavutil/frame.h + libavutil/pixfmt.h) + get_filename_component(STUB_PARENT + "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" DIRECTORY) + file(MAKE_DIRECTORY "${STUB_PARENT}") + file(WRITE "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" "#pragma once\n") +endforeach() + +set(FLM_PRODUCTION_FRONTEND_SOURCES "${FLM_SOURCE_DIR}/src/main.cpp") +function(add_frontend_compile_guard TARGET_NAME ENABLE_CORELIB) + add_library(${TARGET_NAME} OBJECT ${FLM_PRODUCTION_FRONTEND_SOURCES}) + target_include_directories(${TARGET_NAME} PRIVATE + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${FRONTEND_STUB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") + target_compile_definitions(${TARGET_NAME} PRIVATE + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX + __FLM_VERSION__="test" __NPU_VERSION__="0.0.0.0" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins") + target_compile_options(${TARGET_NAME} PRIVATE + $<$:/wd4005 /wd4244>) + if(ENABLE_CORELIB) + target_compile_definitions(${TARGET_NAME} PRIVATE + FLM_ENABLE_CORELIB_AIE4=1) + target_include_directories(${TARGET_NAME} PRIVATE + "${RYZENAI_CORELIB_INCLUDE_DIR}") + endif() +endfunction() -# Feature-on frontend code sees both the feature define and pinned ABI headers. -add_library(phi4_frontend_compile_on OBJECT compile_guard.cpp) -target_include_directories(phi4_frontend_compile_on PRIVATE - "${FLM_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") -target_compile_definitions(phi4_frontend_compile_on PRIVATE - FLM_ENABLE_CORELIB_AIE4=1 RYZENAI_CORELIB_STATIC=1) +# The OFF target deliberately has no RYZENAI_CORELIB_INCLUDE_DIR. +add_frontend_compile_guard(phi4_frontend_compile_off FALSE) +add_frontend_compile_guard(phi4_frontend_compile_on TRUE) include(CTest) add_test(NAME test_corelib_api COMMAND test_corelib_api) diff --git a/src/test/phi4_corelib_aie4/compile_guard.cpp b/src/test/phi4_corelib_aie4/compile_guard.cpp deleted file mode 100644 index da963ff5..00000000 --- a/src/test/phi4_corelib_aie4/compile_guard.cpp +++ /dev/null @@ -1,5 +0,0 @@ -#ifdef FLM_ENABLE_CORELIB_AIE4 -#include "corelib/corelib_runtime.hpp" -#endif - -void Phi4CorelibCompileGuard() {} diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index d6232647..0bc2072c 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -1,58 +1,97 @@ #include "fake_corelib.hpp" -#include -#include +#include +#include +#include namespace { fake_corelib::State state; thread_local std::string current_detail; -void GetVersion(std::uint32_t* major, std::uint32_t* minor, std::uint32_t* patch) { - if (major) *major = state.version.major; - if (minor) *minor = state.version.minor; - if (patch) *patch = state.version.patch; -} +#define FLM_DEFINE_FAKE_TAG(member, symbol) \ + struct member##_tag { \ + static constexpr std::string_view name = #symbol; \ + }; +FLM_CORELIB_FUNCTIONS(FLM_DEFINE_FAKE_TAG) +#undef FLM_DEFINE_FAKE_TAG -const char* StatusToString(ryzenai_corelib_status) { - current_detail = "detail overwritten by status_to_string"; - return state.status_text.c_str(); -} +template +inline constexpr bool kAlwaysFalse = false; -const char* GetLastErrorMessage() { - current_detail = state.detail; - return current_detail.c_str(); -} +template +struct TypedFake; -ryzenai_corelib_status SelftestDependencies() { return state.selftest_status; } -bool HasDeviceContext() { return state.has_device_context; } -void ObjectRelease(void* object) { - if (object) { - delete static_cast(object); - --state.live_objects; - ++state.releases; - state.lifetime_events.emplace_back("release"); +template +struct TypedFake { + static Result Invoke(Args... args) { + ++state.call_counts[std::string(Tag::name)]; + auto arguments = std::forward_as_tuple(args...); + + if constexpr (std::is_same_v) { + if (std::get<0>(arguments)) *std::get<0>(arguments) = state.version.major; + if (std::get<1>(arguments)) *std::get<1>(arguments) = state.version.minor; + if (std::get<2>(arguments)) *std::get<2>(arguments) = state.version.patch; + return; + } else if constexpr (std::is_same_v) { + current_detail = "detail overwritten by status_to_string"; + return state.status_text.c_str(); + } else if constexpr (std::is_same_v) { + current_detail = state.detail; + return current_detail.c_str(); + } else if constexpr (std::is_same_v) { + return state.selftest_status; + } else if constexpr (std::is_same_v) { + return state.has_device_context; + } else if constexpr (std::is_same_v) { + void* object = std::get<0>(arguments); + if (object) { + delete static_cast(object); + --state.live_objects; + ++state.releases; + state.lifetime_events.emplace_back("release"); + } + return; + } else if constexpr (std::is_same_v) { + ++state.cleanup_calls; + state.lifetime_events.emplace_back("cleanup"); + return; + } else if constexpr (std::is_same_v) { + const auto configured = state.statuses.find(std::string(Tag::name)); + return configured == state.statuses.end() ? state.default_status + : configured->second; + } else { + static_assert(kAlwaysFalse, "unhandled fake corelib ABI result"); + } } -} -void Cleanup() { - ++state.cleanup_calls; - state.lifetime_events.emplace_back("cleanup"); -} -void UncalledSymbol() {} +}; + +#define FLM_ASSERT_FAKE_ABI(member, symbol) \ + static_assert(std::is_same_v< \ + decltype(&TypedFake::Invoke), \ + decltype(&::symbol)>); +FLM_CORELIB_FUNCTIONS(FLM_ASSERT_FAKE_ABI) +#undef FLM_ASSERT_FAKE_ABI void* FunctionFor(std::string_view name) { - if (name == "ryzenai_corelib_get_version") return reinterpret_cast(&GetVersion); - if (name == "ryzenai_corelib_status_to_string") return reinterpret_cast(&StatusToString); - if (name == "ryzenai_corelib_get_last_error_message") return reinterpret_cast(&GetLastErrorMessage); - if (name == "ryzenai_corelib_selftest_dependencies") return reinterpret_cast(&SelftestDependencies); - if (name == "ryzenai_corelib_has_device_context") return reinterpret_cast(&HasDeviceContext); - if (name == "ryzenai_corelib_object_release") return reinterpret_cast(&ObjectRelease); - if (name == "ryzenai_corelib_cleanup") return reinterpret_cast(&Cleanup); -#define FLM_FAKE_CORELIB_SYMBOL(member, symbol) \ - if (name == #symbol) return reinterpret_cast(&UncalledSymbol); - FLM_CORELIB_FUNCTIONS(FLM_FAKE_CORELIB_SYMBOL) -#undef FLM_FAKE_CORELIB_SYMBOL +#define FLM_MAP_FAKE_FUNCTION(member, symbol) \ + if (name == #symbol) { \ + return reinterpret_cast( \ + &TypedFake::Invoke); \ + } + FLM_CORELIB_FUNCTIONS(FLM_MAP_FAKE_FUNCTION) +#undef FLM_MAP_FAKE_FUNCTION return nullptr; } + +template +void CallAndCollect(Result (*function)(Args...), + std::vector& statuses) { + if constexpr (std::is_same_v) { + statuses.push_back(function(Args{}...)); + } else { + function(Args{}...); + } +} } // namespace namespace fake_corelib { @@ -62,12 +101,15 @@ State& GetState() { return state; } void Reset() { state.version = {0, 3, 0}; state.selftest_status = ryzenai_corelib_status_success; + state.default_status = ryzenai_corelib_status_success; state.has_device_context = true; state.detail.clear(); state.status_text = "success"; state.missing_symbol.clear(); state.resolution_order.clear(); state.resolution_counts.clear(); + state.call_counts.clear(); + state.statuses.clear(); state.lifetime_events.clear(); state.live_objects = 0; state.releases = 0; @@ -85,6 +127,15 @@ flm::corelib::CorelibApi::Resolver Resolver() { }; } +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions) { + std::vector statuses; +#define FLM_CALL_FAKE_FUNCTION(member, symbol) CallAndCollect(functions.member, statuses); + FLM_CORELIB_FUNCTIONS(FLM_CALL_FAKE_FUNCTION) +#undef FLM_CALL_FAKE_FUNCTION + return statuses; +} + void* MakeObject() { ++state.live_objects; return new int(1); diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp index 341880c3..9d0383c0 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.hpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -15,12 +15,15 @@ namespace fake_corelib { struct State { flm::corelib::CorelibVersion version{0, 3, 0}; ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; + ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; bool has_device_context{true}; std::string detail; std::string status_text{"success"}; std::string missing_symbol; std::vector resolution_order; std::unordered_map resolution_counts; + std::unordered_map call_counts; + std::unordered_map statuses; std::vector lifetime_events; std::atomic live_objects{0}; std::atomic releases{0}; @@ -32,6 +35,8 @@ struct State { State& GetState(); void Reset(); flm::corelib::CorelibApi::Resolver Resolver(); +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions); void* MakeObject(); void EnterLease(); void LeaveLease(); diff --git a/src/test/phi4_corelib_aie4/test_corelib_api.cpp b/src/test/phi4_corelib_aie4/test_corelib_api.cpp index 87914810..d3cd9bb2 100644 --- a/src/test/phi4_corelib_aie4/test_corelib_api.cpp +++ b/src/test/phi4_corelib_aie4/test_corelib_api.cpp @@ -79,6 +79,32 @@ void TestEveryRequiredSymbolIsResolvedExactlyOnce() { } } +void TestEveryResolvedFakeFunctionUsesItsExactAbi() { + fake_corelib::Reset(); + const auto api = ValidApi(); + fake_corelib::GetState().call_counts.clear(); + fake_corelib::GetState().default_status = ryzenai_corelib_status_bad_argument; + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_bad_argument; + const auto statuses = fake_corelib::CallEveryResolvedFunction(api->functions()); + TEST_REQUIRE(statuses.size() == 20); + TEST_REQUIRE(std::all_of(statuses.begin(), statuses.end(), [](auto status) { + return status == ryzenai_corelib_status_bad_argument; + })); + TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 26); + for (const auto& [name, count] : fake_corelib::GetState().call_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } + + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = + ryzenai_corelib_status_unsupported; + TEST_REQUIRE(api->functions().tensor_write( + nullptr, ryzenai_corelib_data_type_bf16, nullptr, 0, 0) == + ryzenai_corelib_status_unsupported); + TEST_REQUIRE(fake_corelib::GetState() + .call_counts["ryzenai_corelib_tensor_write"] == 2); +} + void TestMissingSymbolNamesTheSymbolAndUnloadsTheDll() { fake_corelib::Reset(); fake_corelib::GetState().missing_symbol = "ryzenai_corelib_create_stream"; @@ -238,6 +264,7 @@ int main() { RUN_TEST(TestExactlyVersion030IsAccepted); RUN_TEST(TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions); RUN_TEST(TestEveryRequiredSymbolIsResolvedExactlyOnce); + RUN_TEST(TestEveryResolvedFakeFunctionUsesItsExactAbi); RUN_TEST(TestMissingSymbolNamesTheSymbolAndUnloadsTheDll); RUN_TEST(TestCorelibErrorCopiesStatusCallAndThreadLocalDetail); RUN_TEST(TestEnvironmentPathMustBeAnAbsoluteDllPath); From f2c1dba4d91b18a65c1ab56d9b3858d110fa5c63 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 03:12:47 -0700 Subject: [PATCH 05/37] feat: add validated Phi-4 Q8_0 GGUF reader --- src/common/corelib/corelib_sources.cmake | 3 +- src/common/corelib/phi4_corelib_gguf.cpp | 601 ++++++++++++++++++ .../models/phi4/phi4_corelib_constants.hpp | 20 + src/include/models/phi4/phi4_corelib_gguf.hpp | 77 +++ src/test/phi4_corelib_aie4/CMakeLists.txt | 7 + src/test/phi4_corelib_aie4/gguf_fixture.hpp | 374 +++++++++++ src/test/phi4_corelib_aie4/test_phi4_gguf.cpp | 336 ++++++++++ 7 files changed, 1417 insertions(+), 1 deletion(-) create mode 100644 src/common/corelib/phi4_corelib_gguf.cpp create mode 100644 src/include/models/phi4/phi4_corelib_constants.hpp create mode 100644 src/include/models/phi4/phi4_corelib_gguf.hpp create mode 100644 src/test/phi4_corelib_aie4/gguf_fixture.hpp create mode 100644 src/test/phi4_corelib_aie4/test_phi4_gguf.cpp diff --git a/src/common/corelib/corelib_sources.cmake b/src/common/corelib/corelib_sources.cmake index a2ba02f1..09fea42a 100644 --- a/src/common/corelib/corelib_sources.cmake +++ b/src/common/corelib/corelib_sources.cmake @@ -1,3 +1,4 @@ set(FLM_CORELIB_AIE4_SOURCES "${CMAKE_CURRENT_LIST_DIR}/corelib_api.cpp" - "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp") + "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp" + "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_gguf.cpp") diff --git a/src/common/corelib/phi4_corelib_gguf.cpp b/src/common/corelib/phi4_corelib_gguf.cpp new file mode 100644 index 00000000..019614d2 --- /dev/null +++ b/src/common/corelib/phi4_corelib_gguf.cpp @@ -0,0 +1,601 @@ +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" + +#define NOMINMAX +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +constexpr std::uint32_t kMagic = 0x46554747; +constexpr std::uint32_t kVersion = 3; +constexpr std::uint32_t kTypeF32 = 0; +constexpr std::uint32_t kTypeQ8_0 = 8; + +[[noreturn]] void Fail(std::string_view field, std::string actual, + std::string expected) { + throw std::runtime_error(std::string(field) + ": actual " + actual + + ", expected " + expected); +} + +std::uint64_t CheckedAdd(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a > std::numeric_limits::max() - b) + throw std::runtime_error(std::string(field) + ": overflow in addition"); + return a + b; +} + +std::uint64_t CheckedMultiply(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a != 0 && b > std::numeric_limits::max() / a) + throw std::runtime_error(std::string(field) + ": overflow in product"); + return a * b; +} + +std::uint64_t AlignUp(std::uint64_t value, std::uint64_t alignment) { + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + return CheckedAdd(value, alignment - 1, "alignment") & ~(alignment - 1); +} + +std::span RequireRange(std::span file, + std::uint64_t offset, + std::uint64_t length, + std::string_view field) { + const auto end = CheckedAdd(offset, length, field); + if (end > file.size() || offset > std::numeric_limits::max() || + length > std::numeric_limits::max()) + Fail(field, "out-of-file range", "range within mapped file"); + return file.subspan(static_cast(offset), + static_cast(length)); +} + +class Cursor { +public: + Cursor(std::span file, std::uint64_t offset = 0) + : file_(file), offset_(offset) {} + + template + T Read(std::string_view field) { + const auto bytes = RequireRange(file_, offset_, sizeof(T), field); + T value; + std::memcpy(&value, bytes.data(), sizeof(T)); + offset_ = CheckedAdd(offset_, sizeof(T), field); + return value; + } + + std::string ReadString(std::string_view field) { + const auto length = Read(field); + const auto bytes = RequireRange(file_, offset_, length, field); + std::string value(reinterpret_cast(bytes.data()), bytes.size()); + offset_ = CheckedAdd(offset_, length, field); + return value; + } + + void Skip(std::uint64_t length, std::string_view field) { + RequireRange(file_, offset_, length, field); + offset_ = CheckedAdd(offset_, length, field); + } + + std::uint64_t offset() const noexcept { return offset_; } + +private: + std::span file_; + std::uint64_t offset_; +}; + +std::string MetadataTypeName(std::uint32_t type) { + static constexpr const char* names[] = {"UINT8", "INT8", "UINT16", "INT16", + "UINT32", "INT32", "FLOAT32", "BOOL", "STRING", "ARRAY", "UINT64", + "INT64", "FLOAT64"}; + return type < std::size(names) ? names[type] : "unknown(" + std::to_string(type) + ")"; +} + +std::uint64_t FixedMetadataSize(std::uint32_t type) { + switch (type) { + case 0: case 1: case 7: return 1; + case 2: case 3: return 2; + case 4: case 5: case 6: return 4; + case 10: case 11: case 12: return 8; + default: return 0; + } +} + +struct ArrayInfo { std::uint32_t type; std::uint64_t count; }; +using MetadataValue = std::variant; + +MetadataValue ReadMetadataValue(Cursor& cursor, std::uint32_t type, + std::string_view field, bool retain) { + switch (type) { + case 0: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 1: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 2: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 3: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 4: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 5: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 6: { auto v = cursor.Read(field); return retain ? MetadataValue(double(v)) : MetadataValue{}; } + case 7: { auto v = cursor.Read(field); if (v > 1) Fail(field, std::to_string(v), "GGUF boolean 0 or 1"); return retain ? MetadataValue(bool(v)) : MetadataValue{}; } + case 8: { auto v = cursor.ReadString(field); return retain ? MetadataValue(std::move(v)) : MetadataValue{}; } + case 9: { + const std::string array_field = std::string(field) + " array"; + const auto element_type = cursor.Read(array_field); + const auto count = cursor.Read(array_field); + if (element_type == 9 || element_type > 12) + Fail(array_field, MetadataTypeName(element_type), "a skippable GGUF array element type"); + const auto fixed = FixedMetadataSize(element_type); + if (fixed != 0) { + cursor.Skip(CheckedMultiply(count, fixed, array_field), array_field); + } else { + const auto minimum = CheckedMultiply(count, std::uint64_t{8}, array_field); + (void)minimum; + for (std::uint64_t i = 0; i < count; ++i) + (void)ReadMetadataValue(cursor, element_type, array_field, false); + } + return retain ? MetadataValue(ArrayInfo{element_type, count}) : MetadataValue{}; + } + case 10: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 11: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 12: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + default: + Fail(field, MetadataTypeName(type), "a supported metadata type"); + } +} + +bool IsRetainedKey(std::string_view key) { + static constexpr std::string_view keys[] = { + "general.architecture", "general.alignment", "phi3.block_count", + "phi3.context_length", "phi3.embedding_length", "phi3.feed_forward_length", + "phi3.attention.head_count", "phi3.attention.head_count_kv", + "phi3.attention.layer_norm_rms_epsilon", "phi3.rope.dimension_count", + "phi3.rope.freq_base", "phi3.rope.scaling.attn_factor", + "phi3.rope.scaling.original_context_length", "tokenizer.ggml.tokens", + "tokenizer.ggml.add_bos_token", "tokenizer.ggml.eos_token_id"}; + return std::find(std::begin(keys), std::end(keys), key) != std::end(keys); +} + +std::string ShapeText(std::span shape) { + std::ostringstream out; + out << '['; + for (std::size_t i = 0; i < shape.size(); ++i) { + if (i) out << ','; + out << shape[i]; + } + return out.str() + ']'; +} + +std::string GgmlTypeName(std::uint32_t type) { + if (type == kTypeF32) return "F32"; + if (type == kTypeQ8_0) return "Q8_0"; + return "GGML type " + std::to_string(type); +} + +std::uint64_t ElementCount(std::span shape, + std::string_view field) { + std::uint64_t result = 1; + for (const auto dimension : shape) { + if (dimension <= 0) Fail(field, std::to_string(dimension), "positive dimensions"); + result = CheckedMultiply(result, static_cast(dimension), field); + } + return result; +} + +std::uint64_t TensorByteLength(std::uint32_t type, + std::span shape, + std::string_view field) { + const auto elements = ElementCount(shape, field); + if (type == kTypeF32) return CheckedMultiply(elements, 4, field); + if (type == kTypeQ8_0) { + if (elements % 32 != 0) + Fail(field, std::to_string(elements) + " elements", "Q8_0 element count divisible by 32"); + return CheckedMultiply(elements / 32, 34, field); + } + Fail(field, GgmlTypeName(type), "F32 or Q8_0"); +} + +std::string JsonText(const nlohmann::json& value) { + return value.dump(); +} + +template +void RequireJson(const nlohmann::json& object, std::string_view key, + const T& expected) { + const auto it = object.find(std::string(key)); + if (it == object.end()) Fail(key, "missing", nlohmann::json(expected).dump()); + try { + if (it->template get() != expected) + Fail(key, JsonText(*it), nlohmann::json(expected).dump()); + } catch (const nlohmann::json::exception&) { + Fail(key, JsonText(*it), nlohmann::json(expected).dump()); + } +} + +void RequireJsonDouble(const nlohmann::json& object, std::string_view key, + double expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_number()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::to_string(expected)); + const auto actual = it->get(); + if (!std::isfinite(actual) || actual != expected) + Fail(key, JsonText(*it), std::to_string(expected)); +} +} // namespace + +struct Phi4GgufPackage::Impl { + struct TensorRecord { + std::string name; + std::span bytes; + std::vector shape; + std::uint32_t type; + std::uint64_t absolute_offset; + }; + + HANDLE file = INVALID_HANDLE_VALUE; + HANDLE mapping = nullptr; + const std::byte* data = nullptr; + std::uint64_t size = 0; + std::map> tensors; + std::map> metadata; + std::map> metadata_types; + + ~Impl() { + if (data) UnmapViewOfFile(data); + if (mapping) CloseHandle(mapping); + if (file != INVALID_HANDLE_VALUE) CloseHandle(file); + } + + std::span bytes() const { + return {data, static_cast(size)}; + } + + const TensorRecord& Tensor(std::string_view name) const { + const auto it = tensors.find(name); + if (it == tensors.end()) Fail(name, "missing", "present tensor"); + return it->second; + } + + std::uint64_t Unsigned(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "unsigned integer metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "unsigned integer metadata"); + } + + double Number(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "floating-point metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "floating-point metadata"); + } + + bool Boolean(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "boolean metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "boolean metadata"); + } + + std::string String(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "string metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "string metadata"); + } + + std::uint64_t ArrayCount(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "array metadata"); + if (const auto* value = std::get_if(&it->second)) return value->count; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "array metadata"); + } +}; + +Phi4GgufPackage::Phi4GgufPackage(std::unique_ptr impl) + : impl_(std::move(impl)) {} +Phi4GgufPackage::~Phi4GgufPackage() = default; + +std::shared_ptr Phi4GgufPackage::Open( + const std::filesystem::path& gguf_path) { + auto impl = std::make_unique(); + impl->file = CreateFileW(gguf_path.c_str(), GENERIC_READ, FILE_SHARE_READ, + nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); + if (impl->file == INVALID_HANDLE_VALUE) + throw std::runtime_error("GGUF file: actual open failure " + + std::to_string(GetLastError()) + ", expected readable file"); + LARGE_INTEGER size; + if (!GetFileSizeEx(impl->file, &size) || size.QuadPart <= 0 || + static_cast(size.QuadPart) > std::numeric_limits::max()) + Fail("GGUF file size", std::to_string(size.QuadPart), "positive mappable size"); + impl->size = static_cast(size.QuadPart); + impl->mapping = CreateFileMappingW(impl->file, nullptr, PAGE_READONLY, 0, 0, nullptr); + if (!impl->mapping) + throw std::runtime_error("GGUF mapping: actual CreateFileMappingW failure " + + std::to_string(GetLastError()) + ", expected PAGE_READONLY mapping"); + impl->data = static_cast( + MapViewOfFile(impl->mapping, FILE_MAP_READ, 0, 0, 0)); + if (!impl->data) + throw std::runtime_error("GGUF mapping: actual MapViewOfFile failure " + + std::to_string(GetLastError()) + ", expected FILE_MAP_READ view"); + + const auto file = impl->bytes(); + Cursor cursor(file); + if (cursor.Read("GGUF header") != kMagic) + Fail("GGUF magic", "mismatch", "0x46554747"); + const auto version = cursor.Read("GGUF header"); + if (version != kVersion) Fail("GGUF version", std::to_string(version), "3"); + const auto tensor_count = cursor.Read("tensor count"); + const auto metadata_count = cursor.Read("metadata count"); + if (tensor_count > file.size() / 24) Fail("tensor count", std::to_string(tensor_count), "count fitting directory"); + if (metadata_count > file.size() / 12) Fail("metadata count", std::to_string(metadata_count), "count fitting metadata"); + + for (std::uint64_t i = 0; i < metadata_count; ++i) { + const auto key = cursor.ReadString("metadata key string"); + const auto type = cursor.Read(key); + const bool retain = IsRetainedKey(key); + auto value = ReadMetadataValue(cursor, type, key, retain); + if (retain) { + if (!impl->metadata.emplace(key, std::move(value)).second) + Fail(key, "duplicate metadata key", "unique metadata key"); + impl->metadata_types.emplace(key, type); + } + } + + const auto alignment = impl->Unsigned("general.alignment"); + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + + struct DirectoryTensor { + std::string name; + std::vector shape; + std::uint32_t type; + std::uint64_t relative_offset; + std::uint64_t length; + }; + std::vector directory; + directory.reserve(static_cast(tensor_count)); + for (std::uint64_t i = 0; i < tensor_count; ++i) { + auto name = cursor.ReadString("tensor directory name"); + const auto dimension_count = cursor.Read("tensor directory dimensions"); + if (dimension_count == 0 || dimension_count > 4) + Fail(name, std::to_string(dimension_count), "1..4 tensor dimensions"); + std::vector shape; + shape.reserve(dimension_count); + for (std::uint32_t d = 0; d < dimension_count; ++d) { + const auto dimension = cursor.Read("tensor directory dimension"); + if (dimension > static_cast(std::numeric_limits::max())) + Fail(name, std::to_string(dimension), "dimension fitting int64"); + shape.push_back(static_cast(dimension)); + } + std::reverse(shape.begin(), shape.end()); + const auto type = cursor.Read("tensor directory type"); + const auto offset = cursor.Read("tensor directory offset"); + const auto length = TensorByteLength(type, shape, name); + directory.push_back({std::move(name), std::move(shape), type, offset, length}); + } + + const auto data_start = AlignUp(cursor.offset(), alignment); + struct Range { std::uint64_t begin, end; std::string name; }; + std::vector ranges; + ranges.reserve(directory.size()); + for (auto& tensor : directory) { + if (tensor.relative_offset % alignment != 0) + Fail(tensor.name, std::to_string(tensor.relative_offset), "offset aligned to " + std::to_string(alignment)); + const auto absolute = CheckedAdd(data_start, tensor.relative_offset, tensor.name); + const auto bytes = RequireRange(file, absolute, tensor.length, tensor.name + " range"); + const auto end = CheckedAdd(absolute, tensor.length, tensor.name); + ranges.push_back({absolute, end, tensor.name}); + auto [it, inserted] = impl->tensors.emplace(tensor.name, + Impl::TensorRecord{tensor.name, bytes, std::move(tensor.shape), tensor.type, absolute}); + if (!inserted) Fail(tensor.name, "duplicate tensor name", "unique tensor name"); + } + std::sort(ranges.begin(), ranges.end(), [](const Range& a, const Range& b) { + return a.begin < b.begin; + }); + for (std::size_t i = 1; i < ranges.size(); ++i) { + if (ranges[i].begin < ranges[i - 1].end) + Fail(ranges[i].name, "overlap with " + ranges[i - 1].name, "non-overlapping tensor range"); + } + return std::shared_ptr(new Phi4GgufPackage(std::move(impl))); +} + +TensorView Phi4GgufPackage::RequireQ8( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeQ8_0) + Fail(name, GgmlTypeName(tensor.type), "Q8_0"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeQ8_0, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + return {tensor.name, tensor.bytes, tensor.shape, tensor.type}; +} + +FloatTensorView Phi4GgufPackage::RequireF32( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeF32) + Fail(name, GgmlTypeName(tensor.type), "F32"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeF32, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + return {tensor.name, + {reinterpret_cast(tensor.bytes.data()), + tensor.bytes.size() / sizeof(float)}, + tensor.shape}; +} + +ProjectionViews Phi4GgufPackage::AttentionQkv(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("attention layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".attn_qkv.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{5120, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 3; + result.values[0] = {fused.name, fused.bytes.subspan(0, 3072 * row_bytes), {3072, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(3072 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + result.values[2] = {fused.name, fused.bytes.subspan(4096 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + return result; +} + +ProjectionViews Phi4GgufPackage::GateUp(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("MLP layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".ffn_up.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{16384, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 2; + result.values[0] = {fused.name, fused.bytes.subspan(0, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(8192 * row_bytes, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + return result; +} + +GgufPhi4Metadata Phi4GgufPackage::Metadata() const { + return {impl_->String("general.architecture"), + impl_->Unsigned("phi3.block_count"), + impl_->Unsigned("phi3.embedding_length"), + impl_->Unsigned("phi3.feed_forward_length"), + impl_->Unsigned("phi3.attention.head_count"), + impl_->Unsigned("phi3.attention.head_count_kv"), + impl_->Unsigned("phi3.context_length"), + impl_->Unsigned("phi3.rope.dimension_count"), + impl_->Number("phi3.rope.freq_base"), + impl_->Number("phi3.rope.scaling.attn_factor"), + impl_->Unsigned("phi3.rope.scaling.original_context_length"), + impl_->ArrayCount("tokenizer.ggml.tokens"), + impl_->Boolean("tokenizer.ggml.add_bos_token")}; +} + +void Phi4GgufPackage::ValidatePhi4Contract( + const nlohmann::json& config, const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const { + const auto metadata = Metadata(); + const auto require_unsigned = [](std::string_view field, std::uint64_t actual, + std::uint64_t expected) { + if (actual != expected) Fail(field, std::to_string(actual), std::to_string(expected)); + }; + if (metadata.architecture != "phi3") Fail("general.architecture", metadata.architecture, "phi3"); + require_unsigned("phi3.block_count", metadata.layer_count, kLayerCount); + require_unsigned("phi3.context_length", metadata.context_length, kMaxSequenceLength); + require_unsigned("phi3.embedding_length", metadata.hidden_size, kHiddenSize); + require_unsigned("phi3.feed_forward_length", metadata.intermediate_size, kIntermediateSize); + require_unsigned("phi3.attention.head_count", metadata.attention_head_count, kQueryHeadCount); + require_unsigned("phi3.attention.head_count_kv", metadata.kv_head_count, kKvHeadCount); + require_unsigned("phi3.rope.dimension_count", metadata.rope_dimension_count, kRopeDimension); + require_unsigned("phi3.rope.scaling.original_context_length", metadata.rope_original_context_length, kMaxSequenceLength); + require_unsigned("tokenizer.ggml.tokens", metadata.tokenizer_vocabulary_size, kVocabularySize); + if (metadata.add_bos_token) Fail("tokenizer.ggml.add_bos_token", "true", "false"); + const auto rms = impl_->Number("phi3.attention.layer_norm_rms_epsilon"); + if (!std::isfinite(rms) || rms != static_cast(kRmsEpsilon)) + Fail("phi3.attention.layer_norm_rms_epsilon", std::to_string(rms), std::to_string(kRmsEpsilon)); + for (const auto [field, value] : std::array{ + std::pair{"phi3.rope.freq_base", metadata.rope_frequency_base}, + std::pair{"phi3.rope.scaling.attn_factor", metadata.rope_attention_factor}}) { + if (!std::isfinite(value) || value <= 0) Fail(field, std::to_string(value), "finite positive value"); + } + + RequireQ8("token_embd.weight", std::array{kVocabularySize, kHiddenSize}); + RequireF32("output_norm.weight", std::array{kHiddenSize}); + if (impl_->tensors.contains("output.weight")) Fail("output.weight", "present", "absent (tied token_embd.weight)"); + if (impl_->tensors.contains("rope_factors_long.weight")) Fail("rope_factors_long.weight", "present", "absent for original 4096 window"); + for (std::size_t layer = 0; layer < static_cast(kLayerCount); ++layer) { + const auto prefix = "blk." + std::to_string(layer); + RequireF32(prefix + ".attn_norm.weight", std::array{kHiddenSize}); + RequireF32(prefix + ".ffn_norm.weight", std::array{kHiddenSize}); + RequireQ8(prefix + ".attn_qkv.weight", std::array{kQueryDimension + 2 * kKvDimension, kHiddenSize}); + RequireQ8(prefix + ".attn_output.weight", std::array{kHiddenSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_up.weight", std::array{2 * kIntermediateSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_down.weight", std::array{kHiddenSize, kIntermediateSize}); + } + if (impl_->tensors.contains("rope_factors_short.weight")) + RequireF32("rope_factors_short.weight", std::array{48}); + + RequireJson(config, "model_type", std::string("phi3")); + RequireJson(config, "num_hidden_layers", int(kLayerCount)); + RequireJson(config, "hidden_size", int(kHiddenSize)); + RequireJson(config, "intermediate_size", int(kIntermediateSize)); + RequireJson(config, "num_attention_heads", int(kQueryHeadCount)); + RequireJson(config, "num_key_value_heads", int(kKvHeadCount)); + RequireJson(config, "head_dim", int(kHeadSize)); + RequireJson(config, "vocab_size", int(kVocabularySize)); + RequireJsonDouble(config, "rms_norm_eps", 1.0e-5); + RequireJson(config, "original_max_position_embeddings", int(kMaxSequenceLength)); + RequireJson(config, "eos_token_id", 199999); + + std::set vocabulary_ids; + std::map> token_ids; + const auto add_token = [&](const std::string& token, std::int64_t id) { + const auto [it, inserted] = token_ids.emplace(token, id); + if (!inserted && it->second != id) + Fail(token, std::to_string(id), std::to_string(it->second)); + vocabulary_ids.insert(id); + }; + try { + const auto& vocab = tokenizer.at("model").at("vocab"); + if (!vocab.is_object()) Fail("tokenizer.json model.vocab", JsonText(vocab), "object mapping tokens to IDs"); + for (auto it = vocab.begin(); it != vocab.end(); ++it) { + const auto id = it.value().get(); + add_token(it.key(), id); + } + const auto added = tokenizer.find("added_tokens"); + if (added != tokenizer.end()) { + if (!added->is_array()) Fail("tokenizer.json added_tokens", JsonText(*added), "array"); + for (const auto& item : *added) { + const auto id = item.at("id").get(); + const auto content = item.at("content").get(); + add_token(content, id); + } + } + } catch (const nlohmann::json::exception& error) { + Fail("tokenizer.json vocabulary", error.what(), "valid token-to-ID mappings"); + } + const auto actual_count = vocabulary_ids.size(); + const auto actual_max_plus_one = vocabulary_ids.empty() ? 0 : *vocabulary_ids.rbegin() + 1; + if (actual_count != static_cast(kVocabularySize)) + Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), std::to_string(kVocabularySize)); + if (actual_max_plus_one != kVocabularySize) + Fail("tokenizer.json maximum ID plus one", std::to_string(actual_max_plus_one), std::to_string(kVocabularySize)); + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + const auto it = token_ids.find(token); + if (it == token_ids.end()) Fail(token, "missing", std::to_string(expected)); + if (it->second != expected) Fail(token, std::to_string(it->second), std::to_string(expected)); + } + const auto gguf_eos = impl_->Unsigned("tokenizer.ggml.eos_token_id"); + if (gguf_eos != 200020) Fail("tokenizer.ggml.eos_token_id", std::to_string(gguf_eos), "200020"); + + RequireJson(tokenizer_config, "add_bos_token", false); + const auto template_it = tokenizer_config.find("chat_template"); + if (template_it == tokenizer_config.end() || !template_it->is_string()) + Fail("chat_template", template_it == tokenizer_config.end() ? "missing" : JsonText(*template_it), "string containing Phi-4 markers"); + const auto chat_template = template_it->get(); + for (const auto marker : {"<|user|>", "<|end|>", "<|assistant|>"}) + if (chat_template.find(marker) == std::string::npos) + Fail(marker, "missing from chat_template", "present in chat_template"); +} + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_constants.hpp b/src/include/models/phi4/phi4_corelib_constants.hpp new file mode 100644 index 00000000..fc1a80c0 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_constants.hpp @@ -0,0 +1,20 @@ +#pragma once + +#include + +namespace flm::phi4 { +inline constexpr std::int64_t kLayerCount = 32; +inline constexpr std::int64_t kHiddenSize = 3072; +inline constexpr std::int64_t kIntermediateSize = 8192; +inline constexpr std::int64_t kQueryHeadCount = 24; +inline constexpr std::int64_t kKvHeadCount = 8; +inline constexpr std::int64_t kHeadSize = 128; +inline constexpr std::int64_t kQueryDimension = 3072; +inline constexpr std::int64_t kKvDimension = 1024; +inline constexpr std::int64_t kVocabularySize = 200064; +inline constexpr std::int64_t kRopeDimension = 96; +inline constexpr std::int64_t kMaxSequenceLength = 4096; +inline constexpr std::int64_t kMaxDecodeWindow = 4095; +inline constexpr std::uint32_t kRequantizedGroupSize = 64; +inline constexpr float kRmsEpsilon = 1.0e-5f; +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_gguf.hpp b/src/include/models/phi4/phi4_corelib_gguf.hpp new file mode 100644 index 00000000..7af4e684 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_gguf.hpp @@ -0,0 +1,77 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace flm::phi4 { + +struct TensorView { + std::string_view name; + std::span bytes; + std::vector logical_shape; + std::uint32_t ggml_type; +}; + +struct FloatTensorView { + std::string_view name; + std::span values; + std::vector logical_shape; +}; + +struct ProjectionViews { + std::array values; + std::size_t count; +}; + +struct GgufPhi4Metadata { + std::string architecture; + std::uint64_t layer_count; + std::uint64_t hidden_size; + std::uint64_t intermediate_size; + std::uint64_t attention_head_count; + std::uint64_t kv_head_count; + std::uint64_t context_length; + std::uint64_t rope_dimension_count; + double rope_frequency_base; + double rope_attention_factor; + std::uint64_t rope_original_context_length; + std::uint64_t tokenizer_vocabulary_size; + bool add_bos_token; +}; + +class Phi4GgufPackage final { +public: + static std::shared_ptr Open( + const std::filesystem::path& gguf_path); + ~Phi4GgufPackage(); + + TensorView RequireQ8( + std::string_view name, + std::span expected_shape) const; + FloatTensorView RequireF32( + std::string_view name, + std::span expected_shape) const; + ProjectionViews AttentionQkv(std::size_t layer) const; + ProjectionViews GateUp(std::size_t layer) const; + GgufPhi4Metadata Metadata() const; + void ValidatePhi4Contract( + const nlohmann::json& config, + const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const; + +private: + struct Impl; + explicit Phi4GgufPackage(std::unique_ptr impl); + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index cde9b4b9..bfbfccc0 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -60,6 +60,12 @@ target_include_directories(test_real_corelib PRIVATE "${RYZENAI_CORELIB_INCLUDE_DIR}") target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) +add_executable(test_phi4_gguf + test_phi4_gguf.cpp "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp") +target_include_directories(test_phi4_gguf PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include") + # Compile the actual production frontend translation unit in both feature modes. # Empty declaration-only FFmpeg headers isolate this compile check from an # unrelated optional SDK that is absent on the standalone test host. @@ -111,4 +117,5 @@ add_frontend_compile_guard(phi4_frontend_compile_on TRUE) include(CTest) add_test(NAME test_corelib_api COMMAND test_corelib_api) add_test(NAME test_real_corelib COMMAND test_real_corelib) +add_test(NAME test_phi4_gguf COMMAND test_phi4_gguf) set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/gguf_fixture.hpp b/src/test/phi4_corelib_aie4/gguf_fixture.hpp new file mode 100644 index 00000000..d3d3dcd4 --- /dev/null +++ b/src/test/phi4_corelib_aie4/gguf_fixture.hpp @@ -0,0 +1,374 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace gguf_fixture { + +inline constexpr std::uint32_t kF32 = 0; +inline constexpr std::uint32_t kQ8_0 = 8; + +enum class Mutation { + None, + TruncatedString, + TruncatedDirectory, + CountOverflow, + ProductOverflow, + OffsetOverflow, + ZeroAlignment, + NonPowerOfTwoAlignment, + DuplicateName, + OutOfFileRange, + OverlappingRanges, + UnsupportedMetadataType, + DtypeMismatch, + ShapeMismatch, + PayloadLengthMismatch, +}; + +struct ArrayValue { + std::uint32_t element_type; + std::uint64_t count; + std::vector encoded_elements; +}; +using MetadataValue = std::variant; + +struct Tensor { + std::string name; + std::vector logical_shape; + std::uint32_t type; + std::uint64_t offset = 0; + bool explicit_offset = false; +}; + +struct TempFile { + std::filesystem::path path; + TempFile() = default; + explicit TempFile(std::filesystem::path value) : path(std::move(value)) {} + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; + TempFile(TempFile&& other) noexcept : path(std::move(other.path)) { + other.path.clear(); + } + TempFile& operator=(TempFile&& other) noexcept { + if (this != &other) { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + path = std::move(other.path); + other.path.clear(); + } + return *this; + } + ~TempFile() { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + } +}; + +template +void Append(std::vector& out, T value) { + static_assert(std::is_trivially_copyable_v); + const auto bytes = std::bit_cast>(value); + out.insert(out.end(), bytes.begin(), bytes.end()); +} + +inline void AppendString(std::vector& out, const std::string& value) { + Append(out, static_cast(value.size())); + for (const char c : value) out.push_back(static_cast(c)); +} + +inline std::uint64_t TensorBytes(const Tensor& tensor) { + std::uint64_t elements = 1; + for (const auto dimension : tensor.logical_shape) { + if (dimension != 0 && elements > std::numeric_limits::max() / dimension) + throw std::overflow_error("fixture tensor product"); + elements *= dimension; + } + if (tensor.type == kF32) return elements * 4; + if (tensor.type == kQ8_0) { + if (elements % 32 != 0) throw std::runtime_error("fixture Q8_0 divisibility"); + return elements / 32 * 34; + } + return elements; +} + +class Builder { +public: + Builder() { AddContractMetadata(); } + + Builder& Alignment(std::uint32_t alignment) { + alignment_ = alignment; + SetMetadata("general.alignment", alignment); + return *this; + } + + Builder& AddMetadata(std::string key, MetadataValue value) { + metadata_.emplace_back(std::move(key), std::move(value)); + return *this; + } + + Builder& SetMetadata(std::string key, MetadataValue value) { + for (auto& entry : metadata_) { + if (entry.first == key) { + entry.second = std::move(value); + return *this; + } + } + return AddMetadata(std::move(key), std::move(value)); + } + + Builder& RemoveMetadata(const std::string& key) { + std::erase_if(metadata_, [&](const auto& entry) { return entry.first == key; }); + return *this; + } + + Builder& AddTensor(std::string name, std::vector logical_shape, + std::uint32_t type) { + tensors_.push_back({std::move(name), std::move(logical_shape), type}); + return *this; + } + + Builder& AddExactFixtureTensors() { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_norm.weight", {3072}, kF32); + AddTensor("blk.0.ffn_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor("blk.0.attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor("blk.0.ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor("blk.0.ffn_down.weight", {3072, 8192}, kQ8_0); + AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& AddFullContractTensors(bool short_rope = true) { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + AddTensor(prefix + ".attn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".ffn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor(prefix + ".attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_down.weight", {3072, 8192}, kQ8_0); + } + if (short_rope) AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& MutateTensor(const std::string& name, std::uint32_t type, + std::vector shape) { + auto* tensor = FindTensor(name); + if (!tensor) throw std::runtime_error("fixture tensor not found: " + name); + tensor->type = type; + tensor->logical_shape = std::move(shape); + return *this; + } + + Builder& RemoveTensor(const std::string& name) { + std::erase_if(tensors_, [&](const Tensor& tensor) { return tensor.name == name; }); + return *this; + } + + Builder& AddEverySkippableMetadataType() { + AddMetadata("skip.u8", std::uint8_t{1}); + AddMetadata("skip.i8", std::int8_t{-1}); + AddMetadata("skip.u16", std::uint16_t{2}); + AddMetadata("skip.i16", std::int16_t{-2}); + AddMetadata("skip.u32", std::uint32_t{3}); + AddMetadata("skip.i32", std::int32_t{-3}); + AddMetadata("skip.f32", 1.25f); + AddMetadata("skip.bool", true); + AddMetadata("skip.string", std::string("value")); + std::vector strings; + AppendString(strings, "a"); AppendString(strings, "bc"); + AddMetadata("skip.array", ArrayValue{8, 2, std::move(strings)}); + AddMetadata("skip.u64", std::uint64_t{4}); + AddMetadata("skip.i64", std::int64_t{-4}); + AddMetadata("skip.f64", 2.5); + return *this; + } + + Builder& Apply(Mutation mutation) { mutation_ = mutation; return *this; } + + TempFile Write(std::string_view label = "fixture") const { + static std::uint64_t serial = 0; + auto path = std::filesystem::temp_directory_path() / + ("flm_phi4_" + std::string(label) + "_" + + std::to_string(++serial) + ".gguf"); + auto bytes = Encode(); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + if (!stream) throw std::runtime_error("cannot create fixture"); + stream.write(reinterpret_cast(bytes.prefix.data()), + static_cast(bytes.prefix.size())); + if (bytes.file_size > bytes.prefix.size()) { + stream.seekp(static_cast(bytes.file_size - 1)); + const char zero = 0; + stream.write(&zero, 1); + } + stream.close(); + return TempFile(path); + } + +private: + struct Encoded { std::vector prefix; std::uint64_t file_size; }; + + void AddContractMetadata() { + AddMetadata("general.architecture", std::string("phi3")); + AddMetadata("general.alignment", std::uint32_t{32}); + AddMetadata("phi3.block_count", std::uint32_t{32}); + AddMetadata("phi3.context_length", std::uint32_t{4096}); + AddMetadata("phi3.embedding_length", std::uint32_t{3072}); + AddMetadata("phi3.feed_forward_length", std::uint32_t{8192}); + AddMetadata("phi3.attention.head_count", std::uint32_t{24}); + AddMetadata("phi3.attention.head_count_kv", std::uint32_t{8}); + AddMetadata("phi3.attention.layer_norm_rms_epsilon", 1.0e-5f); + AddMetadata("phi3.rope.dimension_count", std::uint32_t{96}); + AddMetadata("phi3.rope.freq_base", 10000.0f); + AddMetadata("phi3.rope.scaling.attn_factor", 1.0f); + AddMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{4096}); + AddMetadata("tokenizer.ggml.tokens", + ArrayValue{0, 200064, std::vector(200064)}); + AddMetadata("tokenizer.ggml.add_bos_token", false); + AddMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{200020}); + } + + Tensor* FindTensor(const std::string& name) { + const auto it = std::find_if(tensors_.begin(), tensors_.end(), + [&](const Tensor& tensor) { return tensor.name == name; }); + return it == tensors_.end() ? nullptr : &*it; + } + + static std::uint32_t TypeOf(const MetadataValue& value) { + return static_cast(value.index()); + } + + static void EncodeValue(std::vector& out, const MetadataValue& value) { + std::visit([&](const auto& item) { + using T = std::decay_t; + if constexpr (std::is_same_v) AppendString(out, item); + else if constexpr (std::is_same_v) { + Append(out, item.element_type); Append(out, item.count); + out.insert(out.end(), item.encoded_elements.begin(), item.encoded_elements.end()); + } else if constexpr (std::is_same_v) Append(out, std::uint8_t(item)); + else Append(out, item); + }, value); + } + + Encoded Encode() const { + auto metadata = metadata_; + auto tensors = tensors_; + std::uint32_t alignment = alignment_; + if (mutation_ == Mutation::ZeroAlignment) alignment = 0; + if (mutation_ == Mutation::NonPowerOfTwoAlignment) alignment = 24; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = alignment; + if (mutation_ == Mutation::DuplicateName && !tensors.empty()) tensors.push_back(tensors.front()); + if (mutation_ == Mutation::DtypeMismatch && !tensors.empty()) tensors.front().type = kF32; + if (mutation_ == Mutation::ShapeMismatch && !tensors.empty()) tensors.front().logical_shape[0]--; + + std::vector out; + Append(out, std::uint32_t{0x46554747}); Append(out, std::uint32_t{3}); + Append(out, mutation_ == Mutation::CountOverflow ? std::numeric_limits::max() + : static_cast(tensors.size())); + Append(out, static_cast(metadata.size())); + for (const auto& [key, value] : metadata) { + AppendString(out, key); + if (mutation_ == Mutation::UnsupportedMetadataType && key == metadata.front().first) { + Append(out, std::uint32_t{99}); + } else { + Append(out, TypeOf(value)); EncodeValue(out, value); + } + } + std::uint64_t running = 0; + for (std::size_t index = 0; index < tensors.size(); ++index) { + auto& tensor = tensors[index]; + if (mutation_ == Mutation::ProductOverflow && index == 0) + tensor.logical_shape = { + static_cast(std::numeric_limits::max()), 3}; + if (tensor.explicit_offset) running = tensor.offset; + if (alignment != 0 && (alignment & (alignment - 1)) == 0) + running = (running + alignment - 1) & ~(std::uint64_t(alignment) - 1); + tensor.offset = running; + const auto size = mutation_ == Mutation::ProductOverflow && index == 0 + ? 0 : TensorBytes(tensor); + if (mutation_ == Mutation::OverlappingRanges && index == 1) { + tensor.offset = 0; + running += size; + } else if (mutation_ == Mutation::OutOfFileRange && index == 0) + tensor.offset = std::uint64_t{1} << 40; + else if (mutation_ == Mutation::OffsetOverflow && index == 0) + tensor.offset = std::numeric_limits::max() - 31; + else running += size; + AppendString(out, tensor.name); + Append(out, static_cast(tensor.logical_shape.size())); + for (auto it = tensor.logical_shape.rbegin(); it != tensor.logical_shape.rend(); ++it) + Append(out, *it); + Append(out, tensor.type); Append(out, tensor.offset); + } + if (mutation_ == Mutation::TruncatedDirectory && !out.empty()) { + out.pop_back(); return {std::move(out), static_cast(out.size())}; + } + const auto data_start = alignment == 0 ? static_cast(out.size()) + : (static_cast(out.size()) + alignment - 1) & ~(std::uint64_t(alignment) - 1); + out.resize(static_cast(data_start), std::byte{0}); + std::uint64_t file_size = data_start + running; + if (mutation_ == Mutation::PayloadLengthMismatch && file_size > data_start) --file_size; + if (mutation_ == Mutation::TruncatedString) { + const auto impossible = std::bit_cast>( + std::numeric_limits::max()); + std::copy(impossible.begin(), impossible.end(), out.begin() + 24); + } + return {std::move(out), file_size}; + } + + std::uint32_t alignment_ = 32; + std::vector> metadata_; + std::vector tensors_; + Mutation mutation_ = Mutation::None; +}; + +inline nlohmann::json ValidConfig() { + return {{"model_type", "phi3"}, {"num_hidden_layers", 32}, + {"hidden_size", 3072}, {"intermediate_size", 8192}, + {"num_attention_heads", 24}, {"num_key_value_heads", 8}, + {"head_dim", 128}, {"vocab_size", 200064}, + {"rms_norm_eps", 1.0e-5}, {"original_max_position_embeddings", 4096}, + {"eos_token_id", 199999}}; +} + +inline nlohmann::json ValidTokenizer() { + nlohmann::json vocab = nlohmann::json::object(); + for (int id = 0; id < 200062; ++id) vocab["t" + std::to_string(id)] = id; + vocab["<|endoftext|>"] = 199999; + vocab["<|end|>"] = 200020; + return {{"model", {{"vocab", std::move(vocab)}}}, + {"added_tokens", nlohmann::json::array({ + {{"id", 200062}, {"content", "added-a"}}, + {{"id", 200063}, {"content", "added-b"}}, + {{"id", 200020}, {"content", "<|end|>"}}, + {{"id", 199999}, {"content", "<|endoftext|>"}}})}}; +} + +inline nlohmann::json ValidTokenizerConfig() { + return {{"add_bos_token", false}, + {"chat_template", "<|user|>{{ message }}<|end|><|assistant|>"}}; +} + +} // namespace gguf_fixture diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp new file mode 100644 index 00000000..34749723 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -0,0 +1,336 @@ +#include "gguf_fixture.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::phi4::Phi4GgufPackage; +using gguf_fixture::Builder; +using gguf_fixture::Mutation; + +std::shared_ptr Open(Builder builder, + gguf_fixture::TempFile& file, + std::string_view label) { + file = builder.Write(label); + return Phi4GgufPackage::Open(file.path); +} + +std::string OpenFailure(Builder builder, Mutation mutation, + std::string_view label) { + auto file = builder.Apply(mutation).Write(label); + return RequireThrows([&] { Phi4GgufPackage::Open(file.path); }); +} + +Builder SplitFixture() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0) + .AddTensor("blk.0.ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0) + .AddTensor("f32", {48}, gguf_fixture::kF32); + return builder; +} + +void TestValidV3HeaderMetadataDirectoryAndAlignment() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "valid"); + const auto metadata = package->Metadata(); + TEST_REQUIRE(metadata.architecture == "phi3"); + TEST_REQUIRE(metadata.layer_count == 32); + TEST_REQUIRE(metadata.tokenizer_vocabulary_size == 200064); + TEST_REQUIRE(!metadata.add_bos_token); +} + +void TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().AddEverySkippableMetadataType(), file, + "metadata-types"); + TEST_REQUIRE(package->Metadata().hidden_size == 3072); +} + +void TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail() { + const auto truncated_header = std::filesystem::temp_directory_path() / "flm_phi4_short_header.gguf"; + { std::ofstream out(truncated_header, std::ios::binary | std::ios::trunc); out << "GG"; } + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(truncated_header); }), "header"); + std::error_code ignored; std::filesystem::remove(truncated_header, ignored); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedString, "truncated-string"), "string"); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedDirectory, "truncated-directory"), "tensor"); + + Builder array; + array.RemoveMetadata("tokenizer.ggml.tokens") + .AddMetadata("tokenizer.ggml.tokens", gguf_fixture::ArrayValue{ + 8, std::numeric_limits::max(), {}}) + .AddTensor("x", {32}, gguf_fixture::kQ8_0); + auto file = array.Write("truncated-array"); + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(file.path); }), "array"); +} + +void TestCountProductAlignmentAndOffsetOverflowFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::CountOverflow, "count-overflow"), "count"); + RequireContains(OpenFailure(SplitFixture(), Mutation::ProductOverflow, "product-overflow"), "overflow"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OffsetOverflow, "offset-overflow"), "overflow"); +} + +void TestZeroAndNonPowerOfTwoAlignmentFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::ZeroAlignment, "zero-align"), "alignment"); + RequireContains(OpenFailure(SplitFixture(), Mutation::NonPowerOfTwoAlignment, "bad-align"), "alignment"); +} + +void TestDuplicateTensorNamesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::DuplicateName, "duplicate"), "duplicate"); +} + +void TestOutOfFileAndOverlappingTensorRangesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::OutOfFileRange, "outside"), "range"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OverlappingRanges, "overlap"), "overlap"); + RequireContains(OpenFailure(SplitFixture(), Mutation::PayloadLengthMismatch, "short-payload"), "range"); +} + +void TestUnsupportedUnskippableMetadataTypeFails() { + RequireContains(OpenFailure(SplitFixture(), Mutation::UnsupportedMetadataType, "unsupported"), "metadata type"); +} + +void TestRequireQ8AndRequireF32ReportNameActualAndExpected() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "requires"); + auto error = RequireThrows([&] { package->RequireQ8("f32", std::array{48}); }); + RequireContains(error, "f32"); RequireContains(error, "actual F32"); RequireContains(error, "expected Q8_0"); + error = RequireThrows([&] { package->RequireF32("f32", std::array{47}); }); + RequireContains(error, "f32"); RequireContains(error, "48"); RequireContains(error, "47"); +} + +void TestAttentionQkvReturnsThreeZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "qkv"); + const auto fused = package->RequireQ8("blk.0.attn_qkv.weight", std::array{5120, 3072}); + const auto views = package->AttentionQkv(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 3); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 3072 * row_bytes); + TEST_REQUIRE(views.values[2].bytes.data() == fused.bytes.data() + 4096 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({3072, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({1024, 3072})); + TEST_REQUIRE(views.values[2].logical_shape == std::vector({1024, 3072})); +} + +void TestGateUpReturnsTwoZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "gate-up"); + const auto fused = package->RequireQ8("blk.0.ffn_up.weight", std::array{16384, 3072}); + const auto views = package->GateUp(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 2); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 8192 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({8192, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({8192, 3072})); +} + +void TestSplitRejectsNonIntegralQ8RowBoundary() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3073}, gguf_fixture::kQ8_0); + auto file = builder.Write("bad-row"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { package->AttentionQkv(0); }), "row"); +} + +void TestViewsPointIntoTheReadOnlyMapping() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "mapping"); + const auto first = package->RequireF32("f32", std::array{48}); + const auto second = package->RequireF32("f32", std::array{48}); + TEST_REQUIRE(first.values.data() == second.values.data()); + TEST_REQUIRE(first.values.size() == 48); +} + +struct ContractFixture { + gguf_fixture::TempFile file; + std::shared_ptr package; + ContractFixture() { + package = Open(Builder().AddFullContractTensors(), file, "contract"); + } +}; + +void TestAcceptsExactPhi3Phi4Contract() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsWrongArchitectureAndEveryDimension() { + const std::vector> cases = { + {"general.architecture", std::string("llama")}, {"phi3.block_count", std::uint32_t{31}}, + {"phi3.context_length", std::uint32_t{4095}}, {"phi3.embedding_length", std::uint32_t{3071}}, + {"phi3.feed_forward_length", std::uint32_t{8191}}, {"phi3.attention.head_count", std::uint32_t{23}}, + {"phi3.attention.head_count_kv", std::uint32_t{7}}, {"phi3.rope.dimension_count", std::uint32_t{95}}, + {"tokenizer.ggml.tokens", gguf_fixture::ArrayValue{0, 200063, std::vector(200063)}}}; + for (const auto& [field, value] : cases) { + auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("wrong-field"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, field); RequireContains(error, "actual"); RequireContains(error, "expected"); + } +} + +void TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole() { + ContractFixture valid; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + valid.package->RequireF32(prefix + ".attn_norm.weight", std::array{3072}); + valid.package->RequireF32(prefix + ".ffn_norm.weight", std::array{3072}); + valid.package->RequireQ8(prefix + ".attn_qkv.weight", std::array{5120, 3072}); + valid.package->RequireQ8(prefix + ".attn_output.weight", std::array{3072, 3072}); + valid.package->RequireQ8(prefix + ".ffn_up.weight", std::array{16384, 3072}); + valid.package->RequireQ8(prefix + ".ffn_down.weight", std::array{3072, 8192}); + } + std::vector required_names = {"token_embd.weight", "output_norm.weight"}; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + required_names.push_back(prefix + ".attn_norm.weight"); + required_names.push_back(prefix + ".ffn_norm.weight"); + required_names.push_back(prefix + ".attn_qkv.weight"); + required_names.push_back(prefix + ".attn_output.weight"); + required_names.push_back(prefix + ".ffn_up.weight"); + required_names.push_back(prefix + ".ffn_down.weight"); + } + const nlohmann::json unused; + for (const auto& name : required_names) { + auto file = Builder().AddFullContractTensors().RemoveTensor(name).Write("missing-role"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { + package->ValidatePhi4Contract(unused, unused, unused); + }); + RequireContains(error, name); RequireContains(error, "actual missing"); RequireContains(error, "expected"); + } + auto type_file = Builder().AddFullContractTensors().MutateTensor("blk.0.attn_output.weight", gguf_fixture::kF32, {3072,3072}).Write("wrong-type"); + auto type_package = Phi4GgufPackage::Open(type_file.path); + auto error = RequireThrows([&] { type_package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, "blk.0.attn_output.weight"); RequireContains(error, "actual F32"); RequireContains(error, "expected Q8_0"); + auto shape_file = Builder().AddFullContractTensors().MutateTensor("blk.0.ffn_down.weight", gguf_fixture::kQ8_0, {3072,8160}).Write("wrong-shape"); + auto shape_package = Phi4GgufPackage::Open(shape_file.path); + error = RequireThrows([&] { shape_package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, "blk.0.ffn_down.weight"); RequireContains(error, "actual"); RequireContains(error, "expected"); + RequireContains(OpenFailure(Builder().AddTensor("x", {32}, gguf_fixture::kQ8_0), Mutation::PayloadLengthMismatch, "wrong-length"), "range"); +} + +void TestRejectsMixedQuantizationAndOutputWeightPresence() { + auto mixed_file = Builder().AddFullContractTensors().MutateTensor("token_embd.weight", gguf_fixture::kF32, {200064,3072}).Write("mixed"); + auto mixed = Phi4GgufPackage::Open(mixed_file.path); + RequireContains(RequireThrows([&] { mixed->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "token_embd.weight"); + auto output_file = Builder().AddFullContractTensors().AddTensor("output.weight", {200064,3072}, gguf_fixture::kQ8_0).Write("output-weight"); + auto output = Phi4GgufPackage::Open(output_file.path); + const auto error = RequireThrows([&] { output->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, "output.weight"); RequireContains(error, "actual present"); RequireContains(error, "expected absent"); +} + +void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { + auto file = Builder().AddFullContractTensors().RemoveTensor("token_embd.weight").Write("untied"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "token_embd.weight"); +} + +void TestRequiresOriginal4096WindowAndRejectsLongRopeBranch() { + auto wrong_file = Builder().SetMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{8192}).AddFullContractTensors().Write("long-window"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireContains(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "phi3.rope.scaling.original_context_length"); + auto long_file = Builder().AddFullContractTensors().AddTensor("rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); + auto long_rope = Phi4GgufPackage::Open(long_file.path); + RequireContains(RequireThrows([&] { long_rope->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "rope_factors_long.weight"); +} + +void TestValidatesOptionalShortRopeFactorsAsF32Length48() { + auto absent_file = Builder().AddFullContractTensors(false).Write("no-short-rope"); + auto absent = Phi4GgufPackage::Open(absent_file.path); + absent->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + auto wrong_file = Builder().AddFullContractTensors(false).AddTensor("rope_factors_short.weight", {47}, gguf_fixture::kF32).Write("wrong-short-rope"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireContains(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "rope_factors_short.weight"); +} + +void TestRejectsNonFiniteOrNonPositiveRopeValues() { + for (const auto& field : {"phi3.rope.freq_base", "phi3.rope.scaling.attn_factor"}) { + for (const float value : {0.0f, -1.0f, std::numeric_limits::infinity(), std::numeric_limits::quiet_NaN()}) { + auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("bad-rope-value"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), field); + } + } +} + +void TestRejectsConfigDisagreement() { + ContractFixture fixture; + const std::vector> cases = { + {"model_type", "other"}, {"num_hidden_layers", 31}, {"hidden_size", 3071}, + {"intermediate_size", 8191}, {"num_attention_heads", 23}, {"num_key_value_heads", 7}, + {"head_dim", 127}, {"vocab_size", 200063}, {"rms_norm_eps", 2.0e-5}, + {"original_max_position_embeddings", 4095}}; + for (const auto& [field, value] : cases) { + auto config = gguf_fixture::ValidConfig(); config[field] = value; + const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract(config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, field); RequireContains(error, "actual"); RequireContains(error, "expected"); + } +} + +void TestDerivesStopSetFromGgufConfigAndTokenizerIds() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { + ContractFixture fixture; + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"]["<|end|>"] = 1; + RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), tokenizer, gguf_fixture::ValidTokenizerConfig()); }), "<|end|>"); + auto config = gguf_fixture::ValidConfig(); config["eos_token_id"] = 1; + RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "eos_token_id"); + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); tokenizer_config["add_bos_token"] = true; + RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); }), "add_bos_token"); + tokenizer_config = gguf_fixture::ValidTokenizerConfig(); tokenizer_config["chat_template"] = "<|user|><|assistant|>"; + RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); }), "<|end|>"); +} + +void TestValidationCreatesNoCorelibObjects() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); +} +} // namespace + +int main() { +#define RUN(name) RunTest(name, #name) + RUN(TestValidV3HeaderMetadataDirectoryAndAlignment); + RUN(TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely); + RUN(TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail); + RUN(TestCountProductAlignmentAndOffsetOverflowFail); + RUN(TestZeroAndNonPowerOfTwoAlignmentFail); + RUN(TestDuplicateTensorNamesFail); + RUN(TestOutOfFileAndOverlappingTensorRangesFail); + RUN(TestUnsupportedUnskippableMetadataTypeFails); + RUN(TestRequireQ8AndRequireF32ReportNameActualAndExpected); + RUN(TestAttentionQkvReturnsThreeZeroCopyWholeRowViews); + RUN(TestGateUpReturnsTwoZeroCopyWholeRowViews); + RUN(TestSplitRejectsNonIntegralQ8RowBoundary); + RUN(TestViewsPointIntoTheReadOnlyMapping); + RUN(TestAcceptsExactPhi3Phi4Contract); + RUN(TestRejectsWrongArchitectureAndEveryDimension); + RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); + RUN(TestRejectsMixedQuantizationAndOutputWeightPresence); + RUN(TestRequiresTiedQ8TokenEmbeddingAsLmHead); + RUN(TestRequiresOriginal4096WindowAndRejectsLongRopeBranch); + RUN(TestValidatesOptionalShortRopeFactorsAsF32Length48); + RUN(TestRejectsNonFiniteOrNonPositiveRopeValues); + RUN(TestRejectsConfigDisagreement); + RUN(TestDerivesStopSetFromGgufConfigAndTokenizerIds); + RUN(TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement); + RUN(TestValidationCreatesNoCorelibObjects); +#undef RUN + return 0; +} From 5e6423b4ba54aaaa94cb2dabd52df776b44b5ffa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 03:28:13 -0700 Subject: [PATCH 06/37] fix: harden Phi-4 GGUF validation --- src/common/corelib/phi4_corelib_gguf.cpp | 102 ++++--- src/test/phi4_corelib_aie4/CMakeLists.txt | 8 +- src/test/phi4_corelib_aie4/gguf_fixture.hpp | 35 ++- src/test/phi4_corelib_aie4/test_phi4_gguf.cpp | 251 +++++++++++++++--- 4 files changed, 317 insertions(+), 79 deletions(-) diff --git a/src/common/corelib/phi4_corelib_gguf.cpp b/src/common/corelib/phi4_corelib_gguf.cpp index 019614d2..1433d2cf 100644 --- a/src/common/corelib/phi4_corelib_gguf.cpp +++ b/src/common/corelib/phi4_corelib_gguf.cpp @@ -208,17 +208,40 @@ std::string JsonText(const nlohmann::json& value) { return value.dump(); } -template -void RequireJson(const nlohmann::json& object, std::string_view key, - const T& expected) { +void RequireJsonString(const nlohmann::json& object, std::string_view key, + std::string_view expected) { const auto it = object.find(std::string(key)); - if (it == object.end()) Fail(key, "missing", nlohmann::json(expected).dump()); - try { - if (it->template get() != expected) - Fail(key, JsonText(*it), nlohmann::json(expected).dump()); - } catch (const nlohmann::json::exception&) { - Fail(key, JsonText(*it), nlohmann::json(expected).dump()); + if (it == object.end() || !it->is_string()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::string(expected)); + const auto actual = it->get_ref(); + if (actual != expected) Fail(key, actual, std::string(expected)); +} + +void RequireJsonBoolean(const nlohmann::json& object, std::string_view key, + bool expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_boolean()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), expected ? "true" : "false"); + const auto actual = it->get(); + if (actual != expected) Fail(key, actual ? "true" : "false", expected ? "true" : "false"); +} + +void RequireJsonUnsigned(const nlohmann::json& object, std::string_view key, + std::uint64_t expected) { + const auto it = object.find(std::string(key)); + if (it == object.end()) Fail(key, "missing", std::to_string(expected)); + std::uint64_t actual; + if (it->is_number_unsigned()) { + actual = it->get(); + } else if (it->is_number_integer()) { + const auto signed_value = it->get(); + if (signed_value < 0) + Fail(key, JsonText(*it), "non-negative integer " + std::to_string(expected)); + actual = static_cast(signed_value); + } else { + Fail(key, JsonText(*it), "integer " + std::to_string(expected)); } + if (actual != expected) Fail(key, std::to_string(actual), std::to_string(expected)); } void RequireJsonDouble(const nlohmann::json& object, std::string_view key, @@ -432,6 +455,9 @@ FloatTensorView Phi4GgufPackage::RequireF32( const auto expected_length = TensorByteLength(kTypeF32, expected_shape, name); if (tensor.bytes.size() != expected_length) Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + const auto address = reinterpret_cast(tensor.bytes.data()); + if (tensor.absolute_offset % alignof(float) != 0 || address % alignof(float) != 0) + Fail(name, "address/offset not aligned", "alignment 4"); return {tensor.name, {reinterpret_cast(tensor.bytes.data()), tensor.bytes.size() / sizeof(float)}, @@ -533,21 +559,36 @@ void Phi4GgufPackage::ValidatePhi4Contract( if (impl_->tensors.contains("rope_factors_short.weight")) RequireF32("rope_factors_short.weight", std::array{48}); - RequireJson(config, "model_type", std::string("phi3")); - RequireJson(config, "num_hidden_layers", int(kLayerCount)); - RequireJson(config, "hidden_size", int(kHiddenSize)); - RequireJson(config, "intermediate_size", int(kIntermediateSize)); - RequireJson(config, "num_attention_heads", int(kQueryHeadCount)); - RequireJson(config, "num_key_value_heads", int(kKvHeadCount)); - RequireJson(config, "head_dim", int(kHeadSize)); - RequireJson(config, "vocab_size", int(kVocabularySize)); + RequireJsonString(config, "model_type", "phi3"); + RequireJsonUnsigned(config, "num_hidden_layers", kLayerCount); + RequireJsonUnsigned(config, "hidden_size", kHiddenSize); + RequireJsonUnsigned(config, "intermediate_size", kIntermediateSize); + RequireJsonUnsigned(config, "num_attention_heads", kQueryHeadCount); + RequireJsonUnsigned(config, "num_key_value_heads", kKvHeadCount); + RequireJsonUnsigned(config, "head_dim", kHeadSize); + RequireJsonUnsigned(config, "vocab_size", kVocabularySize); RequireJsonDouble(config, "rms_norm_eps", 1.0e-5); - RequireJson(config, "original_max_position_embeddings", int(kMaxSequenceLength)); - RequireJson(config, "eos_token_id", 199999); + RequireJsonUnsigned(config, "original_max_position_embeddings", kMaxSequenceLength); + RequireJsonUnsigned(config, "eos_token_id", 199999); std::set vocabulary_ids; std::map> token_ids; - const auto add_token = [&](const std::string& token, std::int64_t id) { + const auto add_token = [&](const std::string& token, const nlohmann::json& encoded_id) { + const std::string field = "tokenizer.json token ID " + token; + std::uint64_t unsigned_id; + if (encoded_id.is_number_unsigned()) { + unsigned_id = encoded_id.get(); + } else if (encoded_id.is_number_integer()) { + const auto signed_id = encoded_id.get(); + if (signed_id < 0) + Fail(field, std::to_string(signed_id), "0..200063"); + unsigned_id = static_cast(signed_id); + } else { + Fail(field, JsonText(encoded_id), "integer in 0..200063"); + } + if (unsigned_id >= static_cast(kVocabularySize)) + Fail(field, std::to_string(unsigned_id), "0..200063"); + const auto id = static_cast(unsigned_id); const auto [it, inserted] = token_ids.emplace(token, id); if (!inserted && it->second != id) Fail(token, std::to_string(id), std::to_string(it->second)); @@ -556,28 +597,27 @@ void Phi4GgufPackage::ValidatePhi4Contract( try { const auto& vocab = tokenizer.at("model").at("vocab"); if (!vocab.is_object()) Fail("tokenizer.json model.vocab", JsonText(vocab), "object mapping tokens to IDs"); - for (auto it = vocab.begin(); it != vocab.end(); ++it) { - const auto id = it.value().get(); - add_token(it.key(), id); - } + for (auto it = vocab.begin(); it != vocab.end(); ++it) + add_token(it.key(), it.value()); const auto added = tokenizer.find("added_tokens"); if (added != tokenizer.end()) { if (!added->is_array()) Fail("tokenizer.json added_tokens", JsonText(*added), "array"); for (const auto& item : *added) { - const auto id = item.at("id").get(); const auto content = item.at("content").get(); - add_token(content, id); + add_token(content, item.at("id")); } } } catch (const nlohmann::json::exception& error) { Fail("tokenizer.json vocabulary", error.what(), "valid token-to-ID mappings"); } const auto actual_count = vocabulary_ids.size(); - const auto actual_max_plus_one = vocabulary_ids.empty() ? 0 : *vocabulary_ids.rbegin() + 1; + const auto actual_max = vocabulary_ids.empty() ? -1 : *vocabulary_ids.rbegin(); + if (actual_max != kVocabularySize - 1) + Fail("tokenizer.json maximum vocabulary ID", std::to_string(actual_max), + std::to_string(kVocabularySize - 1)); if (actual_count != static_cast(kVocabularySize)) - Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), std::to_string(kVocabularySize)); - if (actual_max_plus_one != kVocabularySize) - Fail("tokenizer.json maximum ID plus one", std::to_string(actual_max_plus_one), std::to_string(kVocabularySize)); + Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), + std::to_string(kVocabularySize)); for (const auto& [token, expected] : std::array{ std::pair{"<|end|>", 200020}, std::pair{"<|endoftext|>", 199999}}) { @@ -588,7 +628,7 @@ void Phi4GgufPackage::ValidatePhi4Contract( const auto gguf_eos = impl_->Unsigned("tokenizer.ggml.eos_token_id"); if (gguf_eos != 200020) Fail("tokenizer.ggml.eos_token_id", std::to_string(gguf_eos), "200020"); - RequireJson(tokenizer_config, "add_bos_token", false); + RequireJsonBoolean(tokenizer_config, "add_bos_token", false); const auto template_it = tokenizer_config.find("chat_template"); if (template_it == tokenizer_config.end() || !template_it->is_string()) Fail("chat_template", template_it == tokenizer_config.end() ? "missing" : JsonText(*template_it), "string containing Phi-4 markers"); diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index bfbfccc0..688182d7 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -61,10 +61,14 @@ target_include_directories(test_real_corelib PRIVATE target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) add_executable(test_phi4_gguf - test_phi4_gguf.cpp "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp") + test_phi4_gguf.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp") target_include_directories(test_phi4_gguf PRIVATE "${CMAKE_CURRENT_LIST_DIR}" - "${FLM_SOURCE_DIR}/include") + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_gguf PRIVATE RYZENAI_CORELIB_STATIC=1) # Compile the actual production frontend translation unit in both feature modes. # Empty declaration-only FFmpeg headers isolate this compile check from an diff --git a/src/test/phi4_corelib_aie4/gguf_fixture.hpp b/src/test/phi4_corelib_aie4/gguf_fixture.hpp index d3d3dcd4..ec460214 100644 --- a/src/test/phi4_corelib_aie4/gguf_fixture.hpp +++ b/src/test/phi4_corelib_aie4/gguf_fixture.hpp @@ -36,6 +36,7 @@ enum class Mutation { DtypeMismatch, ShapeMismatch, PayloadLengthMismatch, + MisalignedF32, }; struct ArrayValue { @@ -186,6 +187,11 @@ class Builder { return *this; } + Builder& TruncateTensorPayload(std::string name) { + truncated_tensor_ = std::move(name); + return *this; + } + Builder& AddEverySkippableMetadataType() { AddMetadata("skip.u8", std::uint8_t{1}); AddMetadata("skip.i8", std::int8_t{-1}); @@ -282,6 +288,20 @@ class Builder { if (mutation_ == Mutation::DuplicateName && !tensors.empty()) tensors.push_back(tensors.front()); if (mutation_ == Mutation::DtypeMismatch && !tensors.empty()) tensors.front().type = kF32; if (mutation_ == Mutation::ShapeMismatch && !tensors.empty()) tensors.front().logical_shape[0]--; + if (!truncated_tensor_.empty()) { + const auto it = std::find_if(tensors.begin(), tensors.end(), [&](const Tensor& tensor) { + return tensor.name == truncated_tensor_; + }); + if (it == tensors.end()) throw std::runtime_error("fixture tensor not found: " + truncated_tensor_); + Tensor target = std::move(*it); + tensors.erase(it); + tensors.push_back(std::move(target)); + } + if (mutation_ == Mutation::MisalignedF32) { + alignment = 1; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = std::uint32_t{1}; + } std::vector out; Append(out, std::uint32_t{0x46554747}); Append(out, std::uint32_t{3}); @@ -297,6 +317,7 @@ class Builder { } } std::uint64_t running = 0; + std::vector encoded_offset_positions; for (std::size_t index = 0; index < tensors.size(); ++index) { auto& tensor = tensors[index]; if (mutation_ == Mutation::ProductOverflow && index == 0) @@ -320,16 +341,25 @@ class Builder { Append(out, static_cast(tensor.logical_shape.size())); for (auto it = tensor.logical_shape.rbegin(); it != tensor.logical_shape.rend(); ++it) Append(out, *it); - Append(out, tensor.type); Append(out, tensor.offset); + Append(out, tensor.type); + encoded_offset_positions.push_back(out.size()); + Append(out, tensor.offset); } if (mutation_ == Mutation::TruncatedDirectory && !out.empty()) { out.pop_back(); return {std::move(out), static_cast(out.size())}; } const auto data_start = alignment == 0 ? static_cast(out.size()) : (static_cast(out.size()) + alignment - 1) & ~(std::uint64_t(alignment) - 1); + if (mutation_ == Mutation::MisalignedF32 && !tensors.empty()) { + const std::uint64_t offset = (1 + alignof(float) - data_start % alignof(float)) % alignof(float); + const auto encoded = std::bit_cast>(offset); + std::copy(encoded.begin(), encoded.end(), out.begin() + encoded_offset_positions.front()); + running = std::max(running, offset + TensorBytes(tensors.front())); + } out.resize(static_cast(data_start), std::byte{0}); std::uint64_t file_size = data_start + running; - if (mutation_ == Mutation::PayloadLengthMismatch && file_size > data_start) --file_size; + if ((mutation_ == Mutation::PayloadLengthMismatch || !truncated_tensor_.empty()) && + file_size > data_start) --file_size; if (mutation_ == Mutation::TruncatedString) { const auto impossible = std::bit_cast>( std::numeric_limits::max()); @@ -342,6 +372,7 @@ class Builder { std::vector> metadata_; std::vector tensors_; Mutation mutation_ = Mutation::None; + std::string truncated_tensor_; }; inline nlohmann::json ValidConfig() { diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp index 34749723..c45103a7 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -1,4 +1,6 @@ #include "gguf_fixture.hpp" +#include "fake_corelib.hpp" +#include "corelib/corelib_api.hpp" #include "models/phi4/phi4_corelib_constants.hpp" #include "models/phi4/phi4_corelib_gguf.hpp" #include "test_support.hpp" @@ -10,6 +12,7 @@ #include #include #include +#include #include namespace { @@ -30,6 +33,41 @@ std::string OpenFailure(Builder builder, Mutation mutation, return RequireThrows([&] { Phi4GgufPackage::Open(file.path); }); } +void RequireMismatch(std::string_view error, std::string_view field, + std::string_view actual, std::string_view expected) { + RequireContains(error, field); + RequireContains(error, "actual " + std::string(actual)); + RequireContains(error, "expected " + std::string(expected)); +} + +void RequireDiagnostic(std::string_view error, std::string_view field) { + RequireContains(error, field); + RequireContains(error, "actual"); + RequireContains(error, "expected"); +} + +struct TensorRole { + std::string name; + std::vector shape; + std::uint32_t type; +}; + +std::vector RequiredTensorRoles() { + std::vector roles = { + {"token_embd.weight", {200064, 3072}, gguf_fixture::kQ8_0}, + {"output_norm.weight", {3072}, gguf_fixture::kF32}}; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + roles.push_back({prefix + ".attn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".ffn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".attn_output.weight", {3072, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_down.weight", {3072, 8192}, gguf_fixture::kQ8_0}); + } + return roles; +} + Builder SplitFixture() { Builder builder; builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0) @@ -149,6 +187,14 @@ void TestViewsPointIntoTheReadOnlyMapping() { const auto second = package->RequireF32("f32", std::array{48}); TEST_REQUIRE(first.values.data() == second.values.data()); TEST_REQUIRE(first.values.size() == 48); + + auto misaligned_file = Builder().AddTensor("misaligned-f32", {48}, gguf_fixture::kF32) + .Apply(Mutation::MisalignedF32).Write("misaligned-f32"); + auto misaligned = Phi4GgufPackage::Open(misaligned_file.path); + const auto error = RequireThrows([&] { + misaligned->RequireF32("misaligned-f32", std::array{48}); + }); + RequireMismatch(error, "misaligned-f32", "address", "alignment 4"); } struct ContractFixture { @@ -177,7 +223,7 @@ void TestRejectsWrongArchitectureAndEveryDimension() { auto package = Phi4GgufPackage::Open(file.path); const auto error = RequireThrows([&] { package->ValidatePhi4Contract( gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, field); RequireContains(error, "actual"); RequireContains(error, "expected"); + RequireDiagnostic(error, field); } } @@ -192,40 +238,47 @@ void TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole() { valid.package->RequireQ8(prefix + ".ffn_up.weight", std::array{16384, 3072}); valid.package->RequireQ8(prefix + ".ffn_down.weight", std::array{3072, 8192}); } - std::vector required_names = {"token_embd.weight", "output_norm.weight"}; - for (std::size_t layer = 0; layer < 32; ++layer) { - const auto prefix = "blk." + std::to_string(layer); - required_names.push_back(prefix + ".attn_norm.weight"); - required_names.push_back(prefix + ".ffn_norm.weight"); - required_names.push_back(prefix + ".attn_qkv.weight"); - required_names.push_back(prefix + ".attn_output.weight"); - required_names.push_back(prefix + ".ffn_up.weight"); - required_names.push_back(prefix + ".ffn_down.weight"); - } const nlohmann::json unused; - for (const auto& name : required_names) { - auto file = Builder().AddFullContractTensors().RemoveTensor(name).Write("missing-role"); - auto package = Phi4GgufPackage::Open(file.path); - const auto error = RequireThrows([&] { - package->ValidatePhi4Contract(unused, unused, unused); + for (const auto& role : RequiredTensorRoles()) { + auto missing_file = Builder().AddFullContractTensors().RemoveTensor(role.name).Write("missing-role"); + auto missing = Phi4GgufPackage::Open(missing_file.path); + RequireMismatch(RequireThrows([&] { missing->ValidatePhi4Contract(unused, unused, unused); }), + role.name, "missing", "present tensor"); + + const auto wrong_type = role.type == gguf_fixture::kQ8_0 + ? gguf_fixture::kF32 : gguf_fixture::kQ8_0; + auto type_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, wrong_type, role.shape).Write("wrong-type"); + auto type_package = Phi4GgufPackage::Open(type_file.path); + RequireMismatch(RequireThrows([&] { type_package->ValidatePhi4Contract(unused, unused, unused); }), + role.name, wrong_type == gguf_fixture::kF32 ? "F32" : "Q8_0", + role.type == gguf_fixture::kF32 ? "F32" : "Q8_0"); + + auto wrong_shape = role.shape; + --wrong_shape.front(); + auto shape_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, role.type, wrong_shape).Write("wrong-shape"); + auto shape_package = Phi4GgufPackage::Open(shape_file.path); + const auto shape_error = RequireThrows([&] { + shape_package->ValidatePhi4Contract(unused, unused, unused); }); - RequireContains(error, name); RequireContains(error, "actual missing"); RequireContains(error, "expected"); + RequireContains(shape_error, role.name); + RequireContains(shape_error, "actual ["); + RequireContains(shape_error, "expected ["); + + auto length_file = Builder().AddFullContractTensors() + .TruncateTensorPayload(role.name).Write("wrong-length"); + const auto length_error = RequireThrows([&] { Phi4GgufPackage::Open(length_file.path); }); + RequireMismatch(length_error, role.name + " range", "out-of-file range", + "range within mapped file"); } - auto type_file = Builder().AddFullContractTensors().MutateTensor("blk.0.attn_output.weight", gguf_fixture::kF32, {3072,3072}).Write("wrong-type"); - auto type_package = Phi4GgufPackage::Open(type_file.path); - auto error = RequireThrows([&] { type_package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, "blk.0.attn_output.weight"); RequireContains(error, "actual F32"); RequireContains(error, "expected Q8_0"); - auto shape_file = Builder().AddFullContractTensors().MutateTensor("blk.0.ffn_down.weight", gguf_fixture::kQ8_0, {3072,8160}).Write("wrong-shape"); - auto shape_package = Phi4GgufPackage::Open(shape_file.path); - error = RequireThrows([&] { shape_package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, "blk.0.ffn_down.weight"); RequireContains(error, "actual"); RequireContains(error, "expected"); - RequireContains(OpenFailure(Builder().AddTensor("x", {32}, gguf_fixture::kQ8_0), Mutation::PayloadLengthMismatch, "wrong-length"), "range"); } void TestRejectsMixedQuantizationAndOutputWeightPresence() { auto mixed_file = Builder().AddFullContractTensors().MutateTensor("token_embd.weight", gguf_fixture::kF32, {200064,3072}).Write("mixed"); auto mixed = Phi4GgufPackage::Open(mixed_file.path); - RequireContains(RequireThrows([&] { mixed->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "token_embd.weight"); + RequireMismatch(RequireThrows([&] { mixed->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "F32", "Q8_0"); auto output_file = Builder().AddFullContractTensors().AddTensor("output.weight", {200064,3072}, gguf_fixture::kQ8_0).Write("output-weight"); auto output = Phi4GgufPackage::Open(output_file.path); const auto error = RequireThrows([&] { output->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); @@ -235,16 +288,19 @@ void TestRejectsMixedQuantizationAndOutputWeightPresence() { void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { auto file = Builder().AddFullContractTensors().RemoveTensor("token_embd.weight").Write("untied"); auto package = Phi4GgufPackage::Open(file.path); - RequireContains(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "token_embd.weight"); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "missing", "present tensor"); } void TestRequiresOriginal4096WindowAndRejectsLongRopeBranch() { auto wrong_file = Builder().SetMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{8192}).AddFullContractTensors().Write("long-window"); auto wrong = Phi4GgufPackage::Open(wrong_file.path); - RequireContains(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "phi3.rope.scaling.original_context_length"); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "phi3.rope.scaling.original_context_length", "8192", "4096"); auto long_file = Builder().AddFullContractTensors().AddTensor("rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); auto long_rope = Phi4GgufPackage::Open(long_file.path); - RequireContains(RequireThrows([&] { long_rope->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "rope_factors_long.weight"); + RequireMismatch(RequireThrows([&] { long_rope->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_long.weight", "present", "absent"); } void TestValidatesOptionalShortRopeFactorsAsF32Length48() { @@ -253,7 +309,8 @@ void TestValidatesOptionalShortRopeFactorsAsF32Length48() { absent->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); auto wrong_file = Builder().AddFullContractTensors(false).AddTensor("rope_factors_short.weight", {47}, gguf_fixture::kF32).Write("wrong-short-rope"); auto wrong = Phi4GgufPackage::Open(wrong_file.path); - RequireContains(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "rope_factors_short.weight"); + RequireDiagnostic(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_short.weight"); } void TestRejectsNonFiniteOrNonPositiveRopeValues() { @@ -261,7 +318,8 @@ void TestRejectsNonFiniteOrNonPositiveRopeValues() { for (const float value : {0.0f, -1.0f, std::numeric_limits::infinity(), std::numeric_limits::quiet_NaN()}) { auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("bad-rope-value"); auto package = Phi4GgufPackage::Open(file.path); - RequireContains(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), field); + RequireDiagnostic(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + field); } } } @@ -272,36 +330,140 @@ void TestRejectsConfigDisagreement() { {"model_type", "other"}, {"num_hidden_layers", 31}, {"hidden_size", 3071}, {"intermediate_size", 8191}, {"num_attention_heads", 23}, {"num_key_value_heads", 7}, {"head_dim", 127}, {"vocab_size", 200063}, {"rms_norm_eps", 2.0e-5}, - {"original_max_position_embeddings", 4095}}; + {"original_max_position_embeddings", 4095}, + {"hidden_size", 3072.0}, + {"hidden_size", std::uint64_t{4294970368ULL}}, + {"eos_token_id", 199999.0}}; for (const auto& [field, value] : cases) { auto config = gguf_fixture::ValidConfig(); config[field] = value; const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract(config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, field); RequireContains(error, "actual"); RequireContains(error, "expected"); + RequireDiagnostic(error, field); } } void TestDerivesStopSetFromGgufConfigAndTokenizerIds() { ContractFixture fixture; - fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + + auto gguf_file = Builder().SetMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{1}) + .AddFullContractTensors().Write("wrong-gguf-eos"); + auto gguf = Phi4GgufPackage::Open(gguf_file.path); + RequireMismatch(RequireThrows([&] { gguf->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.eos_token_id", "1", "200020"); + + auto config = gguf_fixture::ValidConfig(); + config["eos_token_id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "eos_token_id", "1", "199999"); + + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"][token] = 1; + for (auto& added : tokenizer["added_tokens"]) + if (added["content"] == token) added["id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + token, "1", std::to_string(expected)); + } } void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { ContractFixture fixture; + auto tokenizer = gguf_fixture::ValidTokenizer(); - tokenizer["model"]["vocab"]["<|end|>"] = 1; - RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), tokenizer, gguf_fixture::ValidTokenizerConfig()); }), "<|end|>"); - auto config = gguf_fixture::ValidConfig(); config["eos_token_id"] = 1; - RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "eos_token_id"); - auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); tokenizer_config["add_bos_token"] = true; - RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); }), "add_bos_token"); - tokenizer_config = gguf_fixture::ValidTokenizerConfig(); tokenizer_config["chat_template"] = "<|user|><|assistant|>"; - RequireContains(RequireThrows([&] { fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); }), "<|end|>"); + tokenizer["model"]["vocab"].erase("t0"); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json distinct vocabulary ID count", "200063", "200064"); + + for (const auto& [invalid_id, actual, expected] : std::array{ + std::tuple{-1, "-1", "0..200063"}, + std::tuple{200064, "200064", "0..200063"}, + std::tuple{ + std::numeric_limits::max(), "18446744073709551615", "0..200063"}, + std::tuple{0.0, "0.0", "integer in 0..200063"}}) { + tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"]["t0"] = invalid_id; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json token ID", actual, expected); + } + + tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["added_tokens"].erase(tokenizer["added_tokens"].begin() + 1); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json maximum vocabulary ID", "200062", "200063"); + + auto bos_file = Builder().SetMetadata("tokenizer.ggml.add_bos_token", true) + .AddFullContractTensors().Write("wrong-gguf-bos"); + auto bos = Phi4GgufPackage::Open(bos_file.path); + RequireMismatch(RequireThrows([&] { bos->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.add_bos_token", "true", "false"); + + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["add_bos_token"] = true; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "add_bos_token", "true", "false"); + tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = "<|user|><|assistant|>"; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "<|end|>", "missing from chat_template", "present in chat_template"); +} + +void TestRejectsFiniteWrongRmsValue() { + auto file = Builder().SetMetadata("phi3.attention.layer_norm_rms_epsilon", 2.0e-5f) + .AddFullContractTensors().Write("wrong-rms"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }); + RequireContains(error, "phi3.attention.layer_norm_rms_epsilon"); + RequireContains(error, "actual 0.000020"); + RequireContains(error, "expected 0.000010"); } void TestValidationCreatesNoCorelibObjects() { + fake_corelib::Reset(); + auto api = flm::corelib::CorelibApi::ResolveForTest(fake_corelib::Resolver()); + fake_corelib::GetState().call_counts.clear(); + ContractFixture fixture; - fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "hidden_size", "1", "3072"); + + for (const auto name : {"ryzenai_corelib_create_stream", + "ryzenai_corelib_create_device_tensor", + "ryzenai_corelib_create_tensor_window", + "ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized", + "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized", + "ryzenai_corelib_rmsnorm_bf16_weights_create_scale"}) + TEST_REQUIRE(fake_corelib::GetState().call_counts[name] == 0); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + (void)api; } + } // namespace int main() { @@ -328,6 +490,7 @@ int main() { RUN(TestValidatesOptionalShortRopeFactorsAsF32Length48); RUN(TestRejectsNonFiniteOrNonPositiveRopeValues); RUN(TestRejectsConfigDisagreement); + RUN(TestRejectsFiniteWrongRmsValue); RUN(TestDerivesStopSetFromGgufConfigAndTokenizerIds); RUN(TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement); RUN(TestValidationCreatesNoCorelibObjects); From 804b856bb709c28710a64a60ddf6232746835736 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 03:35:24 -0700 Subject: [PATCH 07/37] test: assert exact Phi-4 mismatch diagnostics --- src/test/phi4_corelib_aie4/test_phi4_gguf.cpp | 108 +++++++++++------- 1 file changed, 69 insertions(+), 39 deletions(-) diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp index c45103a7..0edc25a0 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -40,10 +40,13 @@ void RequireMismatch(std::string_view error, std::string_view field, RequireContains(error, "expected " + std::string(expected)); } -void RequireDiagnostic(std::string_view error, std::string_view field) { - RequireContains(error, field); - RequireContains(error, "actual"); - RequireContains(error, "expected"); +std::string ShapeText(const std::vector& shape) { + std::string result = "["; + for (std::size_t index = 0; index < shape.size(); ++index) { + if (index != 0) result += ','; + result += std::to_string(shape[index]); + } + return result + ']'; } struct TensorRole { @@ -139,9 +142,9 @@ void TestRequireQ8AndRequireF32ReportNameActualAndExpected() { gguf_fixture::TempFile file; auto package = Open(SplitFixture(), file, "requires"); auto error = RequireThrows([&] { package->RequireQ8("f32", std::array{48}); }); - RequireContains(error, "f32"); RequireContains(error, "actual F32"); RequireContains(error, "expected Q8_0"); + RequireMismatch(error, "f32", "F32", "Q8_0"); error = RequireThrows([&] { package->RequireF32("f32", std::array{47}); }); - RequireContains(error, "f32"); RequireContains(error, "48"); RequireContains(error, "47"); + RequireMismatch(error, "f32", "[48]", "[47]"); } void TestAttentionQkvReturnsThreeZeroCopyWholeRowViews() { @@ -177,7 +180,9 @@ void TestSplitRejectsNonIntegralQ8RowBoundary() { builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3073}, gguf_fixture::kQ8_0); auto file = builder.Write("bad-row"); auto package = Phi4GgufPackage::Open(file.path); - RequireContains(RequireThrows([&] { package->AttentionQkv(0); }), "row"); + RequireMismatch(RequireThrows([&] { package->AttentionQkv(0); }), + "blk.0.attn_qkv.weight", "3073", + "Q8_0 whole-row width divisible by 32"); } void TestViewsPointIntoTheReadOnlyMapping() { @@ -212,18 +217,31 @@ void TestAcceptsExactPhi3Phi4Contract() { } void TestRejectsWrongArchitectureAndEveryDimension() { - const std::vector> cases = { - {"general.architecture", std::string("llama")}, {"phi3.block_count", std::uint32_t{31}}, - {"phi3.context_length", std::uint32_t{4095}}, {"phi3.embedding_length", std::uint32_t{3071}}, - {"phi3.feed_forward_length", std::uint32_t{8191}}, {"phi3.attention.head_count", std::uint32_t{23}}, - {"phi3.attention.head_count_kv", std::uint32_t{7}}, {"phi3.rope.dimension_count", std::uint32_t{95}}, - {"tokenizer.ggml.tokens", gguf_fixture::ArrayValue{0, 200063, std::vector(200063)}}}; - for (const auto& [field, value] : cases) { - auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("wrong-field"); + struct Case { + std::string field; + gguf_fixture::MetadataValue value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"general.architecture", std::string("llama"), "llama", "phi3"}, + {"phi3.block_count", std::uint32_t{31}, "31", "32"}, + {"phi3.context_length", std::uint32_t{4095}, "4095", "4096"}, + {"phi3.embedding_length", std::uint32_t{3071}, "3071", "3072"}, + {"phi3.feed_forward_length", std::uint32_t{8191}, "8191", "8192"}, + {"phi3.attention.head_count", std::uint32_t{23}, "23", "24"}, + {"phi3.attention.head_count_kv", std::uint32_t{7}, "7", "8"}, + {"phi3.rope.dimension_count", std::uint32_t{95}, "95", "96"}, + {"tokenizer.ggml.tokens", + gguf_fixture::ArrayValue{0, 200063, std::vector(200063)}, + "200063", "200064"}}; + for (const auto& test_case : cases) { + auto file = Builder().SetMetadata(test_case.field, test_case.value) + .AddFullContractTensors().Write("wrong-field"); auto package = Phi4GgufPackage::Open(file.path); const auto error = RequireThrows([&] { package->ValidatePhi4Contract( gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireDiagnostic(error, field); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); } } @@ -262,9 +280,7 @@ void TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole() { const auto shape_error = RequireThrows([&] { shape_package->ValidatePhi4Contract(unused, unused, unused); }); - RequireContains(shape_error, role.name); - RequireContains(shape_error, "actual ["); - RequireContains(shape_error, "expected ["); + RequireMismatch(shape_error, role.name, ShapeText(wrong_shape), ShapeText(role.shape)); auto length_file = Builder().AddFullContractTensors() .TruncateTensorPayload(role.name).Write("wrong-length"); @@ -282,7 +298,7 @@ void TestRejectsMixedQuantizationAndOutputWeightPresence() { auto output_file = Builder().AddFullContractTensors().AddTensor("output.weight", {200064,3072}, gguf_fixture::kQ8_0).Write("output-weight"); auto output = Phi4GgufPackage::Open(output_file.path); const auto error = RequireThrows([&] { output->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, "output.weight"); RequireContains(error, "actual present"); RequireContains(error, "expected absent"); + RequireMismatch(error, "output.weight", "present", "absent (tied token_embd.weight)"); } void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { @@ -309,8 +325,8 @@ void TestValidatesOptionalShortRopeFactorsAsF32Length48() { absent->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); auto wrong_file = Builder().AddFullContractTensors(false).AddTensor("rope_factors_short.weight", {47}, gguf_fixture::kF32).Write("wrong-short-rope"); auto wrong = Phi4GgufPackage::Open(wrong_file.path); - RequireDiagnostic(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), - "rope_factors_short.weight"); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_short.weight", "[47]", "[48]"); } void TestRejectsNonFiniteOrNonPositiveRopeValues() { @@ -318,26 +334,41 @@ void TestRejectsNonFiniteOrNonPositiveRopeValues() { for (const float value : {0.0f, -1.0f, std::numeric_limits::infinity(), std::numeric_limits::quiet_NaN()}) { auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("bad-rope-value"); auto package = Phi4GgufPackage::Open(file.path); - RequireDiagnostic(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), - field); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + field, std::to_string(static_cast(value)), + "finite positive value"); } } } void TestRejectsConfigDisagreement() { ContractFixture fixture; - const std::vector> cases = { - {"model_type", "other"}, {"num_hidden_layers", 31}, {"hidden_size", 3071}, - {"intermediate_size", 8191}, {"num_attention_heads", 23}, {"num_key_value_heads", 7}, - {"head_dim", 127}, {"vocab_size", 200063}, {"rms_norm_eps", 2.0e-5}, - {"original_max_position_embeddings", 4095}, - {"hidden_size", 3072.0}, - {"hidden_size", std::uint64_t{4294970368ULL}}, - {"eos_token_id", 199999.0}}; - for (const auto& [field, value] : cases) { - auto config = gguf_fixture::ValidConfig(); config[field] = value; - const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract(config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireDiagnostic(error, field); + struct Case { + std::string field; + nlohmann::json value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"model_type", "other", "other", "phi3"}, + {"num_hidden_layers", 31, "31", "32"}, + {"hidden_size", 3071, "3071", "3072"}, + {"intermediate_size", 8191, "8191", "8192"}, + {"num_attention_heads", 23, "23", "24"}, + {"num_key_value_heads", 7, "7", "8"}, + {"head_dim", 127, "127", "128"}, + {"vocab_size", 200063, "200063", "200064"}, + {"rms_norm_eps", 2.0e-5, "2e-05", "0.000010"}, + {"original_max_position_embeddings", 4095, "4095", "4096"}, + {"hidden_size", 3072.0, "3072.0", "integer 3072"}, + {"hidden_size", std::uint64_t{4294970368ULL}, "4294970368", "3072"}, + {"eos_token_id", 199999.0, "199999.0", "integer 199999"}}; + for (const auto& test_case : cases) { + auto config = gguf_fixture::ValidConfig(); + config[test_case.field] = test_case.value; + const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); } } @@ -435,9 +466,8 @@ void TestRejectsFiniteWrongRmsValue() { const auto error = RequireThrows([&] { package->ValidatePhi4Contract( gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); - RequireContains(error, "phi3.attention.layer_norm_rms_epsilon"); - RequireContains(error, "actual 0.000020"); - RequireContains(error, "expected 0.000010"); + RequireMismatch(error, "phi3.attention.layer_norm_rms_epsilon", + "0.000020", "0.000010"); } void TestValidationCreatesNoCorelibObjects() { From 0c3c39f71d20f841c41f3a643527f8708fa2e512 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 03:57:25 -0700 Subject: [PATCH 08/37] feat: add corelib-backed Phi-4 AIE4 engine --- src/CMakeLists.txt | 7 +- src/common/corelib/corelib_sources.cmake | 5 +- src/common/corelib/phi4_corelib_aie4.cpp | 168 +++++++++ src/common/corelib/phi4_corelib_host.cpp | 140 ++++++++ .../corelib/phi4_corelib_shape_plan.cpp | 96 +++++ src/include/models/phi4/phi4_corelib_aie4.hpp | 40 +++ src/include/models/phi4/phi4_corelib_host.hpp | 27 ++ .../models/phi4/phi4_corelib_shape_plan.hpp | 35 ++ src/test/phi4_corelib_aie4/CMakeLists.txt | 42 +++ src/test/phi4_corelib_aie4/fake_corelib.cpp | 268 +++++++++++++- src/test/phi4_corelib_aie4/fake_corelib.hpp | 79 +++++ .../phi4_corelib_aie4/test_phi4_engine.cpp | 333 ++++++++++++++++++ src/test/phi4_corelib_aie4/test_phi4_host.cpp | 143 ++++++++ .../test_phi4_shape_plan.cpp | 102 ++++++ 14 files changed, 1475 insertions(+), 10 deletions(-) create mode 100644 src/common/corelib/phi4_corelib_aie4.cpp create mode 100644 src/common/corelib/phi4_corelib_host.cpp create mode 100644 src/common/corelib/phi4_corelib_shape_plan.cpp create mode 100644 src/include/models/phi4/phi4_corelib_aie4.hpp create mode 100644 src/include/models/phi4/phi4_corelib_host.hpp create mode 100644 src/include/models/phi4/phi4_corelib_shape_plan.hpp create mode 100644 src/test/phi4_corelib_aie4/test_phi4_engine.cpp create mode 100644 src/test/phi4_corelib_aie4/test_phi4_host.cpp create mode 100644 src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 12b787d0..8d0d695b 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -56,6 +56,10 @@ if(FLM_ENABLE_CORELIB_AIE4) message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows") endif() find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) + find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" + "C:/dev/boost_1_88_0" REQUIRED) endif() if(FLM_USE_HRX) @@ -283,7 +287,8 @@ if(FLM_ENABLE_CORELIB_AIE4) include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake") add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES}) target_include_directories(flm_corelib_aie4 PUBLIC - "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}" "${FLM_CORELIB_BOOST_INCLUDE_DIR}") target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1) target_link_libraries(flm PRIVATE flm_corelib_aie4) endif() diff --git a/src/common/corelib/corelib_sources.cmake b/src/common/corelib/corelib_sources.cmake index 09fea42a..f9e49a68 100644 --- a/src/common/corelib/corelib_sources.cmake +++ b/src/common/corelib/corelib_sources.cmake @@ -1,4 +1,7 @@ set(FLM_CORELIB_AIE4_SOURCES "${CMAKE_CURRENT_LIST_DIR}/corelib_api.cpp" "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp" - "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_gguf.cpp") + "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_gguf.cpp" + "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_host.cpp" + "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_shape_plan.cpp" + "${CMAKE_CURRENT_LIST_DIR}/phi4_corelib_aie4.cpp") diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp new file mode 100644 index 00000000..f7df494d --- /dev/null +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -0,0 +1,168 @@ +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "corelib/corelib_object.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" +#include "models/phi4/phi4_corelib_host.hpp" +#include "models/phi4/phi4_corelib_shape_plan.hpp" +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +using namespace flm::corelib; +std::string Name(std::size_t i, const char* suffix) { + return "blk." + std::to_string(i) + suffix; +} +} + +struct phi4_corelib_aie4::Impl { + std::shared_ptr package; + std::shared_ptr runtime; + std::shared_ptr api; + Phi4ShapePlan plan; + std::uint32_t max_length; + int position{}; + std::optional saved; + bool poisoned{}; + UniqueStream stream; + UniqueRmsNormWeights first_norm; + std::array q_weights, k_weights, v_weights, o_weights; + std::array mlp_weights; + UniqueMatMulWeights lm_weights; + UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits, cosine, sine; + std::array k_cache, v_cache; + TensorView embedding; + + Impl(LM_Config, std::shared_ptr pkg, + std::shared_ptr rt, std::uint32_t maximum) + : package(std::move(pkg)), runtime(std::move(rt)), + api(runtime ? runtime->api() : nullptr), plan(Phi4ShapePlan::Build(api)), + max_length(maximum) { + if (!package) throw std::invalid_argument("Phi-4 GGUF package is null"); + if (!runtime || !api) throw std::invalid_argument("corelib runtime is null"); + if (!maximum || maximum > kMaxSequenceLength) + throw std::invalid_argument("Phi-4 maximum length must be in 1..4096"); + + // Validate and capture every mapped span before the first device create. + embedding = package->RequireQ8("token_embd.weight", std::array{kVocabularySize,kHiddenSize}); + auto final_norm = package->RequireF32("output_norm.weight", std::array{kHiddenSize}); + std::array an, fn; + std::array qkv, gu; + std::array ow, dw; + for (std::size_t i=0;iRequireF32(Name(i,".attn_norm.weight"),std::array{kHiddenSize}); + fn[i]=package->RequireF32(Name(i,".ffn_norm.weight"),std::array{kHiddenSize}); + qkv[i]=package->AttentionQkv(i); gu[i]=package->GateUp(i); + ow[i]=package->RequireQ8(Name(i,".attn_output.weight"),std::array{kHiddenSize,kHiddenSize}); + dw[i]=package->RequireQ8(Name(i,".ffn_down.weight"),std::array{kHiddenSize,kIntermediateSize}); + } + std::optional factors; + try { factors=package->RequireF32("rope_factors_short.weight",std::array{48}); } + catch (const std::runtime_error&) {} + auto rope=BuildShortRopeTables(package->Metadata(),factors); + auto final_bf=ConvertF32ToBf16(final_norm.values); + std::array,kLayerCount> an_bf,fn_bf; + for(std::size_t i=0;i epsf{kRmsEpsilon}; auto eps=ConvertF32ToBf16(epsf); + + auto lease=runtime->AcquireExecution(); void* raw=nullptr; + api->Check(api->functions().create_stream(&raw),"ryzenai_corelib_create_stream"); stream=UniqueStream(api,raw); + ryzenai_corelib_rmsnorm_bf16_weights_desc rd{kHiddenSize,kRmsEpsilon}; raw=nullptr; + api->Check(api->functions().rmsnorm_weights_create_scale(&rd,an_bf[0].data(),&raw),"ryzenai_corelib_rmsnorm_bf16_weights_create_scale blk.0.attn_norm.weight"); + first_norm=UniqueRmsNormWeights(api,raw); + auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ + ryzenai_corelib_matmul_bf16_weights_desc d{kk,nn,kRequantizedGroupSize,false}; + ryzenai_corelib_matmul_bf16_gguf_components c{tv.bytes.data(),ryzenai_corelib_gguf_quant_type_q8_0}; void* p=nullptr; + api->Check(api->functions().matmul_weights_create_gguf_requantized(&d,&c,0,&p),"ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized "+label); + return UniqueMatMulWeights(api,p); + }; + for(std::size_t i=0;iCheck(api->functions().ssmlp_weights_create_gguf_requantized(&d,&c,0,&raw),"ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized layer "+std::to_string(i)); + mlp_weights[i]=UniqueSsMlpWeights(api,raw); + } + lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); + const auto& e=plan.ForRows(kMaxSequenceLength); + auto rows=std::max({e.query_rows,e.output_rows,e.ssmlp_rows,e.rmsnorm_rows}); + auto tensor=[&](ryzenai_corelib_data_type type,std::initializer_list dims,const char* label){ + std::vector shape(dims);void* p=nullptr; + api->Check(api->functions().create_device_tensor(type,shape.data(),shape.size(),&p),std::string("ryzenai_corelib_create_device_tensor ")+label); + return UniqueTensor(api,p); + }; + hidden=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"hidden"); + residual=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"residual"); + skip=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"skip"); + q=tensor(ryzenai_corelib_data_type_bf16,{e.query_rows,kQueryDimension},"query"); + k=tensor(ryzenai_corelib_data_type_bf16,{e.kv_rows,kKvDimension},"key"); + attention=tensor(ryzenai_corelib_data_type_bf16,{e.flat_mha_rows,kQueryDimension},"attention"); + lm_input=tensor(ryzenai_corelib_data_type_bf16,{1,kHiddenSize},"lm input"); + logits=tensor(ryzenai_corelib_data_type_bf16,{1,kVocabularySize},"logits"); + cosine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"cosine"); + sine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"sine"); + for(std::size_t i=0;iCheck(api->functions().tensor_write(cosine.get(),ryzenai_corelib_data_type_fp32,rope.cosine.data(),rope.cosine.size(),0),"ryzenai_corelib_tensor_write cosine"); + api->Check(api->functions().tensor_write(sine.get(),ryzenai_corelib_data_type_fp32,rope.sine.data(),rope.sine.size(),0),"ryzenai_corelib_tensor_write sine"); + } + + void usable() const {if(poisoned)throw std::runtime_error("Phi-4 corelib engine is poisoned");} + buffer run(std::span ids,bool prefill){ + usable(); if(ids.empty())throw std::invalid_argument("Phi-4 request contains no token IDs"); + if(prefill&&position)throw std::runtime_error("Phi-4 prefill must start at logical position zero"); + if(ids.size()>max_length||position+ids.size()>max_length||position+ids.size()>kMaxSequenceLength)throw std::out_of_range("Phi-4 request exceeds configured context capacity"); + if(!prefill&&position+ids.size()>kMaxDecodeWindow)throw std::out_of_range("Phi-4 decode window stops at position 4095"); + auto decoded=DecodeEmbeddingRowsQ8(embedding,ids);const auto&e=plan.ForRows(ids.size()); + auto rows=std::max({e.query_rows,e.output_rows,e.ssmlp_rows,e.rmsnorm_rows}); + std::vector input(static_cast(rows*kHiddenSize),0);std::copy(decoded.begin(),decoded.end(),input.begin()); + std::vector zeros(static_cast(rows*kHiddenSize),0); + auto lease=runtime->AcquireExecution();bool submitted=false; + try{ + api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); + api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_bf16,zeros.data(),zeros.size(),0),"ryzenai_corelib_tensor_write residual padding"); + api->Check(api->functions().rmsnorm(stream.get(),hidden.get(),ids.size(),first_norm.get(),hidden.get()),"ryzenai_corelib_rmsnorm_bf16 initial");submitted=true; + void* res=residual.get();void* sk=skip.get(); + for(std::size_t i=0;iCheck(api->functions().matmul(stream.get(),hidden.get(),ids.size(),q_weights[i].get(),q.get()),"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),hidden.get(),ids.size(),k_weights[i].get(),k.get()),"ryzenai_corelib_matmul_bf16 key layer "+std::to_string(i)); + std::array shape{8,kMaxSequenceLength-position,128};void* p=nullptr; + api->Check(api->functions().create_tensor_window(v_cache[i].get(),shape.data(),shape.size(),static_cast(position)*128,&p),"ryzenai_corelib_create_tensor_window V");UniqueTensorWindow win(api,p); + api->Check(api->functions().matmul(stream.get(),hidden.get(),ids.size(),v_weights[i].get(),win.get()),"ryzenai_corelib_matmul_bf16 value layer "+std::to_string(i)); + api->Check(api->functions().flat_mha(stream.get(),&plan.attention_desc(),q.get(),k.get(),ids.size(),position,cosine.get(),sine.get(),k_cache[i].get(),v_cache[i].get(),attention.get()),"ryzenai_corelib_flat_mha_bf16 layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),attention.get(),ids.size(),o_weights[i].get(),hidden.get()),"ryzenai_corelib_matmul_bf16 output layer "+std::to_string(i)); + api->Check(api->functions().ssmlp(stream.get(),hidden.get(),res,ids.size(),mlp_weights[i].get(),sk,hidden.get()),"ryzenai_corelib_ssmlp_bf16 layer "+std::to_string(i));std::swap(res,sk); + } + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize hidden"); + std::vector row(kHiddenSize);api->Check(api->functions().tensor_read(hidden.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),(ids.size()-1)*kHiddenSize),"ryzenai_corelib_tensor_read final hidden row"); + api->Check(api->functions().tensor_write(lm_input.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),0),"ryzenai_corelib_tensor_write LM head input"); + api->Check(api->functions().matmul(stream.get(),lm_input.get(),1,lm_weights.get(),logits.get()),"ryzenai_corelib_matmul_bf16 LM head"); + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize logits"); + buffer out(kVocabularySize);api->Check(api->functions().tensor_read(logits.get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),0),"ryzenai_corelib_tensor_read logits");position+=static_cast(ids.size());return out; + }catch(...){if(submitted){(void)api->functions().stream_synchronize(stream.get());poisoned=true;position=0;saved.reset();}throw;} + } + buffer read_cache(bool is_k,int layer,int index){usable();if(layer<0||layer>=kLayerCount||index<0||index>=kMaxSequenceLength)throw std::out_of_range("Phi-4 cache index is out of range");auto lease=runtime->AcquireExecution();api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize cache read");buffer out(kKvHeadCount*kHeadSize);api->Check(api->functions().tensor_read(is_k?k_cache[layer].get():v_cache[layer].get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),static_cast(index)*128),"ryzenai_corelib_tensor_read cache");return out;} +}; + +phi4_corelib_aie4::phi4_corelib_aie4(LM_Config c,std::shared_ptr p,std::shared_ptr r,std::uint32_t m):impl_(std::make_unique(std::move(c),std::move(p),std::move(r),m)){} +phi4_corelib_aie4::~phi4_corelib_aie4()=default; +buffer phi4_corelib_aie4::forward(int id){return impl_->run(std::span(&id,1),false);} +buffer phi4_corelib_aie4::prefill(std::vector&ids,void*){return impl_->run(ids,true);} +void phi4_corelib_aie4::set_context_length(int n){impl_->usable();if(n<0||static_cast(n)>impl_->max_length)throw std::out_of_range("Phi-4 context length is out of range");impl_->position=n;} +void phi4_corelib_aie4::load_weights(Q4NX&){impl_->usable();throw std::runtime_error("Phi-4 AIE4 weights are loaded only from GGUF");} +void phi4_corelib_aie4::update_max_length(std::uint32_t n){impl_->usable();if(!n||n>kMaxSequenceLength||n(impl_->position))throw std::out_of_range("Phi-4 maximum length is invalid");impl_->max_length=n;} +void phi4_corelib_aie4::clear_context(){impl_->usable();impl_->position=0;impl_->saved.reset();} +buffer phi4_corelib_aie4::get_k_cache(int l,int i){return impl_->read_cache(true,l,i);} +buffer phi4_corelib_aie4::get_v_cache(int l,int i){return impl_->read_cache(false,l,i);} +int phi4_corelib_aie4::get_current_context_length(){impl_->usable();return impl_->position;} +int phi4_corelib_aie4::checkpoint(){impl_->usable();impl_->saved=impl_->position;return impl_->position;} +int phi4_corelib_aie4::restore(){impl_->usable();if(!impl_->saved)return -1;return impl_->position=*impl_->saved;} +bool phi4_corelib_aie4::poisoned()const noexcept{return impl_&&impl_->poisoned;} +} // namespace flm::phi4 diff --git a/src/common/corelib/phi4_corelib_host.cpp b/src/common/corelib/phi4_corelib_host.cpp new file mode 100644 index 00000000..792531d0 --- /dev/null +++ b/src/common/corelib/phi4_corelib_host.cpp @@ -0,0 +1,140 @@ +#include "models/phi4/phi4_corelib_host.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" + +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +float HalfToFloat(std::uint16_t half) { + const std::uint32_t sign = static_cast(half & 0x8000) << 16; + const std::uint32_t exponent = (half >> 10) & 0x1f; + std::uint32_t fraction = half & 0x03ff; + std::uint32_t bits; + if (exponent == 0) { + if (fraction == 0) { + bits = sign; + } else { + int shift = 0; + while ((fraction & 0x0400) == 0) { + fraction <<= 1; + ++shift; + } + fraction &= 0x03ff; + bits = sign | (static_cast(127 - 14 - shift) << 23) | + (fraction << 13); + } + } else if (exponent == 0x1f) { + bits = sign | 0x7f800000 | (fraction << 13); + } else { + bits = sign | ((exponent + (127 - 15)) << 23) | (fraction << 13); + } + return std::bit_cast(bits); +} + +} // namespace + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, std::span token_ids) { + if (embedding.ggml_type != 8 || embedding.logical_shape.size() != 2 || + embedding.logical_shape[0] <= 0 || embedding.logical_shape[1] <= 0 || + embedding.logical_shape[1] % 32 != 0) { + throw std::runtime_error("embedding must be a two-dimensional Q8_0 tensor with block-aligned rows"); + } + const auto rows = static_cast(embedding.logical_shape[0]); + const auto width = static_cast(embedding.logical_shape[1]); + const auto blocks_per_row = width / 32; + const auto row_bytes = blocks_per_row * 34; + if (rows > std::numeric_limits::max() / row_bytes || + embedding.bytes.size() != rows * row_bytes) { + throw std::runtime_error("embedding Q8_0 byte length does not match its logical shape"); + } + + std::vector result; + result.reserve(token_ids.size() * width); + for (const int token_id : token_ids) { + if (token_id < 0 || static_cast(token_id) >= rows) { + throw std::out_of_range("embedding token id is outside the vocabulary"); + } + const std::byte* row = embedding.bytes.data() + + static_cast(token_id) * row_bytes; + for (std::size_t block = 0; block < blocks_per_row; ++block) { + const std::byte* encoded = row + block * 34; + std::uint16_t scale_bits; + std::memcpy(&scale_bits, encoded, sizeof(scale_bits)); + const float scale = HalfToFloat(scale_bits); + for (std::size_t element = 0; element < 32; ++element) { + const auto code = static_cast( + std::to_integer(encoded[2 + element])); + result.push_back(scale * static_cast(code)); + } + } + } + return result; +} + +std::vector ConvertF32ToBf16(std::span values) { + std::vector result; + result.reserve(values.size()); + for (const float value : values) { + std::uint32_t bits = std::bit_cast(value); + if ((bits & 0x7fffffffU) > 0x7f800000U) { + bits |= 0x00400000U; + } else { + bits += 0x7fffU + ((bits >> 16) & 1U); + } + result.push_back(static_cast(bits >> 16)); + } + return result; +} + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors) { + if (metadata.context_length != static_cast(kMaxSequenceLength) || + metadata.rope_dimension_count != static_cast(kRopeDimension) || + !std::isfinite(metadata.rope_frequency_base) || metadata.rope_frequency_base <= 0 || + !std::isfinite(metadata.rope_attention_factor)) { + throw std::runtime_error("invalid Phi-4 RoPE metadata"); + } + + std::array factors{}; + factors.fill(1.0); + if (short_factors) { + if (short_factors->logical_shape != std::vector{kRopeDimension / 2} || + short_factors->values.size() != factors.size()) { + throw std::runtime_error("rope_factors_short.weight must have shape [48]"); + } + for (std::size_t i = 0; i < factors.size(); ++i) { + factors[i] = short_factors->values[i]; + if (!std::isfinite(factors[i]) || factors[i] <= 0) + throw std::runtime_error("rope_factors_short.weight must contain finite positive values"); + } + } + + RopeTables tables; + tables.cosine.resize(kMaxSequenceLength * factors.size()); + tables.sine.resize(kMaxSequenceLength * factors.size()); + for (std::size_t i = 0; i < factors.size(); ++i) { + const double inv_freq = 1.0 / + (std::pow(metadata.rope_frequency_base, (2.0 * i) / 96.0) * factors[i]); + for (std::size_t position = 0; position < kMaxSequenceLength; ++position) { + const double angle = static_cast(position) * inv_freq; + const auto index = position * factors.size() + i; + tables.cosine[index] = static_cast( + std::cos(angle) * metadata.rope_attention_factor); + tables.sine[index] = static_cast( + std::sin(angle) * metadata.rope_attention_factor); + } + } + return tables; +} + +} // namespace flm::phi4 diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp new file mode 100644 index 00000000..b2bd6658 --- /dev/null +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -0,0 +1,96 @@ +#include "models/phi4/phi4_corelib_shape_plan.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" + +#include +#include + +namespace flm::phi4 { +namespace { + +std::int64_t MatmulRows(const std::shared_ptr& api, + std::int64_t rows, std::int64_t logical_k, + std::int64_t logical_n, const char* logical_name) { + auto m = rows; + auto k = logical_k; + auto n = logical_n; + const std::string call = std::string("ryzenai_corelib_matmul_bf16_pad_shape ") + + logical_name + " [" + std::to_string(rows) + "," + + std::to_string(logical_k) + "]x[" + std::to_string(logical_k) + "," + + std::to_string(logical_n) + "]"; + api->Check(api->functions().matmul_pad_shape( + &m, &k, &n, kRequantizedGroupSize), call); + if (k != logical_k || n != logical_n) { + throw std::runtime_error(call + ": helper changed padded K/N"); + } + return m; +} + +} // namespace + +Phi4ShapePlan Phi4ShapePlan::Build( + const std::shared_ptr& api) { + if (!api) throw std::invalid_argument("Phi4ShapePlan corelib API is null"); + + Phi4ShapePlan plan; + plan.attention_desc_ = {kQueryHeadCount, kKvHeadCount, kHeadSize, + kMaxSequenceLength, kRopeDimension}; + plan.lm_head_desc_ = {kHiddenSize, kVocabularySize, + kRequantizedGroupSize, false}; + plan.rows_.reserve(kMaxSequenceLength); + + for (std::int64_t rows = 1; rows <= kMaxSequenceLength; ++rows) { + Phi4RowExtents extents{}; + extents.query_rows = MatmulRows(api, rows, kHiddenSize, + kQueryDimension, "query"); + extents.kv_rows = MatmulRows(api, rows, kHiddenSize, + kKvDimension, "key/value"); + extents.output_rows = MatmulRows(api, rows, kHiddenSize, + kHiddenSize, "output"); + + extents.ssmlp_rows = rows; + const std::string ssmlp_call = + "ryzenai_corelib_ssmlp_bf16_pad_rows [" + std::to_string(rows) + + ",3072,8192]"; + api->Check(api->functions().ssmlp_pad_rows( + &extents.ssmlp_rows, kHiddenSize, kIntermediateSize, + kRequantizedGroupSize), ssmlp_call); + + extents.rmsnorm_rows = rows; + const std::string rms_call = + "ryzenai_corelib_rmsnorm_bf16_pad_rows [" + std::to_string(rows) + + ",3072]"; + api->Check(api->functions().rmsnorm_pad_rows( + &extents.rmsnorm_rows, kHiddenSize), rms_call); + + extents.flat_mha_rows = rows; + const std::string mha_call = + "ryzenai_corelib_flat_mha_bf16_pad_rows [" + std::to_string(rows) + + ",24,8,128,4096,96]"; + api->Check(api->functions().flat_mha_pad_rows( + &extents.flat_mha_rows, &plan.attention_desc_), mha_call); + plan.rows_.push_back(extents); + } + + (void)MatmulRows(api, 1, kHiddenSize, kVocabularySize, "lm_head"); + return plan; +} + +const Phi4RowExtents& Phi4ShapePlan::ForRows(std::size_t live_rows) const { + if (live_rows == 0 || live_rows > rows_.size()) { + throw std::out_of_range("Phi-4 live rows must be in 1..4096"); + } + return rows_[live_rows - 1]; +} + +const ryzenai_corelib_flat_mha_bf16_desc& +Phi4ShapePlan::attention_desc() const noexcept { + return attention_desc_; +} + +const ryzenai_corelib_matmul_bf16_weights_desc& +Phi4ShapePlan::lm_head_desc() const noexcept { + return lm_head_desc_; +} + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_aie4.hpp b/src/include/models/phi4/phi4_corelib_aie4.hpp new file mode 100644 index 00000000..678f2e08 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_aie4.hpp @@ -0,0 +1,40 @@ +#pragma once + +#include "causal_lm.hpp" +#include "corelib/corelib_runtime.hpp" +#include "lm_config.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include +#include + +namespace flm::phi4 { + +class phi4_corelib_aie4 final : public causal_lm { +public: + phi4_corelib_aie4( + LM_Config config, + std::shared_ptr package, + std::shared_ptr runtime, + std::uint32_t max_length = 4096); + ~phi4_corelib_aie4() override; + + buffer forward(int id) override; + buffer prefill(std::vector& ids, void* payload = nullptr) override; + void set_context_length(int length) override; + void load_weights(Q4NX&) override; + void update_max_length(std::uint32_t max_length) override; + void clear_context() override; + buffer get_k_cache(int layer, int index) override; + buffer get_v_cache(int layer, int index) override; + int get_current_context_length() override; + int checkpoint() override; + int restore() override; + bool poisoned() const noexcept; + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_host.hpp b/src/include/models/phi4/phi4_corelib_host.hpp new file mode 100644 index 00000000..9695b854 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_host.hpp @@ -0,0 +1,27 @@ +#pragma once + +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct RopeTables { + std::vector cosine; + std::vector sine; +}; + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, + std::span token_ids); + +std::vector ConvertF32ToBf16(std::span values); + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors); + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_shape_plan.hpp b/src/include/models/phi4/phi4_corelib_shape_plan.hpp new file mode 100644 index 00000000..70c83af3 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_shape_plan.hpp @@ -0,0 +1,35 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct Phi4RowExtents { + std::int64_t query_rows; + std::int64_t kv_rows; + std::int64_t output_rows; + std::int64_t ssmlp_rows; + std::int64_t rmsnorm_rows; + std::int64_t flat_mha_rows; +}; + +class Phi4ShapePlan final { +public: + static Phi4ShapePlan Build( + const std::shared_ptr& api); + const Phi4RowExtents& ForRows(std::size_t live_rows) const; + const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; + const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; + +private: + std::vector rows_; + ryzenai_corelib_flat_mha_bf16_desc attention_desc_{}; + ryzenai_corelib_matmul_bf16_weights_desc lm_head_desc_{}; +}; + +} // namespace flm::phi4 diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index 688182d7..44b7f609 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -70,6 +70,45 @@ target_include_directories(test_phi4_gguf PRIVATE "${RYZENAI_CORELIB_INCLUDE_DIR}") target_compile_definitions(test_phi4_gguf PRIVATE RYZENAI_CORELIB_STATIC=1) +add_executable(test_phi4_host + test_phi4_host.cpp + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_host.cpp") +target_include_directories(test_phi4_host PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include") + +add_executable(test_phi4_shape_plan + test_phi4_shape_plan.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_shape_plan.cpp") +target_include_directories(test_phi4_shape_plan PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_shape_plan PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_engine + test_phi4_engine.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/corelib_runtime.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_host.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_shape_plan.cpp" + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_aie4.cpp") +target_include_directories(test_phi4_engine PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_engine PRIVATE + RYZENAI_CORELIB_STATIC=1 USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_engine PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_engine PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_engine PRIVATE xrt_coreutil) + # Compile the actual production frontend translation unit in both feature modes. # Empty declaration-only FFmpeg headers isolate this compile check from an # unrelated optional SDK that is absent on the standalone test host. @@ -122,4 +161,7 @@ include(CTest) add_test(NAME test_corelib_api COMMAND test_corelib_api) add_test(NAME test_real_corelib COMMAND test_real_corelib) add_test(NAME test_phi4_gguf COMMAND test_phi4_gguf) +add_test(NAME test_phi4_host COMMAND test_phi4_host) +add_test(NAME test_phi4_shape_plan COMMAND test_phi4_shape_plan) +add_test(NAME test_phi4_engine COMMAND test_phi4_engine) set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index 0bc2072c..14d40e3c 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -1,5 +1,9 @@ #include "fake_corelib.hpp" +#include +#include +#include +#include #include #include #include @@ -8,6 +12,50 @@ namespace { fake_corelib::State state; thread_local std::string current_detail; +struct FakeObject { + std::string kind; + ryzenai_corelib_data_type data_type{ryzenai_corelib_data_type_bf16}; + std::vector shape; + std::size_t byte_size{}; + std::size_t window_offset{}; +}; + +void* NewObject(std::string kind = "generic") { + ++state.live_objects; + auto* object = new FakeObject; + object->kind = std::move(kind); + return object; +} + +ryzenai_corelib_status Status(std::string_view name) { + const auto configured = state.statuses.find(std::string(name)); + return configured == state.statuses.end() ? state.default_status + : configured->second; +} + +std::size_t Elements(const std::vector& shape) { + std::size_t result = 1; + for (const auto dimension : shape) result *= static_cast(dimension); + return result; +} + +std::size_t TypeBytes(ryzenai_corelib_data_type type) { + return RYZENAI_CORELIB_DATA_TYPE_BITS(type) / 8; +} + +std::uint16_t Bf16(float value) { + std::uint32_t bits = std::bit_cast(value); + bits += 0x7fffU + ((bits >> 16) & 1U); + return static_cast(bits >> 16); +} + +void ObserveCreateConcurrency() { + const int active = ++state.active_weight_creates; + int maximum = state.maximum_active_weight_creates.load(); + while (active > maximum && + !state.maximum_active_weight_creates.compare_exchange_weak(maximum, active)) {} +} + #define FLM_DEFINE_FAKE_TAG(member, symbol) \ struct member##_tag { \ static constexpr std::string_view name = #symbol; \ @@ -25,6 +73,7 @@ template struct TypedFake { static Result Invoke(Args... args) { ++state.call_counts[std::string(Tag::name)]; + state.call_log.emplace_back(Tag::name); auto arguments = std::forward_as_tuple(args...); if constexpr (std::is_same_v) { @@ -45,7 +94,7 @@ struct TypedFake { } else if constexpr (std::is_same_v) { void* object = std::get<0>(arguments); if (object) { - delete static_cast(object); + delete static_cast(object); --state.live_objects; ++state.releases; state.lifetime_events.emplace_back("release"); @@ -55,10 +104,200 @@ struct TypedFake { ++state.cleanup_calls; state.lifetime_events.emplace_back("cleanup"); return; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* out = std::get<0>(arguments); + if (out) *out = status == ryzenai_corelib_status_success ? NewObject("stream") : nullptr; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("tensor")); + object->data_type = type; + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(type); + *out = object; + state.tensor_creates.push_back({type, object->shape, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + void* parent = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + const auto offset = std::get<3>(arguments); + auto* out = std::get<4>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("window")); + if (parent) object->data_type = static_cast(parent)->data_type; + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(object->data_type); + object->window_offset = offset; + *out = object; + state.tensor_windows.push_back({parent, object->shape, offset, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->byte_size; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->data_type; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<1>(arguments); + const void* source = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + bool all_zero = true; + if (source) { + const auto* bytes = static_cast(source); + all_zero = std::all_of(bytes, bytes + count * TypeBytes(type), + [](unsigned char value) { return value == 0; }); + } + state.tensor_writes.push_back({std::get<0>(arguments), type, count, offset, all_zero}); + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<2>(arguments)) { + std::memset(std::get<2>(arguments), 0, + std::get<3>(arguments) * TypeBytes(std::get<1>(arguments))); + } + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + auto* k = std::get<1>(arguments); + auto* n = std::get<2>(arguments); + const auto group = std::get<3>(arguments); + state.matmul_pad_calls.push_back({m ? *m : -1, k ? *k : -1, + n ? *n : -1, group}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success) { + if (m && state.pad_multiple > 0 && *m != 1) + *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + if (k) *k += state.matmul_k_delta; + if (n) *n += state.matmul_n_delta; + } + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + state.rows_pad_calls.push_back({"ssmlp", m ? *m : -1, + std::get<1>(arguments), std::get<2>(arguments), std::get<3>(arguments)}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) + *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + state.rows_pad_calls.push_back({"rmsnorm", m ? *m : -1, + std::get<1>(arguments), 0, 0}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) + *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + auto* desc = std::get<1>(arguments); + state.mha_pad_calls.push_back({m ? *m : -1, desc ? *desc : ryzenai_corelib_flat_mha_bf16_desc{}}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) + *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + if (desc && components) state.weight_creates.push_back({"matmul", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), {components->blocks}}); + if (status == ryzenai_corelib_status_success && out) *out = NewObject("matmul_weights"); + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + if (desc && components) { + fake_corelib::WeightCreateRecord record{"ssmlp", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), + {components->gate_blocks, components->up_blocks, components->down_blocks}}; + if (components->epsilon) record.epsilon = *static_cast(components->epsilon); + if (components->norm0) record.norm0.assign(static_cast(components->norm0), + static_cast(components->norm0) + desc->k); + if (components->norm1) record.norm1.assign(static_cast(components->norm1), + static_cast(components->norm1) + desc->k); + state.weight_creates.push_back(std::move(record)); + } + if (status == ryzenai_corelib_status_success && out) *out = NewObject("ssmlp_weights"); + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* out = std::get<2>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + if (desc) { + fake_corelib::WeightCreateRecord record{"rmsnorm", desc->k, 0, 0, 0, {}}; + record.epsilon = Bf16(desc->epsilon); + if (std::get<1>(arguments)) + record.norm0.assign(static_cast(std::get<1>(arguments)), + static_cast(std::get<1>(arguments)) + desc->k); + state.weight_creates.push_back(std::move(record)); + } + if (status == ryzenai_corelib_status_success && out) *out = NewObject("rmsnorm_weights"); + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + state.work_in_flight = false; + return Status(Tag::name); + } else if constexpr (std::is_same_v || + std::is_same_v || + std::is_same_v || + std::is_same_v) { + const auto status = Status(Tag::name); + if (status != ryzenai_corelib_status_success) return status; + fake_corelib::DispatchRecord record{}; + record.kind = std::is_same_v ? "matmul" : + std::is_same_v ? "ssmlp" : + std::is_same_v ? "rmsnorm" : "mha"; + record.stream = std::get<0>(arguments); + if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); record.rows = std::get<2>(arguments); + record.output = std::get<4>(arguments); + } else if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); record.rows = std::get<3>(arguments); + record.output = std::get<6>(arguments); + } else if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); record.rows = std::get<2>(arguments); + record.output = std::get<4>(arguments); + } else { + record.input = std::get<2>(arguments); record.rows = std::get<4>(arguments); + record.position = std::get<5>(arguments); record.output = std::get<10>(arguments); + } + if (record.output && static_cast(record.output)->kind == "window") + record.window_offset = static_cast(record.output)->window_offset; + state.dispatches.push_back(record); + state.work_in_flight = true; + if (state.fail_after_submit == Tag::name) return ryzenai_corelib_status_failure; + return ryzenai_corelib_status_success; } else if constexpr (std::is_same_v) { - const auto configured = state.statuses.find(std::string(Tag::name)); - return configured == state.statuses.end() ? state.default_status - : configured->second; + return Status(Tag::name); } else { static_assert(kAlwaysFalse, "unhandled fake corelib ABI result"); } @@ -116,6 +355,22 @@ void Reset() { state.cleanup_calls = 0; state.active_leases = 0; state.maximum_active_leases = 0; + state.matmul_pad_calls.clear(); + state.rows_pad_calls.clear(); + state.mha_pad_calls.clear(); + state.pad_multiple = 64; + state.matmul_k_delta = 0; + state.matmul_n_delta = 0; + state.tensor_creates.clear(); + state.tensor_windows.clear(); + state.weight_creates.clear(); + state.dispatches.clear(); + state.tensor_writes.clear(); + state.call_log.clear(); + state.active_weight_creates = 0; + state.maximum_active_weight_creates = 0; + state.work_in_flight = false; + state.fail_after_submit.clear(); } flm::corelib::CorelibApi::Resolver Resolver() { @@ -136,10 +391,7 @@ std::vector CallEveryResolvedFunction( return statuses; } -void* MakeObject() { - ++state.live_objects; - return new int(1); -} +void* MakeObject() { return NewObject(); } void EnterLease() { const int active = ++state.active_leases; diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp index 9d0383c0..7f372605 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.hpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -12,6 +12,69 @@ namespace fake_corelib { +struct MatmulPadCall { + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct RowsPadCall { + std::string helper; + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct MhaPadCall { + std::int64_t m; + ryzenai_corelib_flat_mha_bf16_desc desc; +}; + +struct TensorCreateRecord { + ryzenai_corelib_data_type data_type; + std::vector shape; + void* object; +}; + +struct TensorWindowRecord { + void* parent; + std::vector shape; + std::size_t offset; + void* object; +}; + +struct WeightCreateRecord { + std::string kind; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; + std::uint32_t threads; + std::vector pointers; + std::vector norm0; + std::vector norm1; + std::uint16_t epsilon{}; +}; + +struct DispatchRecord { + std::string kind; + void* stream; + void* input; + void* output; + std::int64_t rows; + std::int64_t position; + std::size_t window_offset; +}; + +struct TensorWriteRecord { + void* tensor; + ryzenai_corelib_data_type source_type; + std::size_t count; + std::size_t offset; + bool all_zero; +}; + struct State { flm::corelib::CorelibVersion version{0, 3, 0}; ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; @@ -30,6 +93,22 @@ struct State { std::atomic cleanup_calls{0}; std::atomic active_leases{0}; std::atomic maximum_active_leases{0}; + std::vector matmul_pad_calls; + std::vector rows_pad_calls; + std::vector mha_pad_calls; + std::int64_t pad_multiple{64}; + std::int64_t matmul_k_delta{0}; + std::int64_t matmul_n_delta{0}; + std::vector tensor_creates; + std::vector tensor_windows; + std::vector weight_creates; + std::vector dispatches; + std::vector tensor_writes; + std::vector call_log; + std::atomic active_weight_creates{0}; + std::atomic maximum_active_weight_creates{0}; + bool work_in_flight{false}; + std::string fail_after_submit; }; State& GetState(); diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp new file mode 100644 index 00000000..70f078da --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -0,0 +1,333 @@ +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "models/phi4/phi4_corelib_host.hpp" +#include "fake_corelib.hpp" +#include "gguf_fixture.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibRuntime; +using flm::phi4::Phi4GgufPackage; +using flm::phi4::phi4_corelib_aie4; + +const std::filesystem::path& FullPackagePath() { + static auto file = gguf_fixture::Builder().AddFullContractTensors(false).Write("engine"); + return file.path; +} + +struct Harness { + std::shared_ptr runtime; + std::shared_ptr package; + std::unique_ptr engine; + + Harness() { + fake_corelib::Reset(); + runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + package = Phi4GgufPackage::Open(FullPackagePath()); + engine = std::make_unique(LM_Config{}, package, runtime); + } + ~Harness() { + engine.reset(); + package.reset(); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } +}; + +void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { + Harness h; + const auto& state = fake_corelib::GetState(); + TEST_REQUIRE(state.call_counts.at("ryzenai_corelib_create_stream") == 1); + TEST_REQUIRE(state.tensor_creates.size() == 74); + TEST_REQUIRE(state.tensor_creates[0].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[3].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[4].shape == std::vector({4096, 1024})); + TEST_REQUIRE(state.tensor_creates[6].shape == std::vector({1, 3072})); + TEST_REQUIRE(state.tensor_creates[7].shape == std::vector({1, 200064})); +} + +void TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight() { + Harness h; + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.size() == 162); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "matmul"; }) == 129); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "ssmlp"; }) == 32); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; }) == 1); +} + +void TestEveryProjectionUsesQ8RequantizedGroup64Threads0() { + Harness h; + for (const auto& record : fake_corelib::GetState().weight_creates) { + if (record.kind == "rmsnorm") continue; + TEST_REQUIRE(record.group_size == 64); + TEST_REQUIRE(record.threads == 0); + } + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_matmul_bf16_weights_create_gguf"] == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_ssmlp_bf16_weights_create_gguf"] == 0); +} + +void TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate() { + Harness h; + TEST_REQUIRE(fake_corelib::GetState().maximum_active_weight_creates == 1); + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.front().kind == "rmsnorm"); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto base = 1 + layer * 5; + TEST_REQUIRE(records[base + 0].kind == "matmul"); + TEST_REQUIRE(records[base + 1].kind == "matmul"); + TEST_REQUIRE(records[base + 2].kind == "matmul"); + TEST_REQUIRE(records[base + 3].kind == "matmul"); + TEST_REQUIRE(records[base + 4].kind == "ssmlp"); + } + TEST_REQUIRE(records.back().kind == "matmul"); +} + +void TestQkvAndGateUpPointersMatchExactMappedSubranges() { + Harness h; + const auto qkv = h.package->AttentionQkv(0); + const auto gate_up = h.package->GateUp(0); + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records[1].pointers[0] == qkv.values[0].bytes.data()); + TEST_REQUIRE(records[2].pointers[0] == qkv.values[1].bytes.data()); + TEST_REQUIRE(records[3].pointers[0] == qkv.values[2].bytes.data()); + TEST_REQUIRE(records[5].pointers[0] == gate_up.values[0].bytes.data()); + TEST_REQUIRE(records[5].pointers[1] == gate_up.values[1].bytes.data()); +} + +void TestNormsAndEpsilonReachCorelibAsBf16() { + Harness h; + const auto expected = flm::phi4::ConvertF32ToBf16(std::array{1.0e-5f})[0]; + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.front().epsilon == expected); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto& record = records[1 + layer * 5 + 4]; + TEST_REQUIRE(record.epsilon == expected); + TEST_REQUIRE(record.norm0.size() == 3072); + TEST_REQUIRE(record.norm1.size() == 3072); + } +} + +void TestEmbeddingMappingOutlivesAllLazyRowReads() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + std::weak_ptr lifetime = package; + auto engine = std::make_unique(LM_Config{}, package, runtime); + package.reset(); + TEST_REQUIRE(!lifetime.expired()); + (void)engine->forward(0); + engine.reset(); + TEST_REQUIRE(lifetime.expired()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); +} + +void TestNoDeviceObjectExistsWhenPackageValidationFails() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto file = gguf_fixture::Builder().AddExactFixtureTensors().Write("invalid-engine"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { + phi4_corelib_aie4 engine(LM_Config{}, package, runtime); + }), "blk.1"); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_create_stream"] == 0); + package.reset(); runtime.reset(); CorelibRuntime::ShutdownProcess(); +} + +void TestPrefillDecodesEmbeddingRowsAndAdvancesPosition() { + Harness h; + std::vector ids{2, 1, 2}; + const auto logits = h.engine->prefill(ids); + TEST_REQUIRE(logits.size() == 200064); + TEST_REQUIRE(h.engine->get_current_context_length() == 3); + TEST_REQUIRE(fake_corelib::GetState().tensor_writes[2].source_type == ryzenai_corelib_data_type_fp32); +} + +void TestDecodeUsesOneRowAndAdvancesPosition() { + Harness h; + (void)h.engine->forward(4); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().dispatches.front().rows == 1); +} + +void TestVProjectionWritesWindowAtPositionTimes128() { + Harness h; + h.engine->set_context_length(7); + (void)h.engine->forward(1); + const auto& windows = fake_corelib::GetState().tensor_windows; + TEST_REQUIRE(windows.size() == 32); + TEST_REQUIRE(windows.front().shape == std::vector({8, 4089, 128})); + TEST_REQUIRE(windows.front().offset == 7 * 128); + TEST_REQUIRE(fake_corelib::GetState().dispatches[3].window_offset == 7 * 128); +} + +void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { + Harness h; + (void)h.engine->forward(1); + const auto& calls = fake_corelib::GetState().dispatches; + TEST_REQUIRE(calls.size() == 194); + const void* stream = calls.front().stream; + TEST_REQUIRE(calls.front().kind == "rmsnorm"); + for (std::size_t layer = 0; layer < 32; ++layer) { + const std::size_t base = 1 + layer * 6; + TEST_REQUIRE(calls[base + 0].kind == "matmul"); + TEST_REQUIRE(calls[base + 1].kind == "matmul"); + TEST_REQUIRE(calls[base + 2].kind == "matmul"); + TEST_REQUIRE(calls[base + 3].kind == "mha"); + TEST_REQUIRE(calls[base + 4].kind == "matmul"); + TEST_REQUIRE(calls[base + 5].kind == "ssmlp"); + } + TEST_REQUIRE(std::all_of(calls.begin(), calls.end(), [&](const auto& c) { return c.stream == stream; })); +} + +void TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket() { + Harness h; + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes[0].count == 64 * 3072); + TEST_REQUIRE(writes[1].count == 64 * 3072); + TEST_REQUIRE(writes[1].all_zero); +} + +void TestForwardSynchronizesBeforeHostReadAndLmHeadRead() { + Harness h; + fake_corelib::GetState().call_log.clear(); + (void)h.engine->forward(1); + const auto& log = fake_corelib::GetState().call_log; + const auto first_sync = std::find(log.begin(), log.end(), "ryzenai_corelib_stream_synchronize"); + const auto first_read = std::find(log.begin(), log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(first_sync < first_read); + const auto lm_submit = std::find(first_read, log.end(), "ryzenai_corelib_matmul_bf16"); + const auto second_sync = std::find(lm_submit, log.end(), "ryzenai_corelib_stream_synchronize"); + const auto logits_read = std::find(second_sync, log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(lm_submit < second_sync && second_sync < logits_read); +} + +void TestKVCachesRemainFixedAt8By4096By128() { + Harness h; + const auto& creates = fake_corelib::GetState().tensor_creates; + const auto count = std::count_if(creates.begin(), creates.end(), [](const auto& record) { + return record.shape == std::vector({8, 4096, 128}); + }); + TEST_REQUIRE(count == 64); +} + +void TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken() { + Harness h; + std::vector ids(4096, 0); + (void)h.engine->prefill(ids); + TEST_REQUIRE(h.engine->get_current_context_length() == 4096); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "capacity"); +} + +void TestTotalDecodeWindowStopsAt4095() { + Harness h; + h.engine->set_context_length(4094); + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->get_current_context_length() == 4095); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "4095"); +} + +void TestClearContextResetsLogicalPositionWithoutRecreatingWeights() { + Harness h; + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + h.engine->clear_context(); + TEST_REQUIRE(h.engine->get_current_context_length() == 0); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestCheckpointRestoreChangesOnlyLogicalPosition() { + Harness h; + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->checkpoint() == 1); + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + TEST_REQUIRE(h.engine->restore() == 1); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestPreSubmitFailureIsRecoverable() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_rmsnorm_bf16"] = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "rmsnorm"); + TEST_REQUIRE(!h.engine->poisoned()); + fake_corelib::GetState().statuses.erase("ryzenai_corelib_rmsnorm_bf16"); + (void)h.engine->forward(0); +} + +void TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState() { + Harness h; + h.engine->set_context_length(3); + h.engine->checkpoint(); + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "matmul"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestSynchronizeFailurePoisonsAndClearsState() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_stream_synchronize"] = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "synchronize"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestPoisonedInstanceRejectsEveryLaterEntryPoint() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + RequireContains(RequireThrows([&] { h.engine->clear_context(); }), "poisoned"); + RequireContains(RequireThrows([&] { (void)h.engine->get_current_context_length(); }), "poisoned"); + std::vector ids{0}; + RequireContains(RequireThrows([&] { (void)h.engine->prefill(ids); }), "poisoned"); +} + +void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_ssmlp_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); + RUN_TEST(TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight); + RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64Threads0); + RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); + RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); + RUN_TEST(TestNormsAndEpsilonReachCorelibAsBf16); + RUN_TEST(TestEmbeddingMappingOutlivesAllLazyRowReads); + RUN_TEST(TestNoDeviceObjectExistsWhenPackageValidationFails); + RUN_TEST(TestPrefillDecodesEmbeddingRowsAndAdvancesPosition); + RUN_TEST(TestDecodeUsesOneRowAndAdvancesPosition); + RUN_TEST(TestVProjectionWritesWindowAtPositionTimes128); + RUN_TEST(TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream); + RUN_TEST(TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket); + RUN_TEST(TestForwardSynchronizesBeforeHostReadAndLmHeadRead); + RUN_TEST(TestKVCachesRemainFixedAt8By4096By128); + RUN_TEST(TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken); + RUN_TEST(TestTotalDecodeWindowStopsAt4095); + RUN_TEST(TestClearContextResetsLogicalPositionWithoutRecreatingWeights); + RUN_TEST(TestCheckpointRestoreChangesOnlyLogicalPosition); + RUN_TEST(TestPreSubmitFailureIsRecoverable); + RUN_TEST(TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState); + RUN_TEST(TestSynchronizeFailurePoisonsAndClearsState); + RUN_TEST(TestPoisonedInstanceRejectsEveryLaterEntryPoint); + RUN_TEST(TestCancellationBoundaryLeavesNoOutstandingFakeWork); +#undef RUN_TEST +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_host.cpp b/src/test/phi4_corelib_aie4/test_phi4_host.cpp new file mode 100644 index 00000000..376b7712 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_host.cpp @@ -0,0 +1,143 @@ +#include "models/phi4/phi4_corelib_host.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NOMINMAX +#include + +namespace { +using namespace flm::phi4; + +void PutHalf(std::vector& bytes, std::size_t offset, std::uint16_t bits) { + bytes[offset] = static_cast(bits & 0xff); + bytes[offset + 1] = static_cast(bits >> 8); +} + +TensorView ThreeRows() { + static std::vector bytes(3 * 34, std::byte{0x7f}); + std::fill(bytes.begin(), bytes.end(), std::byte{0x7f}); + for (std::size_t row = 0; row < 3; ++row) { + const std::size_t base = row * 34; + PutHalf(bytes, base, row == 0 ? 0x3800 : row == 1 ? 0x3c00 : 0x4000); + for (std::size_t column = 0; column < 32; ++column) { + const auto value = static_cast(row == 1 ? -static_cast(column) : + static_cast(row + column)); + bytes[base + 2 + column] = static_cast(value); + } + } + return {"token_embd.weight", bytes, {3, 32}, 8}; +} + +GgufPhi4Metadata Metadata(double attention = 1.0) { + return {"phi3", 32, 3072, 8192, 24, 8, 4096, 96, 10000.0, + attention, 4096, 200064, false}; +} + +void TestLazyEmbeddingDecodesOnlyRequestedRows() { + constexpr std::size_t width = 65536; + constexpr std::size_t row_bytes = width / 32 * 34; // 17 Windows pages. + auto* mapping = static_cast(VirtualAlloc( + nullptr, 3 * row_bytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); + TEST_REQUIRE(mapping != nullptr); + for (std::size_t block = 0; block < width / 32; ++block) { + const std::uint16_t scale = 0x3c00; + std::memcpy(mapping + row_bytes + block * 34, &scale, sizeof(scale)); + std::fill_n(mapping + row_bytes + block * 34 + 2, 32, std::byte{0xff}); + } + DWORD old_protection{}; + TEST_REQUIRE(VirtualProtect(mapping, row_bytes, PAGE_NOACCESS, &old_protection)); + TEST_REQUIRE(VirtualProtect(mapping + 2 * row_bytes, row_bytes, + PAGE_NOACCESS, &old_protection)); + const TensorView embedding{"token_embd.weight", {mapping, 3 * row_bytes}, + {3, static_cast(width)}, 8}; + const std::array ids{1}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == width); + TEST_REQUIRE(decoded.front() == -1.0f && decoded.back() == -1.0f); + VirtualFree(mapping, 0, MEM_RELEASE); +} + +void TestLazyEmbeddingPreservesRequestOrderAndDuplicates() { + const auto embedding = ThreeRows(); + const std::array ids{2, 0, 2}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == 96); + TEST_REQUIRE(decoded[0] == 4.0f); + TEST_REQUIRE(decoded[32] == 0.0f); + TEST_REQUIRE(decoded[64] == 4.0f); + TEST_REQUIRE(decoded[95] == 66.0f); +} + +void TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds() { + const auto embedding = ThreeRows(); + std::array negative{-1}; + std::array too_large{3}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, negative); }), + "token id"); + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, too_large); }), + "token id"); + auto malformed = embedding; + malformed.bytes = malformed.bytes.first(malformed.bytes.size() - 1); + std::array valid{0}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(malformed, valid); }), + "Q8_0"); +} + +void TestF32ToBf16UsesRoundToNearestEven() { + const std::array values{ + std::bit_cast(std::uint32_t{0x3f808000}), + std::bit_cast(std::uint32_t{0x3f818000}), + -2.5f, + std::numeric_limits::infinity()}; + const auto result = ConvertF32ToBf16(values); + TEST_REQUIRE(result == std::vector({0x3f80, 0x3f82, 0xc020, 0x7f80})); +} + +void TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + constexpr std::size_t i = 47; + constexpr std::size_t p = 4095; + const double inv = 1.0 / std::pow(10000.0, (2.0 * i) / 96.0); + const float expected = static_cast(std::cos(p * inv)); + TEST_REQUIRE(tables.cosine[p * 48 + i] == expected); +} + +void TestRopeTablesApplyShortFactorsAndAttentionFactor() { + std::array factors{}; + factors.fill(2.0f); + FloatTensorView factor_view{"rope_factors_short.weight", factors, {48}}; + const auto tables = BuildShortRopeTables(Metadata(1.5), factor_view); + const double inv = 1.0 / (std::pow(10000.0, 2.0 / 96.0) * 2.0); + TEST_REQUIRE(tables.cosine[48 + 1] == static_cast(std::cos(inv) * 1.5)); + TEST_REQUIRE(tables.sine[48 + 1] == static_cast(std::sin(inv) * 1.5)); +} + +void TestRopeTablesHaveShape4096By48() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + TEST_REQUIRE(tables.cosine.size() == 4096 * 48); + TEST_REQUIRE(tables.sine.size() == 4096 * 48); + TEST_REQUIRE(tables.cosine[0] == 1.0f); + TEST_REQUIRE(tables.sine[0] == 0.0f); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestLazyEmbeddingDecodesOnlyRequestedRows); + RUN_TEST(TestLazyEmbeddingPreservesRequestOrderAndDuplicates); + RUN_TEST(TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds); + RUN_TEST(TestF32ToBf16UsesRoundToNearestEven); + RUN_TEST(TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs); + RUN_TEST(TestRopeTablesApplyShortFactorsAndAttentionFactor); + RUN_TEST(TestRopeTablesHaveShape4096By48); +#undef RUN_TEST +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp new file mode 100644 index 00000000..96a6aa92 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -0,0 +1,102 @@ +#include "models/phi4/phi4_corelib_shape_plan.hpp" +#include "fake_corelib.hpp" +#include "test_support.hpp" + +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::phi4::Phi4ShapePlan; + +std::shared_ptr Api() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +void TestShapePlanQueriesRows1Through4096AtGroup64() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& state = fake_corelib::GetState(); + TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * 4096 + 1); + TEST_REQUIRE(state.rows_pad_calls.size() == 2 * 4096); + TEST_REQUIRE(state.mha_pad_calls.size() == 4096); + for (std::size_t row = 1; row <= 4096; ++row) { + TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].m == static_cast(row)); + TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].group_size == 64); + TEST_REQUIRE(state.rows_pad_calls[(row - 1) * 2].group_size == 64); + TEST_REQUIRE(state.mha_pad_calls[row - 1].m == static_cast(row)); + } + TEST_REQUIRE(plan.ForRows(65).query_rows == 128); +} + +void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& state = fake_corelib::GetState(); + const auto& q = state.matmul_pad_calls[0]; + const auto& kv = state.matmul_pad_calls[1]; + const auto& output = state.matmul_pad_calls[2]; + TEST_REQUIRE(q.k == 3072 && q.n == 3072); + TEST_REQUIRE(kv.k == 3072 && kv.n == 1024); + TEST_REQUIRE(output.k == 3072 && output.n == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); + TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); + TEST_REQUIRE(state.rows_pad_calls[1].helper == "rmsnorm"); + TEST_REQUIRE(state.rows_pad_calls[1].k == 3072); + const auto& lm = state.matmul_pad_calls.back(); + TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); + TEST_REQUIRE(plan.lm_head_desc().k == 3072); + TEST_REQUIRE(plan.lm_head_desc().n == 200064); +} + +void TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& desc = plan.attention_desc(); + TEST_REQUIRE(desc.num_heads == 24); + TEST_REQUIRE(desc.kv_num_heads == 8); + TEST_REQUIRE(desc.head_size == 128); + TEST_REQUIRE(desc.max_seq == 4096); + TEST_REQUIRE(desc.rope_dim == 96); + TEST_REQUIRE(fake_corelib::GetState().mha_pad_calls.front().desc.rope_dim == 96); +} + +void TestShapePlanRejectsPaddedKOrNChanges() { + fake_corelib::Reset(); + fake_corelib::GetState().matmul_k_delta = 1; + RequireContains(RequireThrows([&] { Phi4ShapePlan::Build(Api()); }), "padded K/N"); + fake_corelib::Reset(); + fake_corelib::GetState().matmul_n_delta = 1; + RequireContains(RequireThrows([&] { Phi4ShapePlan::Build(Api()); }), "padded K/N"); +} + +void TestShapePlanRejectsRowsOutsideCachedRange() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + RequireContains(RequireThrows([&] { plan.ForRows(0); }), "1..4096"); + RequireContains(RequireThrows([&] { plan.ForRows(4097); }), "1..4096"); +} + +void TestShapePlanFailureNamesHelperAndLogicalShape() { + fake_corelib::Reset(); + auto api = Api(); + fake_corelib::GetState().statuses["ryzenai_corelib_ssmlp_bf16_pad_rows"] = + ryzenai_corelib_status_unsupported; + const auto error = RequireThrows([&] { Phi4ShapePlan::Build(api); }); + RequireContains(error, "ryzenai_corelib_ssmlp_bf16_pad_rows"); + RequireContains(error, "[1,3072,8192]"); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestShapePlanQueriesRows1Through4096AtGroup64); + RUN_TEST(TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions); + RUN_TEST(TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96); + RUN_TEST(TestShapePlanRejectsPaddedKOrNChanges); + RUN_TEST(TestShapePlanRejectsRowsOutsideCachedRange); + RUN_TEST(TestShapePlanFailureNamesHelperAndLogicalShape); +#undef RUN_TEST +} From ee665518d3065048028c1f5bac9db95c748c8eaa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 04:09:49 -0700 Subject: [PATCH 09/37] fix: harden Phi-4 corelib engine state --- src/common/corelib/phi4_corelib_aie4.cpp | 44 +++++-- .../corelib/phi4_corelib_shape_plan.cpp | 17 +++ .../models/phi4/phi4_corelib_shape_plan.hpp | 2 + src/test/phi4_corelib_aie4/fake_corelib.cpp | 100 +++++++++++++-- src/test/phi4_corelib_aie4/fake_corelib.hpp | 2 + .../phi4_corelib_aie4/test_phi4_engine.cpp | 115 +++++++++++++++++- 6 files changed, 254 insertions(+), 26 deletions(-) diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp index f7df494d..b13a2bdd 100644 --- a/src/common/corelib/phi4_corelib_aie4.cpp +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -92,8 +92,12 @@ struct phi4_corelib_aie4::Impl { mlp_weights[i]=UniqueSsMlpWeights(api,raw); } lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); - const auto& e=plan.ForRows(kMaxSequenceLength); - auto rows=std::max({e.query_rows,e.output_rows,e.ssmlp_rows,e.rmsnorm_rows}); + const auto& e=plan.maximum_extents(); + const auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, + e.ssmlp_rows,e.rmsnorm_rows}); + const auto query_rows=std::max(e.query_rows,e.flat_mha_rows); + const auto key_rows=std::max(e.kv_rows,e.flat_mha_rows); + const auto attention_rows=std::max(e.flat_mha_rows,e.output_rows); auto tensor=[&](ryzenai_corelib_data_type type,std::initializer_list dims,const char* label){ std::vector shape(dims);void* p=nullptr; api->Check(api->functions().create_device_tensor(type,shape.data(),shape.size(),&p),std::string("ryzenai_corelib_create_device_tensor ")+label); @@ -102,9 +106,9 @@ struct phi4_corelib_aie4::Impl { hidden=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"hidden"); residual=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"residual"); skip=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"skip"); - q=tensor(ryzenai_corelib_data_type_bf16,{e.query_rows,kQueryDimension},"query"); - k=tensor(ryzenai_corelib_data_type_bf16,{e.kv_rows,kKvDimension},"key"); - attention=tensor(ryzenai_corelib_data_type_bf16,{e.flat_mha_rows,kQueryDimension},"attention"); + q=tensor(ryzenai_corelib_data_type_bf16,{query_rows,kQueryDimension},"query"); + k=tensor(ryzenai_corelib_data_type_bf16,{key_rows,kKvDimension},"key"); + attention=tensor(ryzenai_corelib_data_type_bf16,{attention_rows,kQueryDimension},"attention"); lm_input=tensor(ryzenai_corelib_data_type_bf16,{1,kHiddenSize},"lm input"); logits=tensor(ryzenai_corelib_data_type_bf16,{1,kVocabularySize},"logits"); cosine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"cosine"); @@ -121,14 +125,19 @@ struct phi4_corelib_aie4::Impl { if(ids.size()>max_length||position+ids.size()>max_length||position+ids.size()>kMaxSequenceLength)throw std::out_of_range("Phi-4 request exceeds configured context capacity"); if(!prefill&&position+ids.size()>kMaxDecodeWindow)throw std::out_of_range("Phi-4 decode window stops at position 4095"); auto decoded=DecodeEmbeddingRowsQ8(embedding,ids);const auto&e=plan.ForRows(ids.size()); - auto rows=std::max({e.query_rows,e.output_rows,e.ssmlp_rows,e.rmsnorm_rows}); + auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, + e.ssmlp_rows,e.rmsnorm_rows}); std::vector input(static_cast(rows*kHiddenSize),0);std::copy(decoded.begin(),decoded.end(),input.begin()); std::vector zeros(static_cast(rows*kHiddenSize),0); auto lease=runtime->AcquireExecution();bool submitted=false; try{ api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_bf16,zeros.data(),zeros.size(),0),"ryzenai_corelib_tensor_write residual padding"); - api->Check(api->functions().rmsnorm(stream.get(),hidden.get(),ids.size(),first_norm.get(),hidden.get()),"ryzenai_corelib_rmsnorm_bf16 initial");submitted=true; + const auto rms_status=api->functions().rmsnorm( + stream.get(),hidden.get(),ids.size(),first_norm.get(),hidden.get()); + submitted=rms_status==ryzenai_corelib_status_success || + rms_status==ryzenai_corelib_status_failure; + api->Check(rms_status,"ryzenai_corelib_rmsnorm_bf16 initial"); void* res=residual.get();void* sk=skip.get(); for(std::size_t i=0;iCheck(api->functions().matmul(stream.get(),hidden.get(),ids.size(),q_weights[i].get(),q.get()),"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); @@ -148,7 +157,26 @@ struct phi4_corelib_aie4::Impl { buffer out(kVocabularySize);api->Check(api->functions().tensor_read(logits.get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),0),"ryzenai_corelib_tensor_read logits");position+=static_cast(ids.size());return out; }catch(...){if(submitted){(void)api->functions().stream_synchronize(stream.get());poisoned=true;position=0;saved.reset();}throw;} } - buffer read_cache(bool is_k,int layer,int index){usable();if(layer<0||layer>=kLayerCount||index<0||index>=kMaxSequenceLength)throw std::out_of_range("Phi-4 cache index is out of range");auto lease=runtime->AcquireExecution();api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize cache read");buffer out(kKvHeadCount*kHeadSize);api->Check(api->functions().tensor_read(is_k?k_cache[layer].get():v_cache[layer].get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),static_cast(index)*128),"ryzenai_corelib_tensor_read cache");return out;} + buffer read_cache(bool is_k,int layer,int index){ + usable(); + if(layer<0||layer>=kLayerCount||index<0||index>=kMaxSequenceLength) + throw std::out_of_range("Phi-4 cache index is out of range"); + auto lease=runtime->AcquireExecution(); + api->Check(api->functions().stream_synchronize(stream.get()), + "ryzenai_corelib_stream_synchronize cache read"); + buffer out(kKvHeadCount*kHeadSize); + void* cache=is_k?k_cache[layer].get():v_cache[layer].get(); + for(std::size_t head=0;head(index))*kHeadSize; + api->Check(api->functions().tensor_read( + cache,ryzenai_corelib_data_type_bf16, + out.data()+head*kHeadSize,kHeadSize,offset), + "ryzenai_corelib_tensor_read cache head "+ + std::to_string(head)); + } + return out; + } }; phi4_corelib_aie4::phi4_corelib_aie4(LM_Config c,std::shared_ptr p,std::shared_ptr r,std::uint32_t m):impl_(std::make_unique(std::move(c),std::move(p),std::move(r),m)){} diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp index b2bd6658..951411b4 100644 --- a/src/common/corelib/phi4_corelib_shape_plan.cpp +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -2,6 +2,7 @@ #include "models/phi4/phi4_corelib_constants.hpp" +#include #include #include @@ -69,6 +70,18 @@ Phi4ShapePlan Phi4ShapePlan::Build( ",24,8,128,4096,96]"; api->Check(api->functions().flat_mha_pad_rows( &extents.flat_mha_rows, &plan.attention_desc_), mha_call); + plan.maximum_extents_.query_rows = std::max( + plan.maximum_extents_.query_rows, extents.query_rows); + plan.maximum_extents_.kv_rows = std::max( + plan.maximum_extents_.kv_rows, extents.kv_rows); + plan.maximum_extents_.output_rows = std::max( + plan.maximum_extents_.output_rows, extents.output_rows); + plan.maximum_extents_.ssmlp_rows = std::max( + plan.maximum_extents_.ssmlp_rows, extents.ssmlp_rows); + plan.maximum_extents_.rmsnorm_rows = std::max( + plan.maximum_extents_.rmsnorm_rows, extents.rmsnorm_rows); + plan.maximum_extents_.flat_mha_rows = std::max( + plan.maximum_extents_.flat_mha_rows, extents.flat_mha_rows); plan.rows_.push_back(extents); } @@ -83,6 +96,10 @@ const Phi4RowExtents& Phi4ShapePlan::ForRows(std::size_t live_rows) const { return rows_[live_rows - 1]; } +const Phi4RowExtents& Phi4ShapePlan::maximum_extents() const noexcept { + return maximum_extents_; +} + const ryzenai_corelib_flat_mha_bf16_desc& Phi4ShapePlan::attention_desc() const noexcept { return attention_desc_; diff --git a/src/include/models/phi4/phi4_corelib_shape_plan.hpp b/src/include/models/phi4/phi4_corelib_shape_plan.hpp index 70c83af3..f27af767 100644 --- a/src/include/models/phi4/phi4_corelib_shape_plan.hpp +++ b/src/include/models/phi4/phi4_corelib_shape_plan.hpp @@ -23,11 +23,13 @@ class Phi4ShapePlan final { static Phi4ShapePlan Build( const std::shared_ptr& api); const Phi4RowExtents& ForRows(std::size_t live_rows) const; + const Phi4RowExtents& maximum_extents() const noexcept; const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; private: std::vector rows_; + Phi4RowExtents maximum_extents_{}; ryzenai_corelib_flat_mha_bf16_desc attention_desc_{}; ryzenai_corelib_matmul_bf16_weights_desc lm_head_desc_{}; }; diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index 14d40e3c..38490ed5 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -12,12 +12,18 @@ namespace { fake_corelib::State state; thread_local std::string current_detail; +struct FakeStorage { + std::size_t byte_size{}; + std::unique_ptr> bytes; +}; + struct FakeObject { std::string kind; ryzenai_corelib_data_type data_type{ryzenai_corelib_data_type_bf16}; std::vector shape; std::size_t byte_size{}; std::size_t window_offset{}; + std::shared_ptr storage; }; void* NewObject(std::string kind = "generic") { @@ -43,12 +49,32 @@ std::size_t TypeBytes(ryzenai_corelib_data_type type) { return RYZENAI_CORELIB_DATA_TYPE_BITS(type) / 8; } +std::int64_t PaddedRows(std::string_view helper, std::int64_t rows) { + const auto helpers = state.pad_row_overrides.find(std::string(helper)); + if (helpers != state.pad_row_overrides.end()) { + const auto found = helpers->second.find(rows); + if (found != helpers->second.end()) return found->second; + } + if (rows == 1 || state.pad_multiple <= 0) return rows; + return (rows + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; +} + std::uint16_t Bf16(float value) { std::uint32_t bits = std::bit_cast(value); bits += 0x7fffU + ((bits >> 16) & 1U); return static_cast(bits >> 16); } +float FloatFromBf16(std::uint16_t value) { + return std::bit_cast(static_cast(value) << 16); +} + +void EnsureStorage(FakeObject& object) { + if (!object.storage->bytes) + object.storage->bytes = std::make_unique>( + object.storage->byte_size, std::byte{0}); +} + void ObserveCreateConcurrency() { const int active = ++state.active_weight_creates; int maximum = state.maximum_active_weight_creates.load(); @@ -121,6 +147,8 @@ struct TypedFake { object->data_type = type; object->shape.assign(shape, shape + shape_len); object->byte_size = Elements(object->shape) * TypeBytes(type); + object->storage = std::make_shared(); + object->storage->byte_size = object->byte_size; *out = object; state.tensor_creates.push_back({type, object->shape, object}); } @@ -135,10 +163,14 @@ struct TypedFake { if (out) *out = nullptr; if (status == ryzenai_corelib_status_success && out && shape) { auto* object = static_cast(NewObject("window")); - if (parent) object->data_type = static_cast(parent)->data_type; + if (parent) { + const auto* parent_object = static_cast(parent); + object->data_type = parent_object->data_type; + object->storage = parent_object->storage; + object->window_offset = parent_object->window_offset + offset; + } object->shape.assign(shape, shape + shape_len); object->byte_size = Elements(object->shape) * TypeBytes(object->data_type); - object->window_offset = offset; *out = object; state.tensor_windows.push_back({parent, object->shape, offset, object}); } @@ -166,12 +198,54 @@ struct TypedFake { [](unsigned char value) { return value == 0; }); } state.tensor_writes.push_back({std::get<0>(arguments), type, count, offset, all_zero}); + auto* object = static_cast(std::get<0>(arguments)); + if (status == ryzenai_corelib_status_success && object && source) { + const auto target_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + if (!all_zero || object->storage->bytes) EnsureStorage(*object); + if (object->storage->bytes) { + auto* target = object->storage->bytes->data() + target_offset; + if (object->data_type == type) { + std::memcpy(target, source, count * TypeBytes(type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + type == ryzenai_corelib_data_type_fp32) { + const auto* values = static_cast(source); + for (std::size_t i = 0; i < count; ++i) { + const auto converted = Bf16(values[i]); + std::memcpy(target + i * sizeof(converted), &converted, + sizeof(converted)); + } + } + } + } return status; } else if constexpr (std::is_same_v) { const auto status = Status(Tag::name); - if (status == ryzenai_corelib_status_success && std::get<2>(arguments)) { - std::memset(std::get<2>(arguments), 0, - std::get<3>(arguments) * TypeBytes(std::get<1>(arguments))); + auto* object = static_cast(std::get<0>(arguments)); + const auto destination_type = std::get<1>(arguments); + void* destination = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + if (status == ryzenai_corelib_status_success && destination) { + std::memset(destination, 0, count * TypeBytes(destination_type)); + if (object && object->storage && object->storage->bytes) { + const auto source_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + const auto* source = object->storage->bytes->data() + source_offset; + if (object->data_type == destination_type) { + std::memcpy(destination, source, + count * TypeBytes(destination_type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + destination_type == ryzenai_corelib_data_type_fp32) { + auto* values = static_cast(destination); + for (std::size_t i = 0; i < count; ++i) { + std::uint16_t encoded; + std::memcpy(&encoded, source + i * sizeof(encoded), + sizeof(encoded)); + values[i] = FloatFromBf16(encoded); + } + } + } } return status; } else if constexpr (std::is_same_v) { @@ -183,8 +257,7 @@ struct TypedFake { n ? *n : -1, group}); const auto status = Status(Tag::name); if (status == ryzenai_corelib_status_success) { - if (m && state.pad_multiple > 0 && *m != 1) - *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + if (m) *m = PaddedRows(n && *n == 1024 ? "matmul-1024" : "matmul-3072", *m); if (k) *k += state.matmul_k_delta; if (n) *n += state.matmul_n_delta; } @@ -194,24 +267,24 @@ struct TypedFake { state.rows_pad_calls.push_back({"ssmlp", m ? *m : -1, std::get<1>(arguments), std::get<2>(arguments), std::get<3>(arguments)}); const auto status = Status(Tag::name); - if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) - *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("ssmlp", *m); return status; } else if constexpr (std::is_same_v) { auto* m = std::get<0>(arguments); state.rows_pad_calls.push_back({"rmsnorm", m ? *m : -1, std::get<1>(arguments), 0, 0}); const auto status = Status(Tag::name); - if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) - *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("rmsnorm", *m); return status; } else if constexpr (std::is_same_v) { auto* m = std::get<0>(arguments); auto* desc = std::get<1>(arguments); state.mha_pad_calls.push_back({m ? *m : -1, desc ? *desc : ryzenai_corelib_flat_mha_bf16_desc{}}); const auto status = Status(Tag::name); - if (status == ryzenai_corelib_status_success && m && state.pad_multiple > 0 && *m != 1) - *m = (*m + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("mha", *m); return status; } else if constexpr (std::is_same_v) { const auto status = Status(Tag::name); @@ -361,6 +434,7 @@ void Reset() { state.pad_multiple = 64; state.matmul_k_delta = 0; state.matmul_n_delta = 0; + state.pad_row_overrides.clear(); state.tensor_creates.clear(); state.tensor_windows.clear(); state.weight_creates.clear(); diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp index 7f372605..e63bd91b 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.hpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -99,6 +99,8 @@ struct State { std::int64_t pad_multiple{64}; std::int64_t matmul_k_delta{0}; std::int64_t matmul_n_delta{0}; + std::unordered_map> pad_row_overrides; std::vector tensor_creates; std::vector tensor_windows; std::vector weight_creates; diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index 70f078da..475a6eec 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -6,6 +6,8 @@ #include #include +#include +#include #include #include #include @@ -26,8 +28,9 @@ struct Harness { std::shared_ptr package; std::unique_ptr engine; - Harness() { + explicit Harness(std::function configure = {}) { fake_corelib::Reset(); + if (configure) configure(fake_corelib::GetState()); runtime = CorelibRuntime::CreateForTest( CorelibApi::ResolveForTest(fake_corelib::Resolver())); package = Phi4GgufPackage::Open(FullPackagePath()); @@ -53,6 +56,23 @@ void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { TEST_REQUIRE(state.tensor_creates[7].shape == std::vector({1, 200064})); } +void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { + Harness h([](auto& state) { + state.pad_row_overrides["matmul-3072"][2048] = 5000; + state.pad_row_overrides["matmul-1024"][2048] = 6000; + state.pad_row_overrides["ssmlp"][2048] = 7000; + state.pad_row_overrides["rmsnorm"][2048] = 8000; + state.pad_row_overrides["mha"][2048] = 9000; + }); + const auto& tensors = fake_corelib::GetState().tensor_creates; + TEST_REQUIRE(tensors[0].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[1].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[2].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[3].shape == std::vector({9000, 3072})); + TEST_REQUIRE(tensors[4].shape == std::vector({9000, 1024})); + TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); +} + void TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight() { Harness h; const auto& records = fake_corelib::GetState().weight_creates; @@ -189,13 +209,15 @@ void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { } void TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket() { - Harness h; + Harness h([](auto& state) { + state.pad_row_overrides["matmul-1024"][2] = 96; + }); fake_corelib::GetState().tensor_writes.clear(); std::vector ids{1, 2}; (void)h.engine->prefill(ids); const auto& writes = fake_corelib::GetState().tensor_writes; - TEST_REQUIRE(writes[0].count == 64 * 3072); - TEST_REQUIRE(writes[1].count == 64 * 3072); + TEST_REQUIRE(writes[0].count == 96 * 3072); + TEST_REQUIRE(writes[1].count == 96 * 3072); TEST_REQUIRE(writes[1].all_zero); } @@ -260,13 +282,22 @@ void TestCheckpointRestoreChangesOnlyLogicalPosition() { void TestPreSubmitFailureIsRecoverable() { Harness h; - fake_corelib::GetState().statuses["ryzenai_corelib_rmsnorm_bf16"] = ryzenai_corelib_status_failure; + fake_corelib::GetState().statuses["ryzenai_corelib_rmsnorm_bf16"] = ryzenai_corelib_status_bad_argument; RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "rmsnorm"); TEST_REQUIRE(!h.engine->poisoned()); fake_corelib::GetState().statuses.erase("ryzenai_corelib_rmsnorm_bf16"); (void)h.engine->forward(0); } +void TestInitialRmsNormPostSubmitFailureSynchronizesAndPoisons() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_rmsnorm_bf16"; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "rmsnorm"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_stream_synchronize"] == 1); +} + void TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState() { Harness h; h.engine->set_context_length(3); @@ -295,6 +326,75 @@ void TestPoisonedInstanceRejectsEveryLaterEntryPoint() { RequireContains(RequireThrows([&] { (void)h.engine->prefill(ids); }), "poisoned"); } +void TestFakeTensorWindowRetainsAndPropagatesParentStorage() { + fake_corelib::Reset(); + auto api = CorelibApi::ResolveForTest(fake_corelib::Resolver()); + const std::array parent_shape{16}; + void* parent = nullptr; + api->Check(api->functions().create_device_tensor( + ryzenai_corelib_data_type_bf16, parent_shape.data(), parent_shape.size(), &parent), + "create parent"); + const std::array original{11, 22, 33, 44}; + api->Check(api->functions().tensor_write(parent, ryzenai_corelib_data_type_bf16, + original.data(), original.size(), 4), + "write parent"); + const std::array window_shape{4}; + void* window = nullptr; + api->Check(api->functions().create_tensor_window( + parent, window_shape.data(), window_shape.size(), 4, &window), "create window"); + std::array read{}; + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + read.data(), read.size(), 0), "read window"); + TEST_REQUIRE(read == original); + const std::array replacement{77, 88}; + api->Check(api->functions().tensor_write(window, ryzenai_corelib_data_type_bf16, + replacement.data(), replacement.size(), 1), + "write window"); + std::array reread{}; + api->Check(api->functions().tensor_read(parent, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 4), "read parent"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(parent); + reread.fill(0); + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 0), "reread retained window"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(window); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); +} + +void WriteCacheRow(Harness& h, std::size_t tensor_index, int position, + std::uint16_t base) { + auto& record = fake_corelib::GetState().tensor_creates[tensor_index]; + for (std::size_t head = 0; head < 8; ++head) { + std::array values{}; + values.fill(static_cast(base + head)); + h.runtime->api()->Check(h.runtime->api()->functions().tensor_write( + record.object, ryzenai_corelib_data_type_bf16, values.data(), values.size(), + (head * 4096 + position) * 128), "seed cache row"); + } +} + +void TestGetKCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 10, 7, 100); + const auto result = h.engine->get_k_cache(0, 7); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 100 + head); +} + +void TestGetVCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 11, 9, 200); + const auto result = h.engine->get_v_cache(0, 9); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 200 + head); +} + void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { Harness h; fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_ssmlp_bf16"; @@ -306,6 +406,7 @@ void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { int main() { #define RUN_TEST(name) RunTest(&name, #name) RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); + RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); RUN_TEST(TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight); RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64Threads0); RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); @@ -325,9 +426,13 @@ int main() { RUN_TEST(TestClearContextResetsLogicalPositionWithoutRecreatingWeights); RUN_TEST(TestCheckpointRestoreChangesOnlyLogicalPosition); RUN_TEST(TestPreSubmitFailureIsRecoverable); + RUN_TEST(TestInitialRmsNormPostSubmitFailureSynchronizesAndPoisons); RUN_TEST(TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState); RUN_TEST(TestSynchronizeFailurePoisonsAndClearsState); RUN_TEST(TestPoisonedInstanceRejectsEveryLaterEntryPoint); + RUN_TEST(TestFakeTensorWindowRetainsAndPropagatesParentStorage); + RUN_TEST(TestGetKCacheGathersHeadMajorPosition); + RUN_TEST(TestGetVCacheGathersHeadMajorPosition); RUN_TEST(TestCancellationBoundaryLeavesNoOutstandingFakeWork); #undef RUN_TEST } From 5d503617e5a0529413b6e4a8865b14e398b94021 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 04:38:49 -0700 Subject: [PATCH 10/37] feat: route Phi-4 GGUF models through AIE4 --- src/common/AutoModel/automodel.cpp | 45 +- src/common/AutoModel/modeling_phi4.cpp | 433 +++++++++++++---- src/include/AutoModel/automodel.hpp | 22 + src/include/AutoModel/modeling_phi4.hpp | 76 ++- src/runner/runner.cpp | 2 + src/server/rest_handler.cpp | 131 ++++-- src/server/server.cpp | 41 +- src/server/server.hpp | 31 +- src/test/phi4_corelib_aie4/CMakeLists.txt | 61 ++- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 440 ++++++++++++++++++ 10 files changed, 1079 insertions(+), 203 deletions(-) create mode 100644 src/test/phi4_corelib_aie4/test_phi4_frontend.cpp diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index 5df2f2d2..b7ef3efb 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -8,6 +8,14 @@ #include "AutoModel/automodel.hpp" +ModelRequestError::ModelRequestError( + int http_code, bool session_cleared, std::string message) + : std::runtime_error(std::move(message)), + http_code_(http_code), session_cleared_(session_cleared) {} + +int ModelRequestError::http_code() const noexcept { return http_code_; } +bool ModelRequestError::session_cleared() const noexcept { return session_cleared_; } + AutoModel::AutoModel(flm_rt::device* npu_device_inst, std::string current_model) { this->npu_device_inst = npu_device_inst; this->current_model = current_model; @@ -116,34 +124,39 @@ void AutoModel::_shared_load_model(std::string model_path, json model_info, int header_print("FLM", "Model already loaded: " << this->model_path); return; } + const int context_length = default_context_length != -1 + ? default_context_length + : model_info["default_context_length"].get(); + this->_shared_initialize_model_state( + std::move(model_path), std::move(model_info), context_length); + this->_shared_initialize_legacy_npu(enable_preemption); +} - this->model_path = model_path; +void AutoModel::_shared_initialize_model_state( + std::string model_path, json, int context_length) { + this->model_path = std::move(model_path); header_print("FLM", "Loading model: " << this->model_path); this->lm_config = std::make_unique(); this->lm_config->from_pretrained(this->model_path); - if (this->npu_device_inst == nullptr) { - header_print("ERROR", "NPU device instance is nullptr"); - exit(1); - } - this->npu = std::make_unique(npu_device::device_npu2, this->npu_device_inst, enable_preemption); - this->enable_preemption = enable_preemption; - // Set context length: use provided value if not -1, otherwise use model default - if (default_context_length != -1) { - this->MAX_L = default_context_length; - } else { - this->MAX_L = model_info["default_context_length"]; - } - + this->MAX_L = context_length; this->is_model_loaded = true; - this->token_history.clear(); this->token_history.reserve(this->MAX_L); this->tokenizer = std::make_unique(this->model_path); - this->last_token = -1; this->total_tokens = 0; } +void AutoModel::_shared_initialize_legacy_npu(bool enable_preemption) { + if (this->npu_device_inst == nullptr) { + header_print("ERROR", "NPU device instance is nullptr"); + exit(1); + } + this->npu = std::make_unique( + npu_device::device_npu2, this->npu_device_inst, enable_preemption); + this->enable_preemption = enable_preemption; +} + bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled, void* payload, int first_len_run) { // print token history diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 06c164f5..39a52577 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -1,125 +1,372 @@ -/// \file phi4.cpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class - +/// \file modeling_phi4.cpp +/// \brief Phi-4 frontend and backend routing #include "AutoModel/modeling_phi4.hpp" -/************ Phi4 family **************/ -Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" +#endif + +#include +#include +#include +#include +#include +#include -void Phi4::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == phi4 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - - this->lm_engine->clear_context(); - this->setup_tokenizer(model_path); - this->sampler.reset(); +namespace { +constexpr std::string_view kAie4Backend = "corelib_aie4_gguf"; +constexpr std::string_view kAie4Gguf = "Phi-4-mini-instruct.Q8_0.gguf"; +constexpr int kAie4DecodeLimit = 4095; + +enum class Phi4Backend { LegacyNpu2, CorelibAie4Gguf }; + +Phi4Backend ResolveBackend(const json& model_info) { + const auto details = model_info.find("details"); + if (details == model_info.end() || !details->is_object() || + !details->contains("execution_backend")) { + return Phi4Backend::LegacyNpu2; + } + const auto& backend = details->at("execution_backend"); + if (!backend.is_string()) { + throw std::invalid_argument( + "Phi-4 details.execution_backend must be a string"); + } + const std::string value = backend.get(); + if (value == kAie4Backend) return Phi4Backend::CorelibAie4Gguf; + throw std::invalid_argument("Unknown Phi-4 execution backend '" + value + "'"); +} + +std::uint32_t ResolveContext(const json& model_info, int requested) { + const std::int64_t value = requested == -1 + ? model_info.at("default_context_length").get() + : requested; + if (value < 1 || value > 4096) { + throw std::out_of_range("Phi-4 AIE4 context length must be in 1..4096"); + } + return static_cast(value); +} +nlohmann::json ReadJson(const std::filesystem::path& path) { + std::ifstream input(path, std::ios::binary); + if (!input) throw std::runtime_error("Cannot open " + path.string()); + try { + return nlohmann::json::parse(input); + } catch (const std::exception& error) { + throw std::runtime_error("Cannot parse " + path.string() + ": " + error.what()); + } +} + +void ConfigureSampler(Phi4& model) { sampler_config config; config.top_k = 40; config.top_p = 0.9; config.min_p = 0.1; config.temperature = 0.8; + model.set_sampler(config); +} +} // namespace + +#if defined(FLM_CORELIB_TESTING) +Phi4::EngineFactoryForTesting Phi4::engine_factory_for_testing_; +std::function Phi4::engine_poisoned_for_testing_; +#endif + +Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} + +void Phi4::load_model(std::string model_path, json model_info, + int default_context_length, bool enable_preemption) { + const Phi4Backend backend = ResolveBackend(model_info); + if (backend == Phi4Backend::LegacyNpu2) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + uses_corelib_aie4_ = false; + aie4_poisoned_ = false; + corelib_runtime_.reset(); +#endif + _shared_load_model(model_path, model_info, default_context_length, enable_preemption); + std::unique_ptr engine; +#if defined(FLM_CORELIB_TESTING) + if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); + engine = engine_factory_for_testing_(false, *lm_config, npu.get(), model_path, MAX_L); +#else + q4nx = std::make_unique(this->model_path); + engine = std::make_unique(*lm_config, npu.get(), MAX_L); + engine->load_weights(*q4nx); + q4nx.reset(); +#endif + engine->clear_context(); + setup_tokenizer(model_path); + lm_engine = std::move(engine); + sampler.reset(); + ConfigureSampler(*this); + } else { +#if !defined(FLM_ENABLE_CORELIB_AIE4) + throw std::runtime_error( + "This binary was built without Phi-4 AIE4 corelib support"); +#else + if (enable_preemption) { + throw std::invalid_argument("Phi-4 AIE4 does not support preemption"); + } + const std::uint32_t context_length = ResolveContext(model_info, default_context_length); + const std::filesystem::path root(model_path); + + // Read and validate every source of truth before runtime acquisition or + // engine/device creation. There is deliberately no alternate filename. + const auto config = ReadJson(root / "config.json"); + const auto tokenizer_json = ReadJson(root / "tokenizer.json"); + const auto tokenizer_config = ReadJson(root / "tokenizer_config.json"); + auto package = flm::phi4::Phi4GgufPackage::Open(root / kAie4Gguf); + package->ValidatePhi4Contract(config, tokenizer_json, tokenizer_config); + + uses_corelib_aie4_ = false; + aie4_poisoned_ = false; + try { + _shared_initialize_model_state(model_path, model_info, + static_cast(context_length)); + npu.reset(); + this->enable_preemption = false; + setup_tokenizer(model_path, &tokenizer_config); + sampler.reset(); + ConfigureSampler(*this); - this->set_sampler(config); - for (size_t i = 0; i < PROFILER_TYPE_NUM; i++) { - this->profiler_list[i].reset(); + std::unique_ptr engine; +#if defined(FLM_CORELIB_TESTING) + if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); + engine = engine_factory_for_testing_(true, *lm_config, nullptr, + root, context_length); +#else + auto runtime = flm::corelib::CorelibRuntime::GetOrCreate( + utils::get_executable_directory()); + engine = std::make_unique( + *lm_config, package, runtime, context_length); + corelib_runtime_ = std::move(runtime); +#endif + engine->clear_context(); + lm_engine = std::move(engine); + uses_corelib_aie4_ = true; + } catch (...) { + lm_engine.reset(); + corelib_runtime_.reset(); + tokenizer.reset(); + sampler.reset(); + lm_config.reset(); + is_model_loaded = false; + uses_corelib_aie4_ = false; + throw; + } +#endif } + + for (auto& item : profiler_list) item.reset(); } -void Phi4::setup_tokenizer(std::string model_path) { - // load tokenizer configurations - #ifdef _WIN32 - std::string tokenizer_config_path = model_path + "\\tokenizer_config.json"; - #else - std::string tokenizer_config_path = model_path + "/tokenizer_config.json"; - #endif - std::ifstream fs_config(tokenizer_config_path, std::ios::in | std::ios::binary); - if (fs_config.fail()) { - std::cerr << "Cannot open " << tokenizer_config_path << std::endl; - exit(1); - } - std::string data_config; - fs_config.seekg(0, std::ios::end); - size_t size_config = static_cast(fs_config.tellg()); - fs_config.seekg(0, std::ios::beg); - data_config.resize(size_config); - fs_config.read(data_config.data(), size_config); - fs_config.close(); - auto tokenizer_config = nlohmann::json::parse(data_config); - this->has_bos_token = false; - // load chat template - this->chat_tmpl = std::make_unique( - tokenizer_config["chat_template"], - "", - "" - ); - - if (this->has_bos_token) { - this->bos_token_id = tokenizer_config["bos_token_id"].get(); - } - else { - this->bos_token_id = -1; - } - this->eos_token = ""; - for (auto& token : tokenizer_config["eos_token_id"]) { - this->eos_token_ids.push_back(token.get()); +void Phi4::setup_tokenizer(const std::string& model_path, + const nlohmann::json* verified_tokenizer_config) { + nlohmann::json config = verified_tokenizer_config + ? *verified_tokenizer_config + : ReadJson(std::filesystem::path(model_path) / "tokenizer_config.json"); + if (!config.contains("chat_template") || !config["chat_template"].is_string()) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string chat_template"); + + auto chat = std::make_unique( + config["chat_template"].get(), "", ""); + std::vector eos; + if (verified_tokenizer_config) { + // ValidatePhi4Contract proved these exact independent sources. + eos = {200020, 199999}; + } else { + if (!config.contains("eos_token_id")) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires eos_token_id"); + const auto& ids = config["eos_token_id"]; + if (ids.is_number_integer()) eos.push_back(ids.get()); + else if (ids.is_array()) for (const auto& id : ids) eos.push_back(id.get()); + else throw std::invalid_argument("Phi-4 tokenizer_config.json eos_token_id must be integer or array"); } - this->user_system_prompt = ""; - this->extra_context["user_system_prompt"] = this->user_system_prompt; + has_bos_token = false; + bos_token_id = -1; + eos_token.clear(); + eos_token_ids = std::move(eos); + chat_tmpl = std::move(chat); + user_system_prompt.clear(); + extra_context["user_system_prompt"] = user_system_prompt; } -std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools) { +std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json) { minja::chat_template_inputs inputs; inputs.add_generation_prompt = true; inputs.messages = messages; - inputs.extra_context = this->extra_context; - return this->chat_tmpl->apply(inputs); + inputs.extra_context = extra_context; + return chat_tmpl->apply(inputs); } -bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled) { - // preprocess - this->profiler_list[TKOEN_ENCODE_TIME].start(); - std::string templated_text; - if (input.messages.empty() && input.prompt.empty()) { - header_print("WARNING", "No messages or prompt provided"); - return false; - } - if (!input.messages.empty()) { // already a formated messages, usually from REST API - templated_text = this->apply_chat_template(input.messages); +#if defined(FLM_ENABLE_CORELIB_AIE4) +bool Phi4::engine_is_poisoned() const noexcept { +#if defined(FLM_CORELIB_TESTING) + return engine_poisoned_for_testing_ && lm_engine + ? engine_poisoned_for_testing_(lm_engine.get()) + : false; +#else + const auto* engine = dynamic_cast(lm_engine.get()); + return engine && engine->poisoned(); +#endif +} + +void Phi4::validate_aie4_capacity(std::size_t rendered_tokens, + std::optional requested) const { + const std::size_t cap = std::min(MAX_L, kAie4DecodeLimit); + const auto normalized = normalize_requested_max_new_tokens(requested); + if (rendered_tokens >= cap || + (normalized && static_cast(*normalized) > cap - rendered_tokens)) { + std::ostringstream message; + message << "Phi-4 AIE4 request exceeds the 4095-token decode limit: rendered prompt has " + << rendered_tokens << " tokens"; + if (normalized) message << " and requested output has " << *normalized << " tokens"; + throw ModelRequestError(400, false, message.str()); } - else if (!input.prompt.empty()) { // a pure text, usually from the cli - nlohmann::ordered_json messages; +} - messages.push_back({ {"role", "user"}, {"content", input.prompt} }); - templated_text = this->apply_chat_template(messages); +void Phi4::clear_after_inference_failure(bool poisoned) { + aie4_poisoned_ = poisoned; + total_tokens = 0; + last_token = -1; + token_history.clear(); + checkpoint_his.clear(); + if (!poisoned && lm_engine) { + try { lm_engine->clear_context(); } catch (...) {} } + if (sampler) sampler->reset_penalties(); +} - std::vector tokens = this->tokenizer->encode(templated_text); - this->profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); - // hardware +std::string Phi4::generate_aie4(chat_meta_info_t& meta_info, + std::ostream& os, + std::function is_cancelled) { + std::string result; + meta_info.stop_reason = EOT_DETECTED; + int generated = 0; + while (last_token != -1 && generated < aie4_generation_budget_) { + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + break; + } + const int token = last_token; + token_history.push_back(token); + ++total_tokens; + ++generated; + ++meta_info.generated_tokens; + if (is_normal_token(token)) { + const std::string text = tokenizer->run_time_decoder(token); + result += text; + os << text << std::flush; + } + if (is_eos(token)) { + last_token = -1; + break; + } + if (generated >= aie4_generation_budget_ || total_tokens >= std::min(MAX_L, kAie4DecodeLimit)) { + last_token = -1; + meta_info.stop_reason = MAX_LENGTH_REACHED; + break; + } + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + break; + } + auto logits = lm_engine->forward(token); + last_token = sampler->sample(logits); + } + return result; +} +#endif - return this->_shared_insert(meta_info, tokens, is_cancelled); +void Phi4::clear_context() { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_ && aie4_poisoned_) { + total_tokens = 0; + last_token = -1; + token_history.clear(); + checkpoint_his.clear(); + if (sampler) sampler->reset_penalties(); + return; + } +#endif + AutoModel::clear_context(); } +bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_ && aie4_poisoned_) { + throw ModelRequestError(500, true, + "Phi-4 AIE4 model is poisoned; unload/reload is required"); + } +#endif + profiler_list[TKOEN_ENCODE_TIME].start(); + std::string rendered; + if (input.messages.empty() && input.prompt.empty()) return false; + if (!input.messages.empty()) rendered = apply_chat_template(input.messages); + else { + nlohmann::ordered_json messages = nlohmann::ordered_json::array(); + messages.push_back({{"role", "user"}, {"content", input.prompt}}); + rendered = apply_chat_template(messages); + } + std::vector tokens = tokenizer->encode(rendered); + profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); -std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled) { - return this->_shared_generate(meta_info, length_limit, os, is_cancelled); +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + validate_aie4_capacity(tokens.size(), input.requested_max_new_tokens); + const auto normalized = normalize_requested_max_new_tokens(input.requested_max_new_tokens); + const int remaining = static_cast( + std::min(MAX_L, kAie4DecodeLimit) - tokens.size()); + aie4_generation_budget_ = normalized ? *normalized : remaining; + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + return false; + } + try { + return _shared_insert(meta_info, tokens, std::move(is_cancelled)); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + const bool poisoned = engine_is_poisoned(); + clear_after_inference_failure(poisoned); + throw ModelRequestError(500, true, poisoned + ? "AIE4 inference failed; unload/reload is required because the model is poisoned" + : "AIE4 inference failed; the current conversation was cleared"); + } + } +#endif + return _shared_insert(meta_info, tokens, std::move(is_cancelled)); } -std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os) { - if (!this->insert(meta_info, input)) { - return ""; +std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + if (aie4_poisoned_) throw ModelRequestError(500, true, + "Phi-4 AIE4 model is poisoned; unload/reload is required"); + try { + return generate_aie4(meta_info, os, std::move(is_cancelled)); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + const bool poisoned = engine_is_poisoned(); + clear_after_inference_failure(poisoned); + throw ModelRequestError(500, true, poisoned + ? "AIE4 inference failed; unload/reload is required because the model is poisoned" + : "AIE4 inference failed; the current conversation was cleared"); + } } - return this->_shared_generate(meta_info, length_limit, os); -} \ No newline at end of file +#endif + return _shared_generate(meta_info, length_limit, os, std::move(is_cancelled)); +} + +std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os) { + if (!insert(meta_info, input)) return {}; + return generate(meta_info, length_limit, os); +} diff --git a/src/include/AutoModel/automodel.hpp b/src/include/AutoModel/automodel.hpp index ed3a6726..f5fafad1 100644 --- a/src/include/AutoModel/automodel.hpp +++ b/src/include/AutoModel/automodel.hpp @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include "typedef.hpp" #include "causal_lm.hpp" #include "lm_config.hpp" @@ -128,10 +130,26 @@ struct lm_uniform_input_t { std::vector audios; std::vector audio_payload_types; nlohmann::ordered_json tools; + std::optional requested_max_new_tokens; }; +inline std::optional normalize_requested_max_new_tokens( + std::optional requested) { + return requested.has_value() && *requested > 0 ? requested : std::nullopt; +} + using json = nlohmann::ordered_json; +class ModelRequestError final : public std::runtime_error { +public: + ModelRequestError(int http_code, bool session_cleared, std::string message); + int http_code() const noexcept; + bool session_cleared() const noexcept; +private: + int http_code_; + bool session_cleared_; +}; + class AutoModel { protected: std::string model_path = ""; @@ -188,6 +206,8 @@ class AutoModel { void _shared_load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false); + void _shared_initialize_model_state(std::string model_path, json model_info, int context_length); + void _shared_initialize_legacy_npu(bool enable_preemption); nlohmann::json _shared_setup_tokenizer(std::string model_path); /// \brief Insert tokens into the model @@ -220,6 +240,8 @@ class AutoModel { /// \return the current model std::string get_current_model(); + virtual bool uses_corelib_aie4() const noexcept { return false; } + /// \brief Get the current context length /// \return the current context length virtual int get_current_context_length(); diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index 66937d6d..ce71f5d9 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -1,24 +1,68 @@ -/// \file phi4.hpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class +/// \file modeling_phi4.hpp +/// \brief Phi-4 frontend and backend routing #pragma once #include "AutoModel/automodel.hpp" -/************ phi4 family **************/ +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "corelib/corelib_runtime.hpp" +#endif + +#if defined(FLM_CORELIB_TESTING) +#include +#include +namespace flm::phi4::testing { class Phi4FrontendTestAccess; } +#endif + class Phi4 : public AutoModel { private: - void setup_tokenizer(std::string model_path); + void setup_tokenizer(const std::string& model_path, + const nlohmann::json* verified_tokenizer_config = nullptr); +#if defined(FLM_ENABLE_CORELIB_AIE4) + void validate_aie4_capacity(std::size_t rendered_tokens, + std::optional requested) const; + bool engine_is_poisoned() const noexcept; + void clear_after_inference_failure(bool poisoned); + std::string generate_aie4(chat_meta_info_t& meta_info, + std::ostream& os, + std::function is_cancelled); -public: - Phi4(flm_rt::device* npu_device_inst); + bool uses_corelib_aie4_ = false; + bool aie4_poisoned_ = false; + int aie4_generation_budget_ = 0; + std::shared_ptr corelib_runtime_; +#endif - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; - //void toggle_enable_think() override; - bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; - std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; - std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; - std::string apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; +#if defined(FLM_CORELIB_TESTING) + using EngineFactoryForTesting = std::function( + bool, const LM_Config&, npu_xclbin_manager*, + const std::filesystem::path&, std::uint32_t)>; + static EngineFactoryForTesting engine_factory_for_testing_; + static std::function engine_poisoned_for_testing_; + friend class flm::phi4::testing::Phi4FrontendTestAccess; +#endif + +public: + explicit Phi4(flm_rt::device* npu_device_inst); + void load_model(std::string model_path, json model_info, + int default_context_length = -1, + bool enable_preemption = false) override; + bool uses_corelib_aie4() const noexcept override { +#if defined(FLM_ENABLE_CORELIB_AIE4) + return uses_corelib_aie4_; +#else + return false; +#endif + } + void clear_context() override; + bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled = [] { return false; }) override; + std::string generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled = [] { return false; }) override; + std::string generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os = std::cout) override; + std::string apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; }; diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index d38e5e9c..dab30470 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -358,6 +358,8 @@ void Runner::run() { chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; uniformed_input.prompt = input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(this->generate_limit); this->auto_chat_engine->start_total_timer(); diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index 95e90a3a..6ea22272 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -20,6 +20,21 @@ #include #include "server.hpp" +namespace { +json ModelErrorResponse(const ModelRequestError& error) { + return {{"error", {{"message", error.what()}, + {"type", "model_error"}, + {"code", error.http_code()}, + {"session_cleared", error.session_cleared()}}}}; +} + +json ExceptionResponse(const std::exception& error) { + if (const auto* model_error = dynamic_cast(&error)) + return ModelErrorResponse(*model_error); + return {{"error", error.what()}}; +} +} + ///@brief Normalize messages by merging consecutive user messages (like Ollama does) ///@param messages the original messages ///@return normalized messages with consecutive user messages merged @@ -655,6 +670,11 @@ void RestHandler::handle_generate(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); meta_info.max_prefill_len = this->prefill_chunk_len; meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; header_print("FLM", "Start generating..."); @@ -665,7 +685,7 @@ void RestHandler::handle_generate(const json& request, streaming_ostream ostream(model, send_streaming_response, false); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -673,15 +693,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -698,7 +718,7 @@ void RestHandler::handle_generate(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -706,15 +726,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -738,7 +758,7 @@ void RestHandler::handle_generate(const json& request, send_response(response); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -772,6 +792,11 @@ void RestHandler::handle_chat(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + options.contains("num_predict") + ? std::optional(options.at("num_predict").get()) + : std::nullopt); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; header_print("FLM", "Start generating..."); @@ -781,7 +806,7 @@ void RestHandler::handle_chat(const json& request, streaming_ostream ostream(model, send_streaming_response, true); // true for chat format uniformed_input.messages = messages; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -789,21 +814,16 @@ void RestHandler::handle_chat(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); - if (!success){ - json error_response = {{"error", "Max length reached"}}; - send_response(error_response); - this->auto_chat_engine->clear_context(); - return; - } + auto_chat_engine->generate(meta_info, length_limit, ostream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -825,7 +845,7 @@ void RestHandler::handle_chat(const json& request, try { response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, nstream); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -857,7 +877,7 @@ void RestHandler::handle_chat(const json& request, this->auto_chat_engine->clear_context(); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -915,7 +935,7 @@ void RestHandler::handle_embeddings(const json& request, send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -931,7 +951,7 @@ void RestHandler::handle_models(const json& request, json models = supported_models.get_all_models_ollama(); send_response(models); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1020,7 +1040,7 @@ void RestHandler::handle_ps(const json& request, // std::cout << "response: " << response.dump(4) << std::endl; send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1146,13 +1166,22 @@ void RestHandler::handle_openai_chat_completion(const json& request, lm_uniform_input_t uniformed_input; uniformed_input.messages = current_messages; uniformed_input.tools = tools; + const std::optional openai_chat_budget = request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : request.contains("max_completion_tokens") + ? std::optional(request.at("max_completion_tokens").get()) + : std::nullopt; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(openai_chat_budget); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; if (stream){ // Create a wrapper callback that passes the pre-formatted SSE string directly cancellation_token->reset(); auto_chat_engine->reset_parser(); - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; @@ -1184,7 +1213,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1193,8 +1222,18 @@ void RestHandler::handle_openai_chat_completion(const json& request, header_print("FLM", "Start generating..."); try { auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + if (error.session_cleared()) this->prompt_cache.reset(); + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1236,7 +1275,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1246,7 +1285,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, try { response_text = auto_chat_engine->generate(meta_info, length_limit, nstream, [&] { return cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1389,18 +1428,25 @@ void RestHandler::handle_openai_completion(const json& request, chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); header_print("FLM", "Start generating..."); if (stream) { // Create a wrapper callback that passes the pre-formatted SSE string directly - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; streaming_ostream_openai ostream(model, openai_stream_callback); // streaming in completion format uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1408,15 +1454,24 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -1431,7 +1486,7 @@ void RestHandler::handle_openai_completion(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1439,15 +1494,15 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; diff --git a/src/server/server.cpp b/src/server/server.cpp index bc612211..cf2e4c2c 100644 --- a/src/server/server.cpp +++ b/src/server/server.cpp @@ -156,19 +156,6 @@ int NPUAccessManager::get_active_npu_requests() { return g_npu_active_requests.load(); } -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path) { - // NPU-intensive endpoints that should be restricted to one user at a time - if (method == "POST") { - return path == "/api/generate" || - path == "/api/chat" || - path == "/v1/chat/completions" || - path == "/v1/audio/transcriptions" || - path == "/v1/embeddings"; - } - return false; -} - ///@brief HttpSession class implementation ///@param socket the socket ///@param server the server @@ -684,6 +671,9 @@ bool WebServer::handle_request(http::request& req, auto process_task = [this, it, req_ptr, res_ptr, session, needs_npu, key, is_json](bool is_deferred) { auto& req_ref = *req_ptr; auto& res_ref = *res_ptr; + NPURequestCompletionGuard completion([this, needs_npu] { + if (needs_npu) process_next_npu_request(); + }); // Parse JSON request body json request_json; @@ -704,10 +694,6 @@ bool WebServer::handle_request(http::request& req, // Only write from callback when deferred if (is_deferred && session) session->write_response_from_callback(); - - if (needs_npu) { - this->process_next_npu_request(); - } return; } @@ -737,10 +723,9 @@ bool WebServer::handle_request(http::request& req, if (code == 400) { status = http::status::bad_request; + } else if (code == 500) { + status = http::status::internal_server_error; } - //else if () { - - //} } response_ref.result(status); @@ -750,10 +735,6 @@ bool WebServer::handle_request(http::request& req, cancellation_token->complete(); unregister_active_request(request_id); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -769,10 +750,6 @@ bool WebServer::handle_request(http::request& req, } if (is_final) { unregister_active_request(request_id); - - if (needs_npu) { - this->process_next_npu_request(); - } } }; @@ -788,10 +765,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -805,10 +778,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } diff --git a/src/server/server.hpp b/src/server/server.hpp index 910c5884..1bfa7d86 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -47,8 +47,35 @@ extern std::mutex g_npu_access_mutex; extern std::atomic g_npu_in_use; extern std::atomic g_npu_active_requests; -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path); +// Helper function to check if an endpoint requires serialized accelerator access. +inline bool requires_npu_access(const std::string& method, const std::string& path) { + if (method != "POST") return false; + return path == "/api/generate" || path == "/api/chat" || + path == "/v1/chat/completions" || path == "/v1/completions" || + path == "/v1/audio/transcriptions" || path == "/v1/embeddings"; +} + +class NPURequestCompletionGuard final { +public: + explicit NPURequestCompletionGuard(std::function completion) + : completion_(std::move(completion)) {} + NPURequestCompletionGuard(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard& operator=(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard(NPURequestCompletionGuard&& other) noexcept + : completion_(std::move(other.completion_)), active_(other.active_) { + other.active_ = false; + } + NPURequestCompletionGuard& operator=(NPURequestCompletionGuard&&) = delete; + ~NPURequestCompletionGuard() { complete(); } + void complete() noexcept { + if (!active_) return; + active_ = false; + try { if (completion_) completion_(); } catch (...) {} + } +private: + std::function completion_; + bool active_ = true; +}; ///@brief get current time string, format: hh:mm:ss mm:dd:yyyy ///@return the current time string diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index 44b7f609..f6d7bf54 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -109,12 +109,63 @@ target_compile_options(test_phi4_engine PRIVATE target_link_directories(test_phi4_engine PRIVATE "${XRT_INCLUDE_DIR}/../lib") target_link_libraries(test_phi4_engine PRIVATE xrt_coreutil) -# Compile the actual production frontend translation unit in both feature modes. +set(PHI4_FRONTEND_SOURCES + "${FLM_SOURCE_DIR}/common/AutoModel/automodel.cpp" + "${FLM_SOURCE_DIR}/common/AutoModel/modeling_phi4.cpp") + +add_executable(test_phi4_frontend + test_phi4_frontend.cpp + ${PHI4_FRONTEND_SOURCES} + ${CORELIB_SOURCES} + "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp") +target_include_directories(test_phi4_frontend PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend PRIVATE + FLM_ENABLE_CORELIB_AIE4=1 FLM_CORELIB_TESTING=1 + RYZENAI_CORELIB_STATIC=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend PRIVATE xrt_coreutil) + +add_executable(test_phi4_frontend_off + test_phi4_frontend.cpp + ${PHI4_FRONTEND_SOURCES}) +target_include_directories(test_phi4_frontend_off PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend_off PRIVATE + FLM_CORELIB_TESTING=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend_off PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend_off PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend_off PRIVATE xrt_coreutil) + +# Compile the actual production frontend translation units in both feature modes. # Empty declaration-only FFmpeg headers isolate this compile check from an # unrelated optional SDK that is absent on the standalone test host. set(FRONTEND_STUB_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/frontend-compile-stubs") foreach(STUB_HEADER IN ITEMS libavcodec/avcodec.h + libavformat/avformat.h libswscale/swscale.h libavutil/imgutils.h libavutil/frame.h @@ -125,7 +176,11 @@ foreach(STUB_HEADER IN ITEMS file(WRITE "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" "#pragma once\n") endforeach() -set(FLM_PRODUCTION_FRONTEND_SOURCES "${FLM_SOURCE_DIR}/src/main.cpp") +set(FLM_PRODUCTION_FRONTEND_SOURCES + ${PHI4_FRONTEND_SOURCES} + "${FLM_SOURCE_DIR}/runner/runner.cpp" + "${FLM_SOURCE_DIR}/server/rest_handler.cpp" + "${FLM_SOURCE_DIR}/server/server.cpp") function(add_frontend_compile_guard TARGET_NAME ENABLE_CORELIB) add_library(${TARGET_NAME} OBJECT ${FLM_PRODUCTION_FRONTEND_SOURCES}) target_include_directories(${TARGET_NAME} PRIVATE @@ -164,4 +219,6 @@ add_test(NAME test_phi4_gguf COMMAND test_phi4_gguf) add_test(NAME test_phi4_host COMMAND test_phi4_host) add_test(NAME test_phi4_shape_plan COMMAND test_phi4_shape_plan) add_test(NAME test_phi4_engine COMMAND test_phi4_engine) +add_test(NAME test_phi4_frontend COMMAND test_phi4_frontend) +add_test(NAME test_phi4_frontend_off COMMAND test_phi4_frontend_off) set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp new file mode 100644 index 00000000..aac3d834 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -0,0 +1,440 @@ +#include "test_support.hpp" +#include "gguf_fixture.hpp" + +#include +#include "server.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +std::vector g_encoded_tokens; +std::vector g_samples; +std::size_t g_sample_index{}; + +class FakeEngine final : public causal_lm { +public: + explicit FakeEngine(std::uint32_t limit) : max_length(limit) {} + + buffer forward(int token) override { + ++forward_calls; + forwarded.push_back(token); + if (fail_forward) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + ++position; + return buffer(1); + } + buffer prefill(std::vector& tokens, void*) override { + ++prefill_calls; + if (fail_prefill) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + position += static_cast(tokens.size()); + return buffer(1); + } + void set_context_length(int value) override { position = value; } + void load_weights(Q4NX&) override {} + void update_max_length(std::uint32_t value) override { max_length = value; } + void clear_context() override { + if (poisoned_state) throw std::runtime_error("poisoned"); + position = 0; + } + buffer get_k_cache(int, int) override { return buffer(1); } + buffer get_v_cache(int, int) override { return buffer(1); } + int get_current_context_length() override { return position; } + int checkpoint() override { return position; } + int restore() override { return position; } + + std::uint32_t max_length; + int position{}; + int prefill_calls{}; + int forward_calls{}; + bool fail_prefill{}; + bool fail_forward{}; + bool poisoned_state{}; + std::vector forwarded; +}; + +struct FactoryState { + int legacy_calls{}; + int aie4_calls{}; + bool throw_for_aie4{}; + FakeEngine* engine{}; +} g_factory; + +class TempPackage final { +public: + explicit TempPackage(bool valid = true) { + static std::uint64_t serial{}; + path_ = std::filesystem::temp_directory_path() / + ("flm-task4-" + std::to_string(++serial)); + std::filesystem::create_directories(path_); + Write(path_ / "config.json", gguf_fixture::ValidConfig()); + Write(path_ / "tokenizer.json", gguf_fixture::ValidTokenizer()); + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["eos_token_id"] = nlohmann::json::array({200020, 199999}); + Write(path_ / "tokenizer_config.json", tokenizer_config); + auto gguf = gguf_fixture::Builder().AddFullContractTensors(false).Write("frontend"); + std::filesystem::rename(gguf.path, path_ / "Phi-4-mini-instruct.Q8_0.gguf"); + if (!valid) { + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + Write(path_ / "config.json", config); + } + } + ~TempPackage() { + std::error_code ignored; + std::filesystem::remove_all(path_, ignored); + } + const std::filesystem::path& path() const { return path_; } +private: + static void Write(const std::filesystem::path& path, const nlohmann::json& value) { + std::ofstream out(path, std::ios::binary); + if (!out) throw std::runtime_error("cannot write package fixture"); + out << value.dump(); + } + std::filesystem::path path_; +}; + +nlohmann::ordered_json ModelInfo(std::optional backend = std::nullopt) { + nlohmann::ordered_json details = {{"family", "phi4"}}; + if (backend) details["execution_backend"] = *backend; + return {{"default_context_length", 4096}, {"details", details}}; +} + +chat_meta_info_t Meta() { + chat_meta_info_t value; + value.max_prefill_len = 64; + return value; +} + +lm_uniform_input_t Input(std::optional budget = std::nullopt) { + lm_uniform_input_t value; + value.prompt = "prompt"; + value.requested_max_new_tokens = budget; + return value; +} + +template +void ExpectRequestError(F&& action, int code, bool cleared, std::string_view text) { + try { action(); } + catch (const ModelRequestError& error) { + TEST_REQUIRE(error.http_code() == code); + TEST_REQUIRE(error.session_cleared() == cleared); + RequireContains(error.what(), text); + return; + } + throw std::runtime_error("expected ModelRequestError"); +} + +} // namespace + +Tokenizer::Tokenizer(const std::string&) { is_doubled_encoded = false; } +Tokenizer::~Tokenizer() = default; +std::vector Tokenizer::encode(const std::string&) { return g_encoded_tokens; } +std::string Tokenizer::decode(const std::vector&) { return "decoded"; } +std::string Tokenizer::run_time_decoder(int token) { return "t" + std::to_string(token); } +SafeTensors::~SafeTensors() = default; + +Sampler::Sampler(int features, sampler_config& config) + : in_features(features), rep_penalty(config.rep_penalty), + freq_penalty(config.freq_penalty), pre_penalty(config.pre_penalty), + top_k(config.top_k), top_p(config.top_p), min_p(config.min_p), + temperature(config.temperature), total_tokens(0), + freq_penalty_window(config.freq_penalty_window), + rep_penalty_window(config.rep_penalty_window), + repeat_last_n(config.repeat_last_n), + use_optimized_sampling(config.use_optimized_sampling) { + logits.resize(1); counters.resize(1); token_positions.resize(1, -1); +} +void Sampler::reset_penalties() {} +int Sampler::sample(buffer&) { + if (g_sample_index < g_samples.size()) return g_samples[g_sample_index++]; + return 7; +} + +namespace utils { +std::string get_executable_directory() { return std::filesystem::current_path().string(); } +} + +namespace flm::phi4::testing { +class Phi4FrontendTestAccess final { +public: + static void InstallFactory() { + g_factory = {}; + Phi4::engine_factory_for_testing_ = + [](bool aie4, const LM_Config&, npu_xclbin_manager*, + const std::filesystem::path&, std::uint32_t limit) { + if (aie4) { + ++g_factory.aie4_calls; + if (g_factory.throw_for_aie4) throw std::runtime_error("missing corelib"); + } else { + ++g_factory.legacy_calls; + } + auto result = std::make_unique(limit); + g_factory.engine = result.get(); + return std::unique_ptr(std::move(result)); + }; + Phi4::engine_poisoned_for_testing_ = [](const causal_lm* engine) { + return static_cast(engine)->poisoned_state; + }; + } + static void RemoveFactory() { + Phi4::engine_factory_for_testing_ = {}; + Phi4::engine_poisoned_for_testing_ = {}; + } + static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } +}; +} // namespace flm::phi4::testing + +namespace { +using flm::phi4::testing::Phi4FrontendTestAccess; + +struct FactoryScope { + FactoryScope() { Phi4FrontendTestAccess::InstallFactory(); } + ~FactoryScope() { Phi4FrontendTestAccess::RemoveFactory(); } +}; + +std::unique_ptr Load(const TempPackage& package, + nlohmann::ordered_json info, + int context = -1, + bool preemption = false, + flm_rt::device* device = reinterpret_cast(1)) { + auto model = std::make_unique(device); + model->load_model(package.path().string(), std::move(info), context, preemption); + return model; +} + +void TestAbsentBackendStillBuildsQ4nxPhi4Npu() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); + TEST_REQUIRE(!model->uses_corelib_aie4()); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + TEST_REQUIRE(g_factory.legacy_calls == 0); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(model->uses_corelib_aie4()); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +void TestUnknownAndNonStringBackendAreErrors() { + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("other")); }), "Unknown"); + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo(7)); }), "string"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.aie4_calls == 0); +} + +void TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders() { +#if !defined(FLM_ENABLE_CORELIB_AIE4) + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { + (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + }), "This binary was built without Phi-4 AIE4 corelib support"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.aie4_calls == 0); +#endif +} + +void TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation() { + TempPackage package(false); + FactoryScope scope; + RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +void TestMissingCorelibFailsOnlyWhenAie4ModelLoads() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }), "missing corelib"); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }); + g_factory.throw_for_aie4 = false; + auto ordinary = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(!ordinary->uses_corelib_aie4()); +} + +void TestPreemptionIsRejectedForTheAie4Route() { + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, true, nullptr); }), "preemption"); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +std::unique_ptr ReadyAie4(const TempPackage& package) { + return Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); +} + +void TestRenderedPromptPlusExplicitBudgetMayEqual4095() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(95); + TEST_REQUIRE(model->insert(meta, input)); +} + +void TestRenderedPromptPlusExplicitBudgetAbove4095Is400() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(96); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 400, false, "4095"); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow() { + for (const auto requested : {std::optional{}, std::optional{0}, std::optional{-1}}) { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(requested); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); + TEST_REQUIRE(meta.stop_reason == MAX_LENGTH_REACHED); + } + + // /api/chat uses generate_with_prompt and retains 4096 only as the legacy + // loop default; omission must not become an explicit AIE4 budget. + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + (void)model->generate_with_prompt(meta, input, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); +} + +void TestCancellationBeforePrefillSubmitsNothing() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + TEST_REQUIRE(!model->insert(meta, input, [] { return true; })); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); +} + +void TestCancellationBetweenDecodeStepsStopsWithCancelReason() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11, 12}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + int checks = 0; + (void)model->generate(meta, 10, output, [&] { return checks++ == 2; }); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->forward_calls == 1); +} + +void TestCancellationReturnsOnlyAfterSynchronize() { + // Fake calls are synchronous by construction: observing one completed call + // before cancellation proves no work remains outstanding at return. + TestCancellationBetweenDecodeStepsStopsWithCancelReason(); +} + +void TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + TEST_REQUIRE(model->get_current_context_length() == 0); +} + +void TestPoisonedModelReturns500UntilReload() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 500, true, "unload/reload"); + auto reloaded = ReadyAie4(package); + TEST_REQUIRE(reloaded->insert(meta, input)); +} + +void TestEosSelfTerminatesWithoutAnExtraDecode() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 0); + TEST_REQUIRE(meta.stop_reason == EOT_DETECTED); +} + +void TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics() { + for (const auto raw : {std::optional{}, std::optional{0}, std::optional{-2}, std::optional{17}}) { + const auto expected = raw && *raw > 0 ? raw : std::nullopt; + for (int source = 0; source < 5; ++source) + TEST_REQUIRE(normalize_requested_max_new_tokens(raw) == expected); + } +} + +void TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint() { + TEST_REQUIRE(requires_npu_access("POST", "/v1/completions")); + for (int path = 0; path < 5; ++path) { + int releases = 0; + { + NPURequestCompletionGuard guard([&] { ++releases; }); + if (path == 0) guard.complete(); + else if (path == 1) { guard.complete(); guard.complete(); } + else if (path == 2) { NPURequestCompletionGuard moved(std::move(guard)); } + else if (path == 3) { try { throw std::runtime_error("model"); } catch (...) {} } + else { try { throw 1; } catch (...) {} } + } + TEST_REQUIRE(releases == 1); + } +} + +} // namespace + +int main() { +#if defined(FLM_ENABLE_CORELIB_AIE4) + RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); + RunTest(TestCorelibAie4GgufBuildsOnlyTheCorelibEngine, "TestCorelibAie4GgufBuildsOnlyTheCorelibEngine"); + RunTest(TestUnknownAndNonStringBackendAreErrors, "TestUnknownAndNonStringBackendAreErrors"); + RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); + RunTest(TestMissingCorelibFailsOnlyWhenAie4ModelLoads, "TestMissingCorelibFailsOnlyWhenAie4ModelLoads"); + RunTest(TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure, "TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure"); + RunTest(TestPreemptionIsRejectedForTheAie4Route, "TestPreemptionIsRejectedForTheAie4Route"); + RunTest(TestRenderedPromptPlusExplicitBudgetMayEqual4095, "TestRenderedPromptPlusExplicitBudgetMayEqual4095"); + RunTest(TestRenderedPromptPlusExplicitBudgetAbove4095Is400, "TestRenderedPromptPlusExplicitBudgetAbove4095Is400"); + RunTest(TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow, "TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow"); + RunTest(TestCancellationBeforePrefillSubmitsNothing, "TestCancellationBeforePrefillSubmitsNothing"); + RunTest(TestCancellationBetweenDecodeStepsStopsWithCancelReason, "TestCancellationBetweenDecodeStepsStopsWithCancelReason"); + RunTest(TestCancellationReturnsOnlyAfterSynchronize, "TestCancellationReturnsOnlyAfterSynchronize"); + RunTest(TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned, "TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned"); + RunTest(TestPoisonedModelReturns500UntilReload, "TestPoisonedModelReturns500UntilReload"); + RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); + RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); + RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); +#else + RunTest(TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders, "TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders"); +#endif + std::cout << "test_phi4_frontend: PASS\n"; +} From 8f831fe242fe10a807005fcd7082c7df55dbb57c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 04:52:33 -0700 Subject: [PATCH 11/37] fix: harden Phi-4 request lifecycle --- src/common/AutoModel/automodel.cpp | 27 +++++++++-- src/common/AutoModel/modeling_phi4.cpp | 16 +++++-- src/include/AutoModel/automodel.hpp | 6 +++ src/server/rest_handler.cpp | 4 +- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 48 ++++++++++++++++++- 5 files changed, 91 insertions(+), 10 deletions(-) diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index b7ef3efb..9bd61ed9 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -157,6 +157,16 @@ void AutoModel::_shared_initialize_legacy_npu(bool enable_preemption) { this->enable_preemption = enable_preemption; } +std::string AutoModel::generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled) { + if (!insert(meta_info, input, is_cancelled)) return {}; + return generate(meta_info, length_limit, os, std::move(is_cancelled)); +} + bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled, void* payload, int first_len_run) { // print token history @@ -227,6 +237,14 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector max_prefill_len = 1 << static_cast(std::ceil(std::log2(max_prefill_len))); buffer y; if (max_prefill_len < 512) { + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + buffer_.clear(); + current_mode_ = StreamEventType::CONTENT; + tool_name_.clear(); + is_in_tool_block_ = false; + return y; + } y = this->lm_engine->prefill(tokens, payload); } else{ @@ -238,19 +256,18 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector } int chunks = (tokens.size() + max_prefill_len - 1) / max_prefill_len; for (int i = 0; i < chunks; i++) { + int start = i * max_prefill_len; + int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); + std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); + header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); if (is_cancelled()) { meta_info.stop_reason = CANCEL_DETECTED; - // reset stream content buffer_.clear(); current_mode_ = StreamEventType::CONTENT; tool_name_.clear(); is_in_tool_block_ = false; break; } - int start = i * max_prefill_len; - int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); - std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); - header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); buffer chunk_y = this->lm_engine->prefill(chunk_tokens, (i == 0)? payload : nullptr); if (i == chunks - 1) { y = chunk_y; diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 39a52577..f17bd51d 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -79,9 +79,11 @@ void Phi4::load_model(std::string model_path, json model_info, const Phi4Backend backend = ResolveBackend(model_info); if (backend == Phi4Backend::LegacyNpu2) { #if defined(FLM_ENABLE_CORELIB_AIE4) + const bool switching_from_aie4 = uses_corelib_aie4_; uses_corelib_aie4_ = false; aie4_poisoned_ = false; corelib_runtime_.reset(); + if (switching_from_aie4) is_model_loaded = false; #endif _shared_load_model(model_path, model_info, default_context_length, enable_preemption); std::unique_ptr engine; @@ -168,10 +170,18 @@ void Phi4::setup_tokenizer(const std::string& model_path, if (!config.contains("chat_template") || !config["chat_template"].is_string()) throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string chat_template"); + const bool aie4 = verified_tokenizer_config != nullptr; + std::string configured_eos; + if (!aie4) { + if (!config.contains("eos_token") || !config["eos_token"].is_string()) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string eos_token"); + configured_eos = config["eos_token"].get(); + } auto chat = std::make_unique( - config["chat_template"].get(), "", ""); + config["chat_template"].get(), "", + aie4 ? "" : configured_eos); std::vector eos; - if (verified_tokenizer_config) { + if (aie4) { // ValidatePhi4Contract proved these exact independent sources. eos = {200020, 199999}; } else { @@ -184,7 +194,7 @@ void Phi4::setup_tokenizer(const std::string& model_path, } has_bos_token = false; bos_token_id = -1; - eos_token.clear(); + eos_token = std::move(configured_eos); eos_token_ids = std::move(eos); chat_tmpl = std::move(chat); user_system_prompt.clear(); diff --git a/src/include/AutoModel/automodel.hpp b/src/include/AutoModel/automodel.hpp index f5fafad1..e9088f9b 100644 --- a/src/include/AutoModel/automodel.hpp +++ b/src/include/AutoModel/automodel.hpp @@ -383,6 +383,12 @@ class AutoModel { /// \brief Generate the tokens with prompt virtual std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) = 0; + std::string generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled); /// \brief Configure a parameter with type-erased value /// \param parameter_name the name of the parameter diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index 6ea22272..c939ae2b 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -843,7 +843,9 @@ void RestHandler::handle_chat(const json& request, //std::string response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, std::cout); std::string response_text; try { - response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, nstream); + response_text = auto_chat_engine->generate_with_prompt( + meta_info, uniformed_input, length_limit, nstream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { json error_response = ExceptionResponse(e); send_response(error_response); diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index aac3d834..bd933e83 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -86,6 +86,7 @@ class TempPackage final { Write(path_ / "config.json", gguf_fixture::ValidConfig()); Write(path_ / "tokenizer.json", gguf_fixture::ValidTokenizer()); auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["eos_token"] = ""; tokenizer_config["eos_token_id"] = nlohmann::json::array({200020, 199999}); Write(path_ / "tokenizer_config.json", tokenizer_config); auto gguf = gguf_fixture::Builder().AddFullContractTensors(false).Write("frontend"); @@ -198,6 +199,9 @@ class Phi4FrontendTestAccess final { Phi4::engine_poisoned_for_testing_ = {}; } static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } + static const std::string& EosToken(const Phi4& model) { return model.eos_token; } + static const std::vector& EosTokenIds(const Phi4& model) { return model.eos_token_ids; } + static bool HasBosToken(const Phi4& model) { return model.has_bos_token; } }; } // namespace flm::phi4::testing @@ -333,7 +337,9 @@ void TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow() { void TestCancellationBeforePrefillSubmitsNothing() { TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); - TEST_REQUIRE(!model->insert(meta, input, [] { return true; })); + int checks = 0; + TEST_REQUIRE(!model->insert(meta, input, [&] { return ++checks >= 2; })); + TEST_REQUIRE(checks >= 2); TEST_REQUIRE(g_factory.engine->prefill_calls == 0); TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); } @@ -355,6 +361,43 @@ void TestCancellationReturnsOnlyAfterSynchronize() { TestCancellationBetweenDecodeStepsStopsWithCancelReason(); } +void TestNonStreamingChatGenerateWithPromptForwardsCancellation() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + std::ostringstream output; + int checks = 0; + AutoModel* endpoint_model = model.get(); + const auto response = endpoint_model->generate_with_prompt( + meta, input, 4096, output, [&] { return ++checks >= 2; }); + TEST_REQUIRE(response.empty()); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestLegacyTokenizerContractIsPreserved() { + TempPackage package; FactoryScope scope; auto legacy = Load(package, ModelInfo()); + TEST_REQUIRE(Phi4FrontendTestAccess::EosToken(*legacy) == ""); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*legacy) == + std::vector({200020, 199999})); + + auto aie4 = ReadyAie4(package); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*aie4) == + std::vector({200020, 199999})); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasBosToken(*aie4)); +} + +void TestSamePathBackendSwitchForcesLegacyInitialization() { + TempPackage package; FactoryScope scope; + Phi4 model(reinterpret_cast(1)); + model.load_model(package.path().string(), ModelInfo("corelib_aie4_gguf")); + TEST_REQUIRE(model.uses_corelib_aie4()); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(model)); + model.load_model(package.path().string(), ModelInfo()); + TEST_REQUIRE(!model.uses_corelib_aie4()); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(model)); + TEST_REQUIRE(g_factory.legacy_calls == 1); +} + void TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned() { TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; @@ -428,6 +471,9 @@ int main() { RunTest(TestCancellationBeforePrefillSubmitsNothing, "TestCancellationBeforePrefillSubmitsNothing"); RunTest(TestCancellationBetweenDecodeStepsStopsWithCancelReason, "TestCancellationBetweenDecodeStepsStopsWithCancelReason"); RunTest(TestCancellationReturnsOnlyAfterSynchronize, "TestCancellationReturnsOnlyAfterSynchronize"); + RunTest(TestNonStreamingChatGenerateWithPromptForwardsCancellation, "TestNonStreamingChatGenerateWithPromptForwardsCancellation"); + RunTest(TestLegacyTokenizerContractIsPreserved, "TestLegacyTokenizerContractIsPreserved"); + RunTest(TestSamePathBackendSwitchForcesLegacyInitialization, "TestSamePathBackendSwitchForcesLegacyInitialization"); RunTest(TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned, "TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned"); RunTest(TestPoisonedModelReturns500UntilReload, "TestPoisonedModelReturns500UntilReload"); RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); From 17d7525140324a7a1a5fb706001a5255418bc969 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 04:57:31 -0700 Subject: [PATCH 12/37] fix: preserve legacy Phi-4 tokenizer semantics --- src/common/AutoModel/modeling_phi4.cpp | 14 +++++--------- src/test/phi4_corelib_aie4/test_phi4_frontend.cpp | 4 +++- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index f17bd51d..0752f242 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -171,15 +171,11 @@ void Phi4::setup_tokenizer(const std::string& model_path, throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string chat_template"); const bool aie4 = verified_tokenizer_config != nullptr; - std::string configured_eos; - if (!aie4) { - if (!config.contains("eos_token") || !config["eos_token"].is_string()) - throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string eos_token"); - configured_eos = config["eos_token"].get(); - } + // Preserve the legacy Phi-4 contract: minja receives no textual BOS/EOS. + // AIE4 also disables automatic BOS, with stop IDs supplied only after the + // cross-source package contract has been validated. auto chat = std::make_unique( - config["chat_template"].get(), "", - aie4 ? "" : configured_eos); + config["chat_template"].get(), "", ""); std::vector eos; if (aie4) { // ValidatePhi4Contract proved these exact independent sources. @@ -194,7 +190,7 @@ void Phi4::setup_tokenizer(const std::string& model_path, } has_bos_token = false; bos_token_id = -1; - eos_token = std::move(configured_eos); + eos_token.clear(); eos_token_ids = std::move(eos); chat_tmpl = std::move(chat); user_system_prompt.clear(); diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index bd933e83..fac53efe 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -376,7 +376,9 @@ void TestNonStreamingChatGenerateWithPromptForwardsCancellation() { void TestLegacyTokenizerContractIsPreserved() { TempPackage package; FactoryScope scope; auto legacy = Load(package, ModelInfo()); - TEST_REQUIRE(Phi4FrontendTestAccess::EosToken(*legacy) == ""); + // Main's legacy Phi-4 frontend intentionally did not pass the textual EOS + // token into minja and retained an empty eos_token string. + TEST_REQUIRE(Phi4FrontendTestAccess::EosToken(*legacy).empty()); TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*legacy) == std::vector({200020, 199999})); From 1319e0754dda927dbf35e3f06f0f99283e178c55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 05:16:27 -0700 Subject: [PATCH 13/37] feat: pull Phi-4 GGUF and tokenizer from pinned sources --- src/model_info.json | 26 ++ src/model_list.json | 41 +++ src/pull/download_model.cpp | 152 ++++++++- src/pull/download_model.hpp | 17 + src/pull/model_downloader.cpp | 311 +++++++++++------- src/pull/model_downloader.hpp | 14 +- src/test/phi4_corelib_aie4/CMakeLists.txt | 31 ++ .../test_model_downloader.cpp | 255 ++++++++++++++ 8 files changed, 726 insertions(+), 121 deletions(-) create mode 100644 src/test/phi4_corelib_aie4/test_model_downloader.cpp diff --git a/src/model_info.json b/src/model_info.json index cb62db94..4252a6de 100644 --- a/src/model_info.json +++ b/src/model_info.json @@ -3300,5 +3300,31 @@ "xetHash": "7c4d2da22b3de2ed3f3eae66c7034386df6a3c5d81039ad4a1c8067e7eaf0069", "path": "vision_weights.q4nx" } + ], + "phi4-mini-it-aie4:4b": [ + { + "type": "file", + "path": "Phi-4-mini-instruct.Q8_0.gguf", + "size": 4084611040, + "sha256": "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42" + }, + { + "type": "file", + "path": "tokenizer.json", + "size": 15524095, + "sha256": "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea" + }, + { + "type": "file", + "path": "tokenizer_config.json", + "size": 2932, + "sha256": "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db" + }, + { + "type": "file", + "path": "config.json", + "size": 2504, + "sha256": "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b" + } ] } \ No newline at end of file diff --git a/src/model_list.json b/src/model_list.json index d2c7b656..8600df7e 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -469,6 +469,47 @@ "footprint": 3.4 } }, + "phi4-mini-it-aie4": { + "4b": { + "name": "phi4-mini-it-aie4", + "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "size": 4100140571, + "default_context_length": 4096, + "max_prefill_len": 4096, + "details": { + "family": "phi4", + "think": false, + "think_toggleable": false, + "parameter_size": "4B", + "quantization_level": "Q8_0 -> AIE4 group-64", + "execution_backend": "corelib_aie4_gguf" + }, + "flm_min_version": "1.0.3", + "vlm": false, + "files": [ + "Phi-4-mini-instruct.Q8_0.gguf", + "tokenizer.json", + "tokenizer_config.json", + "config.json" + ], + "file_sources": { + "tokenizer.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "tokenizer_config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + } + }, + "footprint": 4.1 + } + }, "embed-gemma": { "300m": { "name": "Embedding-Gemma-300M-NPU2", diff --git a/src/pull/download_model.cpp b/src/pull/download_model.cpp index 38ccf44a..923b8b94 100644 --- a/src/pull/download_model.cpp +++ b/src/pull/download_model.cpp @@ -15,6 +15,9 @@ #include "nlohmann/json.hpp" #include "picosha2.h" #include "sha1.hpp" +#ifdef _WIN32 +#include +#endif namespace download_utils { @@ -122,6 +125,129 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota return 0; } +namespace { + +FILE* open_part_file(const std::filesystem::path& path, bool append) { +#ifdef _WIN32 + return _wfopen(path.c_str(), append ? L"ab" : L"wb"); +#else + return fopen(path.c_str(), append ? "ab" : "wb"); +#endif +} + +bool promote_atomically(const std::filesystem::path& part, + const std::filesystem::path& destination) { +#ifdef _WIN32 + return MoveFileExW(part.c_str(), destination.c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) != 0; +#else + std::error_code error; + std::filesystem::rename(part, destination, error); + return !error; +#endif +} + +bool request_hash_matches(const DownloadRequest& request, + const std::filesystem::path& path) { + const std::string actual = request.hash_algorithm == HashAlgorithm::Sha256 + ? calculate_file_sha256(path.string()) + : calculate_git_blob_oid(path.string()); + return actual == request.expected_hash; +} + +} // namespace + +bool download_file_atomic(const DownloadRequest& request, + std::function progress_cb) { + if (request.expected_hash.empty()) { + std::cerr << "Missing expected hash for: " << request.destination << std::endl; + return false; + } + + std::error_code error; + std::filesystem::create_directories(request.destination.parent_path(), error); + if (error) { + std::cerr << "Failed to create download directory: " << error.message() << std::endl; + return false; + } + + const std::filesystem::path part(request.destination.string() + ".part"); + std::uint64_t offset = 0; + if (std::filesystem::exists(part, error)) { + offset = std::filesystem::file_size(part, error); + if (error) { + return false; + } + if (offset > request.expected_size) { + std::filesystem::remove(part, error); + if (error) { + return false; + } + offset = 0; + } + } + + if (offset < request.expected_size) { + CURL* curl = curl_easy_init(); + if (!curl) { + std::cerr << "Failed to initialize CURL" << std::endl; + return false; + } + FILE* fp = open_part_file(part, offset != 0); + if (!fp) { + curl_easy_cleanup(curl); + std::cerr << "Failed to open partial file for writing: " << part << std::endl; + return false; + } + + g_progress_bar_shown = false; + hide_cursor(); + curl_easy_setopt(curl, CURLOPT_URL, request.url.c_str()); + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data_to_file); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp); + curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); + curl_easy_setopt(curl, CURLOPT_USERAGENT, "FastFlowLM/1.0"); + curl_easy_setopt(curl, CURLOPT_TIMEOUT, 3600L); + if (offset != 0) { + curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, + static_cast(offset)); + } + if (progress_cb) { + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); + curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, progress_callback); + } + + const CURLcode result = curl_easy_perform(curl); + fclose(fp); + curl_easy_cleanup(curl); + show_cursor(); + if (g_progress_bar_shown) { + std::cout << std::endl; + } + if (result != CURLE_OK) { + std::cerr << "CURL error: " << curl_easy_strerror(result) << std::endl; + return false; // Keep the partial file for the next resume attempt. + } + } + + const std::uint64_t completed_size = std::filesystem::file_size(part, error); + if (error || completed_size != request.expected_size || + !request_hash_matches(request, part)) { + std::filesystem::remove(part, error); + header_print("FLM", "Downloaded file size or hash did not match."); + return false; + } + + if (!promote_atomically(part, request.destination)) { + std::cerr << "Failed to atomically promote: " << request.destination << std::endl; + return false; + } + header_print("FLM", "Download completed: " << request.destination.string()); + return true; +} + /// \brief Download a file from URL to a local file /// \param url the URL to download from /// \param local_path the local path to save the file @@ -218,6 +344,22 @@ static bool download_with_retry(const std::string& url, const std::string& local return false; } +static bool download_with_retry(const DownloadRequest& request, + std::function progress_cb, + int max_retries = 3) { + for (int attempt = 0; attempt < max_retries; ++attempt) { + if (download_file_atomic(request, progress_cb)) { + return true; + } + header_print("FLM", "Download failed (attempt " << (attempt + 1) << "/" << max_retries << ")"); + if (attempt + 1 < max_retries) { + header_print("FLM", "Retrying..."); + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + } + return false; +} + /// \brief Download content from URL to a string /// \param url the URL to download from /// \return the downloaded string @@ -268,6 +410,14 @@ bool download_multiple_files(const nlohmann::json downloads, std::string filename = std::filesystem::path(url).filename().string(); std::string remote_oid = file["oid"]; bool is_lfs = file["is_lfs"]; + DownloadRequest request{ + url, + local_path, + file["expected_size"].get(), + file.value("hash_algorithm", std::string()) == "sha256" + ? HashAlgorithm::Sha256 + : HashAlgorithm::GitBlobSha1, + remote_oid}; // cut "?download=true" if (filename.find("?download=true") != std::string::npos) { @@ -282,7 +432,7 @@ bool download_multiple_files(const nlohmann::json downloads, } }; - if (!download_with_retry(url, local_path, is_lfs, remote_oid, file_progress)) { + if (!download_with_retry(request, file_progress)) { std::cerr << "Failed to download: " << url << std::endl; //show_cursor(); // Show cursor on error return false; diff --git a/src/pull/download_model.hpp b/src/pull/download_model.hpp index 12e9a1a1..d160bf9d 100644 --- a/src/pull/download_model.hpp +++ b/src/pull/download_model.hpp @@ -7,6 +7,8 @@ #pragma once #include +#include +#include #include #include #include @@ -15,6 +17,16 @@ namespace download_utils { +enum class HashAlgorithm { Sha256, GitBlobSha1 }; + +struct DownloadRequest { + std::string url; + std::filesystem::path destination; + std::uint64_t expected_size; + HashAlgorithm hash_algorithm; + std::string expected_hash; +}; + std::string calculate_file_sha256(const std::string& file_path); std::string calculate_git_blob_oid(const std::string& file_path); @@ -35,6 +47,11 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota bool download_file(const std::string& url, const std::string& local_path, bool is_lfs, std::string remote_oid, std::function progress_cb = nullptr); +// Download to a same-directory temporary file, verify it, then atomically promote it. +bool download_file_atomic( + const DownloadRequest& request, + std::function progress_cb = nullptr); + // Download content from URL to a string std::string download_string(const std::string& url); diff --git a/src/pull/model_downloader.cpp b/src/pull/model_downloader.cpp index 24fc98d4..93dd0d8e 100644 --- a/src/pull/model_downloader.cpp +++ b/src/pull/model_downloader.cpp @@ -10,6 +10,134 @@ #include #include #include +#include +#include + +namespace { + +std::string percent_encode_filename(std::string_view filename) { + static constexpr char kHex[] = "0123456789ABCDEF"; + std::string encoded; + for (const unsigned char ch : filename) { + if ((ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') || + (ch >= '0' && ch <= '9') || ch == '-' || ch == '_' || ch == '.' || ch == '~') { + encoded.push_back(static_cast(ch)); + } else { + encoded.push_back('%'); + encoded.push_back(kHex[ch >> 4]); + encoded.push_back(kHex[ch & 0x0f]); + } + } + return encoded; +} + +bool is_hex_revision(const std::string& revision) { + return revision.size() == 40 && + std::all_of(revision.begin(), revision.end(), [](unsigned char ch) { + return std::isxdigit(ch) != 0; + }); +} + +nlohmann::json load_model_file_records(const std::string& model_tag) { + std::ifstream stream(utils::find_model_info()); + if (!stream.is_open()) { + throw std::runtime_error("model_info.json could not be opened"); + } + return nlohmann::json::parse(stream).at(model_tag); +} + +const nlohmann::json& find_file_record(const nlohmann::json& records, + const std::string& filename) { + const auto record = std::find_if(records.begin(), records.end(), [&](const auto& value) { + return value.at("path") == filename; + }); + if (record == records.end()) { + throw std::runtime_error("missing model_info record for " + filename); + } + return *record; +} + +struct ResolvedModelFile { + ModelFileSource source; + std::uint64_t size; + bool is_lfs; + download_utils::HashAlgorithm hash_algorithm; + std::string hash; +}; + +} // namespace + +ModelFileSource resolve_file_source(const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope) { + if (model_info.contains("file_sources")) { + if (use_modelscope) { + throw std::runtime_error("pinned Hugging Face per-file sources are required; --modelscope is not supported"); + } + const auto& sources = model_info.at("file_sources"); + if (!sources.is_object()) { + throw std::runtime_error("file_sources must be an object"); + } + std::unordered_set files; + for (const auto& file : model_info.at("files")) { + files.insert(file.get()); + } + for (const auto& [key, value] : sources.items()) { + if (!files.contains(key)) { + throw std::runtime_error("unknown file_sources key: " + key); + } + if (!value.is_object() || value.size() != 2 || + !value.contains("url") || !value.at("url").is_string() || + value.at("url").get().empty()) { + throw std::runtime_error("file source requires exactly a non-empty string url and revision"); + } + if (!value.contains("revision") || !value.at("revision").is_string() || + !is_hex_revision(value.at("revision").get())) { + throw std::runtime_error("file source revision must be a 40-character hexadecimal string"); + } + } + const auto override = sources.find(std::string(filename)); + if (override != sources.end()) { + const std::string base = override->at("url"); + const std::string revision = override->at("revision"); + return {base + "/resolve/" + revision + "/" + + percent_encode_filename(filename) + "?download=true", + revision}; + } + } + + const std::string base_url = use_modelscope + ? model_info.at("ms_url").get() + : model_info.at("url").get(); + if (base_url.find("resolve") != std::string::npos) { + return {base_url + "/" + std::string(filename) + "?download=true", {}}; + } + return {base_url + "/resolve/main/" + std::string(filename) + "?download=true", {}}; +} + +namespace { + +ResolvedModelFile resolve_model_file(const nlohmann::json& model_info, + const nlohmann::json& records, + const std::string& filename, + bool use_modelscope) { + const auto& record = find_file_record(records, filename); + const bool is_lfs = record.contains("lfs"); + const bool has_explicit_sha256 = record.contains("sha256"); + return { + resolve_file_source(model_info, filename, use_modelscope), + record.at("size").get(), + is_lfs, + has_explicit_sha256 || is_lfs + ? download_utils::HashAlgorithm::Sha256 + : download_utils::HashAlgorithm::GitBlobSha1, + has_explicit_sha256 + ? record.at("sha256").get() + : (is_lfs ? record.at("lfs").at("oid").get() + : record.at("oid").get())}; +} + +} // namespace /// \brief Constructor /// \param models the model list @@ -32,13 +160,14 @@ ModelDownloader::ModelStatus ModelDownloader::is_model_downloaded(const std::str if (modelstatus == ModelStatus::Outdated) { if (!fast_check) { header_print("FLM", "Checking outdated files..."); - verify_and_clean_files(model_tag, sub_process_mode); + verify_and_clean_files(model_tag, false, sub_process_mode); } } - else if (modelstatus == ModelStatus::Ready && !missing_files.empty()) { - // config.json is present and the version check passed, but other - // files (e.g. weights) are still missing. - modelstatus = ModelStatus::Missing; + else if (modelstatus == ModelStatus::Ready) { + if (!missing_files.empty() || + (!fast_check && !verify_and_clean_files(model_tag, false, sub_process_mode))) { + modelstatus = ModelStatus::Missing; + } } } return modelstatus; @@ -89,6 +218,10 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::string model_name = model_info["name"]; std::string model_server = use_modelscope ? "ModelScope" : "HuggingFace"; + if (use_modelscope && model_info.contains("file_sources")) { + // Validate this before any ready-state early return. + resolve_file_source(model_info, model_info.at("files").at(0).get(), true); + } header_print("FLM", "Pulling model from " + model_server + "..."); header_print("FLM", "Model: " + new_model_tag); @@ -101,9 +234,11 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco header_print("FLM", "Model already downloaded. Use --force to re-download."); return true; } - verify_and_clean_files(new_model_tag, use_modelscope); break; case ModelStatus::Missing: + // Preserve valid finals, but remove corrupt finals before deciding + // which files need to be downloaded. + verify_and_clean_files(new_model_tag, use_modelscope, true); break; case ModelStatus::Outdated: break; @@ -137,12 +272,12 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco } // Build download list - auto download_list = build_download_list(new_model_tag, use_modelscope); + auto download_list = build_download_list(new_model_tag, use_modelscope, force_redownload); auto downloads = download_list.first; float sum_fize_size = download_list.second; if (downloads.empty()) { header_print("FLM", "No files to download for model: " + new_model_tag); - return true; // Return true since all files are already present + return verify_and_clean_files(new_model_tag, use_modelscope); } header_print("FLM", "Downloading " + std::to_string(downloads.size()) + " missing files..."); @@ -162,17 +297,16 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco if (success) { header_print("FLM", "Model downloaded successfully!"); - // Verify download + // Verify every final file using the same pinned metadata used to download it. auto final_missing = get_missing_files(new_model_tag); - if (final_missing.empty()) { + const bool verified = final_missing.empty() && + verify_and_clean_files(new_model_tag, use_modelscope); + if (verified) { header_print("FLM", "All files verified successfully."); } else { - header_print("WARNING", "Some files may be missing after download:"); - for (const auto& file : final_missing) { - std::cout << " - " << file << std::endl; - } + header_print("WARNING", "Some files are missing or failed verification after download."); } - return true; + return verified; } else { header_print("ERROR", "Failed to download model files."); return false; @@ -285,82 +419,39 @@ std::string ModelDownloader::get_model_file_path(const std::string& model_path, /// \brief Build the download list /// \param model_tag the model tag /// \return the download list -std::pair ModelDownloader::build_download_list(const std::string& model_tag, bool modelscope) { - +std::pair ModelDownloader::build_download_list( + const std::string& model_tag, bool modelscope, bool force_redownload) { nlohmann::json downloads = nlohmann::json::array(); float sum_file_size = 0; - try { - auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); - std::string base_url = modelscope ? model_info["ms_url"] : model_info["url"]; - std::string model_name = model_info["name"]; - std::string file_url = model_info["file_url"]; - std::vector model_files = model_info["files"]; - - // Create model directory - std::string model_path = supported_models.get_model_path(new_model_tag); - std::filesystem::create_directories(model_path); - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } - - for (const auto& filename : model_files) { - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - - const auto& file = *it; - std::string local_path = get_model_file_path(model_path, filename); - - if (!file_exists(local_path)) { - std::string url; - if (std::string(base_url).find("resolve") != std::string::npos) { // resolve provided , may from a specific branch - url = base_url + "/" + filename + "?download=true"; - } - else { - url = base_url + "/resolve/main/" + filename + "?download=true"; - } - // header_print("URL", url); - bool is_lfs = file.contains("lfs"); - std::string oid = is_lfs ? file["lfs"]["oid"] : file["oid"]; - float file_size = static_cast(file["size"]) / 1024 / 1024; - sum_file_size += file_size; - - nlohmann::json entry = { - {"file", filename}, - {"size", file_size}, - {"url", url}, - {"localpath", local_path}, - {"oid", oid}, - {"is_lfs", is_lfs}, - }; - downloads.push_back(entry); - } - + auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const std::vector model_files = model_info.at("files"); + const std::string model_path = supported_models.get_model_path(new_model_tag); + std::filesystem::create_directories(model_path); + const nlohmann::json records = load_model_file_records(new_model_tag); + + for (const auto& filename : model_files) { + const std::string local_path = get_model_file_path(model_path, filename); + if (!force_redownload && file_exists(local_path)) { + continue; } - } - catch (const std::exception& e) { - header_print("ERROR", "Error building download list: " + std::string(e.what())); - } - return std::make_pair(downloads, sum_file_size); + const auto file = resolve_model_file(model_info, records, filename, modelscope); + const float file_size = static_cast(file.size) / 1024 / 1024; + sum_file_size += file_size; + downloads.push_back({ + {"file", filename}, + {"size", file_size}, + {"expected_size", file.size}, + {"url", file.source.url}, + {"localpath", local_path}, + {"oid", file.hash}, + {"is_lfs", file.is_lfs}, + {"hash_algorithm", file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? "sha256" : "git_blob_sha1"}, + }); + } + return {downloads, sum_file_size}; } /// \brief Remove a model and all its files @@ -428,11 +519,11 @@ bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelsc case ModelStatus::Missing: header_print("FLM", "Model not found: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to download it."); - return true; + return false; case ModelStatus::Incompatible: header_print("FLM", "Model is incompatible with this version of FastFlowLM: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to re-download it."); - return true; + return false; case ModelStatus::Outdated: case ModelStatus::Ready: { bool ok = verify_and_clean_files(new_model_tag, use_modelscope, sub_process_mode); @@ -440,10 +531,10 @@ bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelsc header_print("FLM", "Model check completed with errors. Use `flm pull " + new_model_tag + "` to re-download corrupted files."); else header_print("FLM", "Model check completed successfully. All files are present and compatible."); - return true; + return ok; } } - return true; + return false; } /// \brief Verify each model file's hash against HuggingFace metadata and @@ -457,37 +548,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::vector model_files = model_info["files"]; std::string model_path = supported_models.get_model_path(new_model_tag); - std::string file_url = model_info["file_url"]; - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (use_modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } + const nlohmann::json records = load_model_file_records(new_model_tag); for (const auto& filename : model_files) { if (!sub_process_mode) { header_print("FLM", "Checking file: " + filename + "..."); } - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - const auto& file = *it; + const auto file = resolve_model_file( + model_info, records, filename, use_modelscope); std::string local_path = get_model_file_path(model_path, filename); // If the file isn't present locally, there's nothing to verify or @@ -498,11 +567,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool continue; } - bool is_lfs = file.contains("lfs"); - std::string oid_ref = is_lfs ? file["lfs"]["oid"] : file["oid"]; - std::string local_oid = is_lfs ? download_utils::calculate_file_sha256(local_path) : download_utils::calculate_git_blob_oid(local_path); + const std::string local_oid = + file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? download_utils::calculate_file_sha256(local_path) + : download_utils::calculate_git_blob_oid(local_path); + std::error_code size_error; + const auto local_size = std::filesystem::file_size(local_path, size_error); + const bool size_matches = !size_error && local_size == file.size; - if (local_oid == oid_ref) { + if (size_matches && local_oid == file.hash) { if (!sub_process_mode) { header_print("FLM", "Success!"); } diff --git a/src/pull/model_downloader.hpp b/src/pull/model_downloader.hpp index 52a90f81..4c1bd1fb 100644 --- a/src/pull/model_downloader.hpp +++ b/src/pull/model_downloader.hpp @@ -14,6 +14,17 @@ #include #include #include +#include + +struct ModelFileSource { + std::string url; + std::string revision; +}; + +ModelFileSource resolve_file_source( + const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope); class ModelDownloader { public: @@ -59,7 +70,8 @@ class ModelDownloader { std::string get_model_file_path(const std::string& model_path, const std::string& filename); // Build download URLs for model files - std::pair build_download_list(const std::string& model_tag, bool modelscope=0); + std::pair build_download_list( + const std::string& model_tag, bool modelscope=0, bool force_redownload=false); // bool check_model_compatibility(const std::string& model_tag); ModelStatus check_model_compatibility(const std::string& model_tag, bool sub_process_mode=0); diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index f6d7bf54..ac1478df 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -16,6 +16,13 @@ find_path(XRT_INCLUDE_DIR NAMES xrt/xrt_bo.h HINTS "$ENV{XRT_INCLUDE_DIR}" "${CMAKE_CURRENT_LIST_DIR}/../../../../xrt_package/xrt/include" "C:/dev/XRT/src/runtime_src/core/include" REQUIRED) +find_path(CURL_INCLUDE_DIR NAMES curl/curl.h + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_library(CURL_LIBRARY NAMES libcurl curl + HINTS "$ENV{CONDA_PREFIX}/Library/lib" + "$ENV{USERPROFILE}/anaconda3/Library/lib" REQUIRED) +find_package(CURL REQUIRED) set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") set(CORELIB_SOURCES "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" @@ -138,6 +145,29 @@ target_compile_options(test_phi4_frontend PRIVATE target_link_directories(test_phi4_frontend PRIVATE "${XRT_INCLUDE_DIR}/../lib") target_link_libraries(test_phi4_frontend PRIVATE xrt_coreutil) +add_executable(test_model_downloader + test_model_downloader.cpp + "${FLM_SOURCE_DIR}/pull/download_model.cpp" + "${FLM_SOURCE_DIR}/pull/model_downloader.cpp" + "${FLM_SOURCE_DIR}/common/utils.cpp") +target_include_directories(test_model_downloader PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/pull" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_model_downloader PRIVATE + FLM_SOURCE_DIR="${FLM_SOURCE_DIR}" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins" + __FLM_VERSION__="1.0.3" + __NPU_VERSION__="0.0.0.0" + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_model_downloader PRIVATE + $<$:/wd4005 /wd4244>) +target_link_libraries(test_model_downloader PRIVATE CURL::libcurl) + add_executable(test_phi4_frontend_off test_phi4_frontend.cpp ${PHI4_FRONTEND_SOURCES}) @@ -221,4 +251,5 @@ add_test(NAME test_phi4_shape_plan COMMAND test_phi4_shape_plan) add_test(NAME test_phi4_engine COMMAND test_phi4_engine) add_test(NAME test_phi4_frontend COMMAND test_phi4_frontend) add_test(NAME test_phi4_frontend_off COMMAND test_phi4_frontend_off) +add_test(NAME test_model_downloader COMMAND test_model_downloader) set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/test_model_downloader.cpp b/src/test/phi4_corelib_aie4/test_model_downloader.cpp new file mode 100644 index 00000000..e2321338 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_model_downloader.cpp @@ -0,0 +1,255 @@ +#include "download_model.hpp" +#include "model_downloader.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include + +namespace { +namespace fs = std::filesystem; + +constexpr const char* kAie4Tag = "phi4-mini-it-aie4:4b"; +constexpr const char* kUnslothRevision = "78eb92a46fc37e6b524df991ed9aca9bc6aa7b80"; +constexpr const char* kMicrosoftRevision = "cfbefacb99257ffa30c83adab238a50856ac3083"; + +nlohmann::json ReadJson(const fs::path& path) { + std::ifstream stream(path); + TEST_REQUIRE(stream.is_open()); + return nlohmann::json::parse(stream); +} + +void Write(const fs::path& path, std::string_view bytes) { + fs::create_directories(path.parent_path()); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + stream.write(bytes.data(), static_cast(bytes.size())); + TEST_REQUIRE(stream.good()); +} + +std::string Read(const fs::path& path) { + std::ifstream stream(path, std::ios::binary); + return {std::istreambuf_iterator(stream), std::istreambuf_iterator()}; +} + +fs::path TempDirectory(std::string_view name) { + const auto path = fs::temp_directory_path() / ("flm-task5-" + std::string(name)); + std::error_code ignored; + fs::remove_all(path, ignored); + fs::create_directories(path); + return path; +} + +std::string FileUrl(const fs::path& path) { + std::string value = fs::absolute(path).generic_string(); +#ifdef _WIN32 + return "file:///" + value; +#else + return "file://" + value; +#endif +} + +void TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + const std::vector expected = { + "Phi-4-mini-instruct.Q8_0.gguf", "tokenizer.json", + "tokenizer_config.json", "config.json"}; + TEST_REQUIRE(model.at("name") == "phi4-mini-it-aie4"); + TEST_REQUIRE(model.at("files").get>() == expected); + TEST_REQUIRE(model.at("size").get() == 4100140571ULL); +} + +void TestGgufUrlContainsUnslothRevisionAndFilename() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + const auto source = resolve_file_source(model, "Phi-4-mini-instruct.Q8_0.gguf", false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/") + + kUnslothRevision + "/Phi-4-mini-instruct.Q8_0.gguf?download=true"); +} + +void TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + for (const std::string filename : {"tokenizer.json", "tokenizer_config.json", "config.json"}) { + const auto source = resolve_file_source(model, filename, false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/") + + kMicrosoftRevision + "/" + filename + "?download=true"); + } +} + +void TestExistingSingleSourceEntryKeepsItsCurrentUrl() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it").at("4b"); + const auto source = resolve_file_source(model, "config.json", false); + TEST_REQUIRE(source.url == + "https://huggingface.co/FastFlowLM/Phi4-mini-Instruct-NPU2/resolve/main/config.json?download=true"); +} + +void TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail() { + nlohmann::json model = { + {"url", "https://example.invalid/base"}, + {"files", {"config.json"}}, + {"file_sources", {{"unknown.json", {{"url", "https://example.invalid/source"}, + {"revision", std::string(40, 'a')}}}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), + "unknown file_sources key"); + + model["file_sources"] = {{"config.json", {{"revision", std::string(40, 'a')}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "url"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}, + {"revision", "NOT-A-COMMIT"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); +} + +void TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile() { + const auto all_info = ReadJson(FLM_SOURCE_DIR "/model_info.json"); + const auto& records = all_info.at(kAie4Tag); + const std::vector> expected = { + {"Phi-4-mini-instruct.Q8_0.gguf", 4084611040ULL, "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42"}, + {"tokenizer.json", 15524095ULL, "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea"}, + {"tokenizer_config.json", 2932ULL, "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db"}, + {"config.json", 2504ULL, "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b"}}; + TEST_REQUIRE(records.size() == expected.size()); + std::uint64_t total = 0; + for (const auto& [path, size, sha256] : expected) { + const auto match = std::find_if(records.begin(), records.end(), [&](const auto& record) { + return record.at("path") == path; + }); + TEST_REQUIRE(match != records.end()); + TEST_REQUIRE(match->at("size").get() == size); + TEST_REQUIRE(match->at("sha256") == sha256); + total += size; + } + TEST_REQUIRE(total == 4100140571ULL); +} + +struct DownloaderFixture { + fs::path root = TempDirectory("ready"); + fs::path catalog_path = root / "model_list.json"; + fs::path info_path = root / "model_info.json"; + std::string catalog_string; + std::string root_string; + model_list models; + + DownloaderFixture() + : catalog_string(catalog_path.string()), root_string(root.string()), models() { + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"test-model", {{"1b", { + {"name", "test-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "0.0.0"}, + {"files", {"config.json", "a.bin", "b.bin", "c.bin"}} + }}}}}}}; + const nlohmann::json info = {{"test-model:1b", { + {{"path", "config.json"}, {"size", 2}, {"sha256", "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"}}, + {{"path", "a.bin"}, {"size", 5}, {"sha256", "8ed3f6ad685b959ead7022518e1af76cd816f8e8ec7ccdda1ed4018e8f2223f8"}}, + {{"path", "b.bin"}, {"size", 4}, {"sha256", "f44e64e75f3948e9f73f8dfa94721c4ce8cbb4f265c4790c702b2d41cfbf2753"}}, + {{"path", "c.bin"}, {"size", 5}, {"sha256", "be9d587defa1f0c09ef49eb17e206983a5f8f8289e4281860bd0ee5a19592c67"}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + models = model_list(catalog_string, root_string); + } + + fs::path model_path() const { return root / "models" / "test-model"; } + void WriteValidFiles() const { + Write(model_path() / "config.json", "{}"); + Write(model_path() / "a.bin", "alpha"); + Write(model_path() / "b.bin", "beta"); + Write(model_path() / "c.bin", "gamma"); + } +}; + +void TestModelIsReadyOnlyWhenAllFourFinalFilesValidate() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Ready); + Write(fixture.model_path() / "b.bin", "BETA"); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +void TestPartFileNeverMakesModelReady() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + fs::rename(fixture.model_path() / "c.bin", fixture.model_path() / "c.bin.part"); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +download_utils::DownloadRequest Request(const fs::path& source, const fs::path& destination, + std::uint64_t size, std::string hash) { + return {FileUrl(source), destination, size, download_utils::HashAlgorithm::Sha256, std::move(hash)}; +} + +void TestResumeAppendsToPartThenAtomicallyPromotes() { + const auto root = TempDirectory("resume"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "abcdefgh"); + Write(destination.string() + ".part", "abcd"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestWrongSizeOrHashNeverReplacesAValidFinalFile() { + const auto root = TempDirectory("wrong"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(!download_utils::download_file_atomic( + Request(source, destination, 7, "9ac2197d9258257b1ae8463e4214e4cd0a578bc1517f2415928b91be4283fc48"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestInterruptedTransferKeepsPartForNextResume() { + const auto root = TempDirectory("interrupted"); + const auto destination = root / "destination.bin"; + Write(destination.string() + ".part", "abcd"); + auto request = Request(root / "missing.bin", destination, 8, + "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"); + TEST_REQUIRE(!download_utils::download_file_atomic(request)); + TEST_REQUIRE(Read(destination.string() + ".part") == "abcd"); + TEST_REQUIRE(!fs::exists(destination)); +} + +void TestSuccessfulForceDownloadAtomicallyReplacesFinalFile() { + const auto root = TempDirectory("replace"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9ac2197d9258257b1ae8463e4214e4cd0a578bc1517f2415928b91be4283fc48"))); + TEST_REQUIRE(Read(destination) == "ABCDEFGH"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} +} // namespace + +int main() { + RunTest(TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName, "AIE4 catalog"); + RunTest(TestGgufUrlContainsUnslothRevisionAndFilename, "GGUF URL"); + RunTest(TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename, "frontend URLs"); + RunTest(TestExistingSingleSourceEntryKeepsItsCurrentUrl, "legacy URL"); + RunTest(TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail, "source validation"); + RunTest(TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile, "model metadata"); + RunTest(TestModelIsReadyOnlyWhenAllFourFinalFilesValidate, "ready integrity"); + RunTest(TestPartFileNeverMakesModelReady, "part is not ready"); + RunTest(TestResumeAppendsToPartThenAtomicallyPromotes, "resume and promote"); + RunTest(TestWrongSizeOrHashNeverReplacesAValidFinalFile, "invalid transfer isolation"); + RunTest(TestInterruptedTransferKeepsPartForNextResume, "interrupted transfer"); + RunTest(TestSuccessfulForceDownloadAtomicallyReplacesFinalFile, "atomic replacement"); +} From 32f7698b792fe038b11fb256996a3b0d3de502cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 05:28:27 -0700 Subject: [PATCH 14/37] fix: preserve Phi-4 downloader compatibility semantics --- src/pull/model_downloader.cpp | 51 ++++++-- .../test_model_downloader.cpp | 118 +++++++++++++++++- 2 files changed, 154 insertions(+), 15 deletions(-) diff --git a/src/pull/model_downloader.cpp b/src/pull/model_downloader.cpp index 93dd0d8e..9b545480 100644 --- a/src/pull/model_downloader.cpp +++ b/src/pull/model_downloader.cpp @@ -65,6 +65,12 @@ struct ResolvedModelFile { std::string hash; }; +bool uses_pinned_aie4_integrity(const nlohmann::json& model_info) { + const auto details = model_info.find("details"); + return details != model_info.end() && details->is_object() && + details->value("execution_backend", std::string()) == "corelib_aie4_gguf"; +} + } // namespace ModelFileSource resolve_file_source(const nlohmann::json& model_info, @@ -150,22 +156,25 @@ ModelDownloader::ModelDownloader(model_list& models) /// \param model_tag the model tag /// \return true if the model is downloaded, false otherwise ModelDownloader::ModelStatus ModelDownloader::is_model_downloaded(const std::string& model_tag, bool sub_process_mode, bool fast_check) { - auto missing_files = get_missing_files(model_tag); + const auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const bool strict_integrity = uses_pinned_aie4_integrity(model_info); + auto missing_files = get_missing_files(new_model_tag); bool is_config_file_missing = std::find(missing_files.begin(), missing_files.end(), "config.json") != missing_files.end(); ModelStatus modelstatus = ModelStatus::Missing; if (!is_config_file_missing) { - modelstatus = check_model_compatibility(model_tag, sub_process_mode); + modelstatus = check_model_compatibility(new_model_tag, sub_process_mode); if (modelstatus == ModelStatus::Outdated) { if (!fast_check) { header_print("FLM", "Checking outdated files..."); - verify_and_clean_files(model_tag, false, sub_process_mode); + verify_and_clean_files(new_model_tag, false, sub_process_mode); } } else if (modelstatus == ModelStatus::Ready) { if (!missing_files.empty() || - (!fast_check && !verify_and_clean_files(model_tag, false, sub_process_mode))) { + (strict_integrity && !fast_check && + !verify_and_clean_files(new_model_tag, false, sub_process_mode))) { modelstatus = ModelStatus::Missing; } } @@ -180,8 +189,12 @@ ModelDownloader::ModelStatus ModelDownloader::check_model_compatibility(const st auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); LM_Config config; config.from_pretrained(this->supported_models.get_model_path(new_model_tag)); - std::string flm_version = config.flm_version; std::string flm_min_version = model_info["flm_min_version"]; + // The pinned Microsoft frontend config is upstream-native and intentionally + // has no FLM version. Its catalog contract supplies the compatibility floor. + std::string flm_version = uses_pinned_aie4_integrity(model_info) + ? flm_min_version + : config.flm_version; int l_l, m_l, r_l; //left, middle, right on local version int l_r, m_r, r_r; //left, middle, right on requried version int l_f, m_f, r_f; //left, middle, right on flm version @@ -236,9 +249,11 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco } break; case ModelStatus::Missing: - // Preserve valid finals, but remove corrupt finals before deciding - // which files need to be downloaded. - verify_and_clean_files(new_model_tag, use_modelscope, true); + if (uses_pinned_aie4_integrity(model_info)) { + // Preserve valid finals, but remove corrupt pinned finals before + // deciding which files need to be downloaded. + verify_and_clean_files(new_model_tag, use_modelscope, true); + } break; case ModelStatus::Outdated: break; @@ -277,7 +292,8 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco float sum_fize_size = download_list.second; if (downloads.empty()) { header_print("FLM", "No files to download for model: " + new_model_tag); - return verify_and_clean_files(new_model_tag, use_modelscope); + return !uses_pinned_aie4_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope); } header_print("FLM", "Downloading " + std::to_string(downloads.size()) + " missing files..."); @@ -300,7 +316,8 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco // Verify every final file using the same pinned metadata used to download it. auto final_missing = get_missing_files(new_model_tag); const bool verified = final_missing.empty() && - verify_and_clean_files(new_model_tag, use_modelscope); + (!uses_pinned_aie4_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope)); if (verified) { header_print("FLM", "All files verified successfully."); } else { @@ -512,9 +529,21 @@ bool ModelDownloader::remove_model(const std::string& model_tag, bool sub_proces /// \return true if all files are present and compatible, false otherwise bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelscope, bool sub_process_mode) { auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + if (use_modelscope && model_info.contains("file_sources")) { + try { + resolve_file_source( + model_info, model_info.at("files").at(0).get(), true); + } + catch (const std::exception& error) { + header_print("ERROR", error.what()); + return false; + } + } header_print("FLM", "Checking model: " + new_model_tag + "...\n"); - ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode); + // check_model owns the one full integrity pass below. Status discovery must + // remain presence/version-only so a pinned 4.1 GB model is not hashed twice. + ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode, true); switch (status) { case ModelStatus::Missing: header_print("FLM", "Model not found: " + new_model_tag); diff --git a/src/test/phi4_corelib_aie4/test_model_downloader.cpp b/src/test/phi4_corelib_aie4/test_model_downloader.cpp index e2321338..65273cdd 100644 --- a/src/test/phi4_corelib_aie4/test_model_downloader.cpp +++ b/src/test/phi4_corelib_aie4/test_model_downloader.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -33,6 +34,15 @@ std::string Read(const fs::path& path) { return {std::istreambuf_iterator(stream), std::istreambuf_iterator()}; } +std::size_t CountOccurrences(std::string_view text, std::string_view needle) { + std::size_t count = 0; + for (std::size_t position = text.find(needle); position != std::string_view::npos; + position = text.find(needle, position + needle.size())) { + ++count; + } + return count; +} + fs::path TempDirectory(std::string_view name) { const auto path = fs::temp_directory_path() / ("flm-task5-" + std::string(name)); std::error_code ignored; @@ -105,6 +115,30 @@ void TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail() { RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); } +void TestActualAie4CatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible() { + const auto root = TempDirectory("actual-catalog-version"); + const auto committed = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto model = committed.at("models").at("phi4-mini-it-aie4").at("4b"); + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"phi4-mini-it-aie4", {{"4b", model}}}}}}; + const auto catalog_path = root / "model_list.json"; + Write(catalog_path, catalog.dump()); + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / "phi4-mini-it-aie4"; + for (const auto& filename : model.at("files")) { + Write(model_path / filename.get(), "placeholder"); + } + Write(model_path / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded(kAie4Tag, true, true) == + ModelDownloader::ModelStatus::Ready); +} + void TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile() { const auto all_info = ReadJson(FLM_SOURCE_DIR "/model_info.json"); const auto& records = all_info.at(kAie4Tag); @@ -141,11 +175,13 @@ struct DownloaderFixture { {"model_path", "models"}, {"models", {{"test-model", {{"1b", { {"name", "test-model"}, {"url", "https://example.invalid/repo"}, - {"file_url", "https://example.invalid/api"}, {"flm_min_version", "0.0.0"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"details", {{"execution_backend", "corelib_aie4_gguf"}}}, + {"file_sources", nlohmann::json::object()}, {"files", {"config.json", "a.bin", "b.bin", "c.bin"}} }}}}}}}; const nlohmann::json info = {{"test-model:1b", { - {{"path", "config.json"}, {"size", 2}, {"sha256", "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"}}, + {{"path", "config.json"}, {"size", 57}, {"sha256", "b8bfba5e42c4cb0b8660ea39fec6fefafddc42fb6a0b17d472177fb7683b2290"}}, {{"path", "a.bin"}, {"size", 5}, {"sha256", "8ed3f6ad685b959ead7022518e1af76cd816f8e8ec7ccdda1ed4018e8f2223f8"}}, {{"path", "b.bin"}, {"size", 4}, {"sha256", "f44e64e75f3948e9f73f8dfa94721c4ce8cbb4f265c4790c702b2d41cfbf2753"}}, {{"path", "c.bin"}, {"size", 5}, {"sha256", "be9d587defa1f0c09ef49eb17e206983a5f8f8289e4281860bd0ee5a19592c67"}} @@ -162,7 +198,8 @@ struct DownloaderFixture { fs::path model_path() const { return root / "models" / "test-model"; } void WriteValidFiles() const { - Write(model_path() / "config.json", "{}"); + Write(model_path() / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); Write(model_path() / "a.bin", "alpha"); Write(model_path() / "b.bin", "beta"); Write(model_path() / "c.bin", "gamma"); @@ -186,6 +223,73 @@ void TestPartFileNeverMakesModelReady() { TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); } +void TestLegacyReadyCheckDoesNotHashOrDeleteWeights() { + const auto root = TempDirectory("legacy-ready"); + const auto catalog_path = root / "model_list.json"; + const auto info_path = root / "model_info.json"; + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"legacy-model", {{"1b", { + {"name", "legacy-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"files", {"config.json", "model.bin"}} + }}}}}}}; + const nlohmann::json info = {{"legacy-model:1b", { + {{"path", "config.json"}, {"size", 23}, {"oid", std::string(40, '0')}}, + {{"path", "model.bin"}, {"size", 8}, {"oid", std::string(40, '0')}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / "legacy-model"; + Write(model_path / "config.json", R"({"flm_version":"1.0.3"})"); + Write(model_path / "model.bin", "bad-data"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b") == + ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(Read(model_path / "model.bin") == "bad-data"); + + Write(model_path / "config.json", R"({"model_type":"legacy"})"); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b", true, true) == + ModelDownloader::ModelStatus::Outdated); +} + +void TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks() { + DownloaderFixture fixture; + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool pull_ok = downloader.pull_model("test-model:1b", true); + const bool check_ok = downloader.check_model("test-model:1b", true, true); + std::cout.rdbuf(previous); + + TEST_REQUIRE(!pull_ok); + TEST_REQUIRE(!check_ok); + TEST_REQUIRE(CountOccurrences(output.str(), + "pinned Hugging Face per-file sources are required") == 2); +} + +void TestCheckHashesPinnedFilesExactlyOnce() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(output.str(), "Checking file:") == 4); +} + download_utils::DownloadRequest Request(const fs::path& source, const fs::path& destination, std::uint64_t size, std::string hash) { return {FileUrl(source), destination, size, download_utils::HashAlgorithm::Sha256, std::move(hash)}; @@ -210,7 +314,7 @@ void TestWrongSizeOrHashNeverReplacesAValidFinalFile() { Write(source, "ABCDEFGH"); Write(destination, "abcdefgh"); TEST_REQUIRE(!download_utils::download_file_atomic( - Request(source, destination, 7, "9ac2197d9258257b1ae8463e4214e4cd0a578bc1517f2415928b91be4283fc48"))); + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); TEST_REQUIRE(Read(destination) == "abcdefgh"); TEST_REQUIRE(!fs::exists(destination.string() + ".part")); } @@ -245,9 +349,15 @@ int main() { RunTest(TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename, "frontend URLs"); RunTest(TestExistingSingleSourceEntryKeepsItsCurrentUrl, "legacy URL"); RunTest(TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail, "source validation"); + RunTest(TestActualAie4CatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible, + "AIE4 pinned config compatibility"); RunTest(TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile, "model metadata"); RunTest(TestModelIsReadyOnlyWhenAllFourFinalFilesValidate, "ready integrity"); RunTest(TestPartFileNeverMakesModelReady, "part is not ready"); + RunTest(TestLegacyReadyCheckDoesNotHashOrDeleteWeights, "legacy ready behavior"); + RunTest(TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks, + "modelscope rejection"); + RunTest(TestCheckHashesPinnedFilesExactlyOnce, "single check verification"); RunTest(TestResumeAppendsToPartThenAtomicallyPromotes, "resume and promote"); RunTest(TestWrongSizeOrHashNeverReplacesAValidFinalFile, "invalid transfer isolation"); RunTest(TestInterruptedTransferKeepsPartForNextResume, "interrupted transfer"); From dfdd1f4a5e49e4433f95b1099d75496a7cf121a8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 05:44:42 -0700 Subject: [PATCH 15/37] test: validate Phi-4 GGUF AIE4 integration --- src/test/phi4_corelib_aie4/fake_corelib.cpp | 20 ++++ .../phi4_corelib_aie4/test_phi4_engine.cpp | 62 +++++++++++ .../phi4_corelib_aie4/test_phi4_frontend.cpp | 103 ++++++++++++++++++ 3 files changed, 185 insertions(+) diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index 38490ed5..cf25c7da 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -2,8 +2,10 @@ #include #include +#include #include #include +#include #include #include #include @@ -343,6 +345,14 @@ struct TypedFake { std::is_same_v || std::is_same_v || std::is_same_v) { + if (state.statuses.contains("test_observe_dispatch_concurrency")) { + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + --state.active_leases; + } const auto status = Status(Tag::name); if (status != ryzenai_corelib_status_success) return status; fake_corelib::DispatchRecord record{}; @@ -366,6 +376,16 @@ struct TypedFake { if (record.output && static_cast(record.output)->kind == "window") record.window_offset = static_cast(record.output)->window_offset; state.dispatches.push_back(record); + if constexpr (std::is_same_v) { + auto* output = static_cast(record.output); + if (output && output->shape == std::vector({1, 200064})) { + EnsureStorage(*output); + const auto value = Bf16(1.0f); + std::memcpy(output->storage->bytes->data() + + output->window_offset * TypeBytes(output->data_type), + &value, sizeof(value)); + } + } state.work_in_flight = true; if (state.fail_after_submit == Tag::name) return ryzenai_corelib_status_failure; return ryzenai_corelib_status_success; diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index 475a6eec..932b184f 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #include namespace { @@ -121,6 +122,37 @@ void TestQkvAndGateUpPointersMatchExactMappedSubranges() { TEST_REQUIRE(records[5].pointers[1] == gate_up.values[1].bytes.data()); } +void TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates() { + Harness h; + const auto& records = fake_corelib::GetState().weight_creates; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto base = 1 + layer * 5; + const auto qkv = h.package->AttentionQkv(layer); + const auto gate_up = h.package->GateUp(layer); + TEST_REQUIRE(records[base + 0].pointers == + std::vector{qkv.values[0].bytes.data()}); + TEST_REQUIRE(records[base + 1].pointers == + std::vector{qkv.values[1].bytes.data()}); + TEST_REQUIRE(records[base + 2].pointers == + std::vector{qkv.values[2].bytes.data()}); + const std::vector output_pointer{ + h.package->RequireQ8("blk." + std::to_string(layer) + + ".attn_output.weight", std::array{3072, 3072}) + .bytes.data()}; + TEST_REQUIRE(records[base + 3].pointers == output_pointer); + const std::vector mlp_pointers{ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk." + std::to_string(layer) + + ".ffn_down.weight", std::array{3072, 8192}) + .bytes.data()}; + TEST_REQUIRE(records[base + 4].pointers == mlp_pointers); + } + const std::vector embedding_pointer{ + h.package->RequireQ8("token_embd.weight", + std::array{200064, 3072}).bytes.data()}; + TEST_REQUIRE(records.back().pointers == embedding_pointer); +} + void TestNormsAndEpsilonReachCorelibAsBf16() { Harness h; const auto expected = flm::phi4::ConvertF32ToBf16(std::array{1.0e-5f})[0]; @@ -401,6 +433,33 @@ void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { (void)RequireThrows([&] { (void)h.engine->forward(0); }); TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); } + +void TestTwoConcurrentAie4RequestsNeverOverlapDispatch() { + Harness h; + auto second_engine = std::make_unique( + LM_Config{}, h.package, h.runtime); + fake_corelib::GetState().maximum_active_leases = 0; + fake_corelib::GetState().statuses["test_observe_dispatch_concurrency"] = + ryzenai_corelib_status_success; + std::thread first([&] { (void)h.engine->forward(1); }); + std::thread second([&] { (void)second_engine->forward(2); }); + first.join(); + second.join(); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); +} + +void TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits() { + for (int cycle = 0; cycle < 10; ++cycle) { + { + Harness h; + const auto logits = h.engine->forward(cycle); + const auto* bits = reinterpret_cast(logits.data()); + TEST_REQUIRE(std::any_of(bits, bits + logits.size(), + [](std::uint16_t value) { return value != 0; })); + } + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + } +} } // namespace int main() { @@ -411,6 +470,7 @@ int main() { RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64Threads0); RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); + RUN_TEST(TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates); RUN_TEST(TestNormsAndEpsilonReachCorelibAsBf16); RUN_TEST(TestEmbeddingMappingOutlivesAllLazyRowReads); RUN_TEST(TestNoDeviceObjectExistsWhenPackageValidationFails); @@ -434,5 +494,7 @@ int main() { RUN_TEST(TestGetKCacheGathersHeadMajorPosition); RUN_TEST(TestGetVCacheGathersHeadMajorPosition); RUN_TEST(TestCancellationBoundaryLeavesNoOutstandingFakeWork); + RUN_TEST(TestTwoConcurrentAie4RequestsNeverOverlapDispatch); + RUN_TEST(TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits); #undef RUN_TEST } diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index fac53efe..a4bedb87 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -233,6 +233,35 @@ void TestAbsentBackendStillBuildsQ4nxPhi4Npu() { TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(*model)); } +void TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + g_samples = {7}; + g_sample_index = 0; + auto meta = Meta(); + auto input = Input(1); + std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 1, output); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +void TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + auto meta = Meta(); + auto input = Input(1); + TEST_REQUIRE(model->insert(meta, input)); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { TempPackage package; FactoryScope scope; @@ -243,6 +272,20 @@ void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); } +void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { + TempPackage package; + FactoryScope scope; + std::vector names; + for (const auto& entry : std::filesystem::directory_iterator(package.path())) + names.push_back(entry.path().filename().string()); + std::sort(names.begin(), names.end()); + TEST_REQUIRE(names == std::vector({ + "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "tokenizer.json", + "tokenizer_config.json"})); + auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + TEST_REQUIRE(model->uses_corelib_aie4()); +} + void TestUnknownAndNonStringBackendAreErrors() { TempPackage package; FactoryScope scope; @@ -277,6 +320,30 @@ void TestMissingCorelibFailsOnlyWhenAie4ModelLoads() { TEST_REQUIRE(g_factory.legacy_calls == 0); } +void TestAie4SelectionWithMissingDllFailsWithoutChangingBackend() { + TempPackage package; + FactoryScope scope; + Phi4 model(nullptr); + g_factory.throw_for_aie4 = true; + RequireContains(RequireThrows([&] { + model.load_model(package.path().string(), ModelInfo("corelib_aie4_gguf")); + }), "missing corelib"); + TEST_REQUIRE(!model.uses_corelib_aie4()); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + (void)RequireThrows([&] { + (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + }); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + void TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure() { TempPackage package; FactoryScope scope; @@ -456,15 +523,49 @@ void TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint() { } } +void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { + TempPackage package; + FactoryScope scope; + auto model = ReadyAie4(package); + int completions = 0; + auto input = Input(1); + { + auto meta = Meta(); + NPURequestCompletionGuard cancelled([&] { ++completions; }); + g_encoded_tokens = {1}; + TEST_REQUIRE(!model->insert(meta, input, [] { return true; })); + } + { + auto meta = Meta(); + NPURequestCompletionGuard capacity_error([&] { ++completions; }); + g_encoded_tokens.assign(4095, 1); + auto over_capacity = Input(1); + ExpectRequestError([&] { (void)model->insert(meta, over_capacity); }, + 400, false, "4095"); + } + { + auto meta = Meta(); + NPURequestCompletionGuard next_request([&] { ++completions; }); + g_encoded_tokens = {1}; + TEST_REQUIRE(model->insert(meta, input)); + } + TEST_REQUIRE(completions == 3); + TEST_REQUIRE(g_factory.engine->prefill_calls == 1); +} + } // namespace int main() { #if defined(FLM_ENABLE_CORELIB_AIE4) RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); + RunTest(TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing, "TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing"); RunTest(TestCorelibAie4GgufBuildsOnlyTheCorelibEngine, "TestCorelibAie4GgufBuildsOnlyTheCorelibEngine"); + RunTest(TestNoManifestOnnxConvertedWeightOrCachePathIsOpened, "TestNoManifestOnnxConvertedWeightOrCachePathIsOpened"); RunTest(TestUnknownAndNonStringBackendAreErrors, "TestUnknownAndNonStringBackendAreErrors"); RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); RunTest(TestMissingCorelibFailsOnlyWhenAie4ModelLoads, "TestMissingCorelibFailsOnlyWhenAie4ModelLoads"); + RunTest(TestAie4SelectionWithMissingDllFailsWithoutChangingBackend, "TestAie4SelectionWithMissingDllFailsWithoutChangingBackend"); + RunTest(TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback, "TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback"); RunTest(TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure, "TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure"); RunTest(TestPreemptionIsRejectedForTheAie4Route, "TestPreemptionIsRejectedForTheAie4Route"); RunTest(TestRenderedPromptPlusExplicitBudgetMayEqual4095, "TestRenderedPromptPlusExplicitBudgetMayEqual4095"); @@ -481,7 +582,9 @@ int main() { RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); + RunTest(TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable, "TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable"); #else + RunTest(TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib, "TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib"); RunTest(TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders, "TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders"); #endif std::cout << "test_phi4_frontend: PASS\n"; From 1eb82e402a52e3b9dec7c5092dd1bfd5921aa8f9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 06:01:05 -0700 Subject: [PATCH 16/37] test: strengthen Phi-4 AIE4 integration coverage --- src/common/AutoModel/modeling_phi4.cpp | 18 +++++ src/include/AutoModel/modeling_phi4.hpp | 7 +- src/server/server.cpp | 49 +++++--------- src/server/server.hpp | 52 ++++++++++++-- src/test/phi4_corelib_aie4/fake_corelib.cpp | 14 +++- src/test/phi4_corelib_aie4/fake_corelib.hpp | 2 + .../phi4_corelib_aie4/test_phi4_engine.cpp | 44 +++++++++++- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 67 +++++++++++++------ 8 files changed, 191 insertions(+), 62 deletions(-) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 0752f242..a296d383 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -48,6 +48,9 @@ std::uint32_t ResolveContext(const json& model_info, int requested) { } nlohmann::json ReadJson(const std::filesystem::path& path) { +#if defined(FLM_CORELIB_TESTING) + flm::phi4::testing::ObserveFileOpen(path); +#endif std::ifstream input(path, std::ios::binary); if (!input) throw std::runtime_error("Cannot open " + path.string()); try { @@ -68,6 +71,18 @@ void ConfigureSampler(Phi4& model) { } // namespace #if defined(FLM_CORELIB_TESTING) +namespace flm::phi4::testing { +namespace { +FileOpenObserver file_open_observer; +} +void SetFileOpenObserver(FileOpenObserver observer) { + file_open_observer = std::move(observer); +} +void ObserveFileOpen(const std::filesystem::path& path) { + if (file_open_observer) file_open_observer(path); +} +} // namespace flm::phi4::testing + Phi4::EngineFactoryForTesting Phi4::engine_factory_for_testing_; std::function Phi4::engine_poisoned_for_testing_; #endif @@ -117,6 +132,9 @@ void Phi4::load_model(std::string model_path, json model_info, const auto config = ReadJson(root / "config.json"); const auto tokenizer_json = ReadJson(root / "tokenizer.json"); const auto tokenizer_config = ReadJson(root / "tokenizer_config.json"); +#if defined(FLM_CORELIB_TESTING) + flm::phi4::testing::ObserveFileOpen(root / kAie4Gguf); +#endif auto package = flm::phi4::Phi4GgufPackage::Open(root / kAie4Gguf); package->ValidatePhi4Contract(config, tokenizer_json, tokenizer_config); diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index ce71f5d9..ed828fda 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -10,7 +10,12 @@ #if defined(FLM_CORELIB_TESTING) #include #include -namespace flm::phi4::testing { class Phi4FrontendTestAccess; } +namespace flm::phi4::testing { +class Phi4FrontendTestAccess; +using FileOpenObserver = std::function; +void SetFileOpenObserver(FileOpenObserver observer); +void ObserveFileOpen(const std::filesystem::path& path); +} #endif class Phi4 : public AutoModel { diff --git a/src/server/server.cpp b/src/server/server.cpp index cf2e4c2c..c24f78af 100644 --- a/src/server/server.cpp +++ b/src/server/server.cpp @@ -578,37 +578,25 @@ void WebServer::do_accept() { ///@brief process_next_npu_request Handles one queued NPU task at a time void WebServer::process_next_npu_request() { - { - std::lock_guard lock(npu_queue_mutex_); if (npu_request_queue_.empty()) { NPUAccessManager::release_npu_access(); return; // Queue is empty, NPU is free } - } // NPU cooldown before running the next queued task. constexpr auto npu_cooldown = std::chrono::milliseconds(333); std::this_thread::sleep_for(npu_cooldown); - std::function task; - size_t remaining = 0; - { - std::lock_guard lock(npu_queue_mutex_); - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; - } - - task = npu_request_queue_.front(); - npu_request_queue_.pop(); - remaining = npu_request_queue_.size(); + auto task = npu_request_queue_.take_next(); + if (!task) { + NPUAccessManager::release_npu_access(); + return; } - + const auto remaining = npu_request_queue_.size(); header_print("🟡 ", "Dequeuing NPU request (" + std::to_string(remaining) + " remaining)..."); - // Post the task to be executed by the io_context - net::post(ioc, task); - + // Post the task to be executed by the io_context. + net::post(ioc, std::move(task)); } ///@brief handle request @@ -800,28 +788,25 @@ bool WebServer::handle_request(http::request& req, return false; } - //const int NPU_QUEUE_LIMIT = 10; - std::lock_guard lock(npu_queue_mutex_); - - if (npu_request_queue_.size() >= max_npu_queue_) { + if (!npu_request_queue_.try_enqueue([this, process_task]() { + process_task(true); + })) { res.result(http::status::service_unavailable); res.body() = json{ - {"error", "NPU is in use and request queue is full (limit: " + std::to_string(max_npu_queue_) + "). Please try again later."} + {"error", "NPU is in use and request queue is full (limit: " + + std::to_string(npu_request_queue_.capacity()) + + "). Please try again later."} }.dump(); res.set(http::field::content_type, "application/json"); res.prepare_payload(); header_print("🚫 ", "NPU busy and queue full, request denied: " + key); return false; } - else { - // Create a new lambda to bind process_task(true) - npu_request_queue_.push([this, process_task]() { - process_task(true); - }); - header_print("🕒 ", "NPU busy, request queued (" + std::to_string(npu_request_queue_.size()) + "/" + std::to_string(max_npu_queue_) + "): " + key); - return true; - } + header_print("🕒 ", "NPU busy, request queued (" + + std::to_string(npu_request_queue_.size()) + "/" + + std::to_string(npu_request_queue_.capacity()) + "): " + key); + return true; } ///@brief create lm server diff --git a/src/server/server.hpp b/src/server/server.hpp index 1bfa7d86..d2037d93 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -55,6 +55,52 @@ inline bool requires_npu_access(const std::string& method, const std::string& pa path == "/v1/audio/transcriptions" || path == "/v1/embeddings"; } +class NPURequestQueue final { +public: + explicit NPURequestQueue(std::size_t capacity = 10) : capacity_(capacity) {} + void set_capacity(std::size_t capacity) { + std::lock_guard lock(mutex_); + capacity_ = capacity; + } + bool try_enqueue(std::function task) { + std::lock_guard lock(mutex_); + if (tasks_.size() >= capacity_) return false; + tasks_.push(std::move(task)); + return true; + } + std::function take_next() { + std::lock_guard lock(mutex_); + if (tasks_.empty()) return {}; + auto task = std::move(tasks_.front()); + tasks_.pop(); + return task; + } +#if defined(FLM_CORELIB_TESTING) + bool run_next() { + auto task = take_next(); + if (!task) return false; + task(); + return true; + } +#endif + bool empty() const { + std::lock_guard lock(mutex_); + return tasks_.empty(); + } + std::size_t size() const { + std::lock_guard lock(mutex_); + return tasks_.size(); + } + std::size_t capacity() const { + std::lock_guard lock(mutex_); + return capacity_; + } +private: + mutable std::mutex mutex_; + std::queue> tasks_; + std::size_t capacity_; +}; + class NPURequestCompletionGuard final { public: explicit NPURequestCompletionGuard(std::function completion) @@ -143,7 +189,7 @@ class WebServer { void set_max_connections(size_t max_conns) { max_connections_ = max_conns; } void set_request_timeout(std::chrono::seconds timeout) { request_timeout_ = timeout; } void set_io_threads(size_t num_threads) { io_thread_count_ = num_threads; } - void set_npu_queue_length(size_t q_len) { max_npu_queue_ = q_len; } + void set_npu_queue_length(size_t q_len) { npu_request_queue_.set_capacity(q_len); } // Maximum accepted HTTP request body size (in bytes) void set_max_body_size_bytes(std::size_t bytes) { max_body_size_bytes_ = bytes; } std::size_t get_max_body_size_bytes() const { return max_body_size_bytes_; } @@ -188,7 +234,6 @@ class WebServer { std::chrono::seconds request_timeout_ = std::chrono::seconds(600); // 5 minutes size_t io_thread_count_ = 5; std::size_t max_body_size_bytes_ = 256ull * 1024 * 1024; // 256 MB default - size_t max_npu_queue_ = 10; // Request tracking mutable std::mutex active_requests_mutex_; @@ -197,8 +242,7 @@ class WebServer { // Connection tracking std::atomic active_connections_{0}; std::vector io_threads_; - std::queue> npu_request_queue_; - std::mutex npu_queue_mutex_; + NPURequestQueue npu_request_queue_; // Friend declaration for HttpSession to access private members friend class HttpSession; }; diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index cf25c7da..a3cc5b19 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -12,6 +12,7 @@ namespace { fake_corelib::State state; +std::recursive_mutex state_mutex; thread_local std::string current_detail; struct FakeStorage { @@ -100,6 +101,7 @@ struct TypedFake; template struct TypedFake { static Result Invoke(Args... args) { + std::unique_lock state_lock(state_mutex); ++state.call_counts[std::string(Tag::name)]; state.call_log.emplace_back(Tag::name); auto arguments = std::forward_as_tuple(args...); @@ -350,12 +352,15 @@ struct TypedFake { int maximum = state.maximum_active_leases.load(); while (active > maximum && !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} + state_lock.unlock(); std::this_thread::sleep_for(std::chrono::milliseconds(1)); + state_lock.lock(); --state.active_leases; } const auto status = Status(Tag::name); if (status != ryzenai_corelib_status_success) return status; fake_corelib::DispatchRecord record{}; + record.thread_id = std::this_thread::get_id(); record.kind = std::is_same_v ? "matmul" : std::is_same_v ? "ssmlp" : std::is_same_v ? "rmsnorm" : "mha"; @@ -431,6 +436,7 @@ namespace fake_corelib { State& GetState() { return state; } void Reset() { + std::lock_guard lock(state_mutex); state.version = {0, 3, 0}; state.selftest_status = ryzenai_corelib_status_success; state.default_status = ryzenai_corelib_status_success; @@ -469,6 +475,7 @@ void Reset() { flm::corelib::CorelibApi::Resolver Resolver() { return [](std::string_view name) -> void* { + std::lock_guard lock(state_mutex); state.resolution_order.emplace_back(name); ++state.resolution_counts[std::string(name)]; if (name == state.missing_symbol) return nullptr; @@ -485,9 +492,13 @@ std::vector CallEveryResolvedFunction( return statuses; } -void* MakeObject() { return NewObject(); } +void* MakeObject() { + std::lock_guard lock(state_mutex); + return NewObject(); +} void EnterLease() { + std::lock_guard lock(state_mutex); const int active = ++state.active_leases; int maximum = state.maximum_active_leases.load(); while (active > maximum && @@ -495,6 +506,7 @@ void EnterLease() { } void LeaveLease() { + std::lock_guard lock(state_mutex); --state.active_leases; state.lifetime_events.emplace_back("lease_leave"); } diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp index e63bd91b..c46e2529 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.hpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -58,6 +59,7 @@ struct WeightCreateRecord { }; struct DispatchRecord { + std::thread::id thread_id; std::string kind; void* stream; void* input; diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index 932b184f..c300ab82 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -6,6 +6,8 @@ #include #include +#include +#include #include #include #include @@ -441,11 +443,49 @@ void TestTwoConcurrentAie4RequestsNeverOverlapDispatch() { fake_corelib::GetState().maximum_active_leases = 0; fake_corelib::GetState().statuses["test_observe_dispatch_concurrency"] = ryzenai_corelib_status_success; - std::thread first([&] { (void)h.engine->forward(1); }); - std::thread second([&] { (void)second_engine->forward(2); }); + fake_corelib::GetState().dispatches.clear(); + std::barrier start(3); + std::thread first([&] { start.arrive_and_wait(); (void)h.engine->forward(1); }); + std::thread second([&] { start.arrive_and_wait(); (void)second_engine->forward(2); }); + start.arrive_and_wait(); first.join(); second.join(); + + const auto& dispatches = fake_corelib::GetState().dispatches; TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + TEST_REQUIRE(dispatches.size() == 388); + const auto first_request = dispatches.front().thread_id; + TEST_REQUIRE(first_request != dispatches.back().thread_id); + TEST_REQUIRE(std::all_of(dispatches.begin(), dispatches.begin() + 194, + [&](const auto& call) { + return call.thread_id == first_request; + })); + TEST_REQUIRE(std::all_of(dispatches.begin() + 194, dispatches.end(), + [&](const auto& call) { + return call.thread_id != first_request; + })); + + // Prove the fake itself does not serialize or race when the runtime lease is + // intentionally bypassed: the overlap detector must report both calls. + fake_corelib::GetState().dispatches.clear(); + fake_corelib::GetState().maximum_active_leases = 0; + std::barrier unsafe_start(3); + std::atomic unsafe_calls_succeeded{true}; + const auto invoke_without_lease = [&] { + unsafe_start.arrive_and_wait(); + if (h.runtime->api()->functions().rmsnorm( + nullptr, nullptr, 1, nullptr, nullptr) != + ryzenai_corelib_status_success) + unsafe_calls_succeeded = false; + }; + std::thread unsafe_first(invoke_without_lease); + std::thread unsafe_second(invoke_without_lease); + unsafe_start.arrive_and_wait(); + unsafe_first.join(); + unsafe_second.join(); + TEST_REQUIRE(unsafe_calls_succeeded); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 2); + TEST_REQUIRE(fake_corelib::GetState().dispatches.size() == 2); } void TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits() { diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index a4bedb87..2dfdeb1e 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -21,6 +21,7 @@ namespace { std::vector g_encoded_tokens; std::vector g_samples; +std::vector g_opened_paths; std::size_t g_sample_index{}; class FakeEngine final : public causal_lm { @@ -177,6 +178,10 @@ class Phi4FrontendTestAccess final { public: static void InstallFactory() { g_factory = {}; + g_opened_paths.clear(); + flm::phi4::testing::SetFileOpenObserver([](const auto& path) { + g_opened_paths.push_back(path); + }); Phi4::engine_factory_for_testing_ = [](bool aie4, const LM_Config&, npu_xclbin_manager*, const std::filesystem::path&, std::uint32_t limit) { @@ -197,6 +202,7 @@ class Phi4FrontendTestAccess final { static void RemoveFactory() { Phi4::engine_factory_for_testing_ = {}; Phi4::engine_poisoned_for_testing_ = {}; + flm::phi4::testing::SetFileOpenObserver({}); } static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } static const std::string& EosToken(const Phi4& model) { return model.eos_token; } @@ -275,15 +281,21 @@ void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { TempPackage package; FactoryScope scope; + auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + TEST_REQUIRE(model->uses_corelib_aie4()); std::vector names; - for (const auto& entry : std::filesystem::directory_iterator(package.path())) - names.push_back(entry.path().filename().string()); + for (const auto& path : g_opened_paths) { + const auto text = path.generic_string(); + TEST_REQUIRE(text.find("manifest") == std::string::npos); + TEST_REQUIRE(text.find("onnx") == std::string::npos); + TEST_REQUIRE(text.find("converted") == std::string::npos); + TEST_REQUIRE(text.find("cache") == std::string::npos); + names.push_back(path.filename().string()); + } std::sort(names.begin(), names.end()); TEST_REQUIRE(names == std::vector({ "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "tokenizer.json", "tokenizer_config.json"})); - auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); - TEST_REQUIRE(model->uses_corelib_aie4()); } void TestUnknownAndNonStringBackendAreErrors() { @@ -527,30 +539,41 @@ void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); - int completions = 0; - auto input = Input(1); - { + NPURequestQueue queue(2); + bool cancelled = false; + bool capacity_failed = false; + bool queued_request_ran = false; + + TEST_REQUIRE(queue.try_enqueue([&] { auto meta = Meta(); - NPURequestCompletionGuard cancelled([&] { ++completions; }); + auto input = Input(1); g_encoded_tokens = {1}; - TEST_REQUIRE(!model->insert(meta, input, [] { return true; })); - } - { + cancelled = !model->insert(meta, input, [] { return true; }); + })); + TEST_REQUIRE(queue.try_enqueue([&] { auto meta = Meta(); - NPURequestCompletionGuard capacity_error([&] { ++completions; }); + auto input = Input(1); g_encoded_tokens.assign(4095, 1); - auto over_capacity = Input(1); - ExpectRequestError([&] { (void)model->insert(meta, over_capacity); }, - 400, false, "4095"); - } - { + try { (void)model->insert(meta, input); } + catch (const ModelRequestError& error) { + capacity_failed = error.http_code() == 400; + } + })); + TEST_REQUIRE(!queue.try_enqueue([&] { queued_request_ran = true; })); + + TEST_REQUIRE(queue.run_next()); + TEST_REQUIRE(cancelled); + TEST_REQUIRE(queue.try_enqueue([&] { auto meta = Meta(); - NPURequestCompletionGuard next_request([&] { ++completions; }); + auto input = Input(1); g_encoded_tokens = {1}; - TEST_REQUIRE(model->insert(meta, input)); - } - TEST_REQUIRE(completions == 3); - TEST_REQUIRE(g_factory.engine->prefill_calls == 1); + queued_request_ran = model->insert(meta, input); + })); + TEST_REQUIRE(queue.run_next()); + TEST_REQUIRE(capacity_failed); + TEST_REQUIRE(queue.run_next()); + TEST_REQUIRE(queued_request_ran); + TEST_REQUIRE(queue.empty()); } } // namespace From 14f6161c19fd30d89021babdcb00b55add3a5899 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 06:34:28 -0700 Subject: [PATCH 17/37] test: exercise production Phi-4 integration seams --- src/common/AutoModel/modeling_phi4.cpp | 20 +------ src/common/corelib/phi4_corelib_gguf.cpp | 2 + src/common/tokenizer/tokenizer.cpp | 3 + src/include/AutoModel/modeling_phi4.hpp | 7 +-- src/include/lm_config.hpp | 5 +- src/include/utils/file_access.hpp | 27 +++++++++ src/server/server.cpp | 36 +++++------- src/server/server.hpp | 33 ++++++----- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 56 +++++++++++++------ 9 files changed, 112 insertions(+), 77 deletions(-) create mode 100644 src/include/utils/file_access.hpp diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index a296d383..79d4dfca 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -1,6 +1,7 @@ /// \file modeling_phi4.cpp /// \brief Phi-4 frontend and backend routing #include "AutoModel/modeling_phi4.hpp" +#include "utils/file_access.hpp" #if defined(FLM_ENABLE_CORELIB_AIE4) #include "models/phi4/phi4_corelib_aie4.hpp" @@ -48,9 +49,7 @@ std::uint32_t ResolveContext(const json& model_info, int requested) { } nlohmann::json ReadJson(const std::filesystem::path& path) { -#if defined(FLM_CORELIB_TESTING) - flm::phi4::testing::ObserveFileOpen(path); -#endif + flm::file_access::ObserveOpen(path); std::ifstream input(path, std::ios::binary); if (!input) throw std::runtime_error("Cannot open " + path.string()); try { @@ -71,18 +70,6 @@ void ConfigureSampler(Phi4& model) { } // namespace #if defined(FLM_CORELIB_TESTING) -namespace flm::phi4::testing { -namespace { -FileOpenObserver file_open_observer; -} -void SetFileOpenObserver(FileOpenObserver observer) { - file_open_observer = std::move(observer); -} -void ObserveFileOpen(const std::filesystem::path& path) { - if (file_open_observer) file_open_observer(path); -} -} // namespace flm::phi4::testing - Phi4::EngineFactoryForTesting Phi4::engine_factory_for_testing_; std::function Phi4::engine_poisoned_for_testing_; #endif @@ -132,9 +119,6 @@ void Phi4::load_model(std::string model_path, json model_info, const auto config = ReadJson(root / "config.json"); const auto tokenizer_json = ReadJson(root / "tokenizer.json"); const auto tokenizer_config = ReadJson(root / "tokenizer_config.json"); -#if defined(FLM_CORELIB_TESTING) - flm::phi4::testing::ObserveFileOpen(root / kAie4Gguf); -#endif auto package = flm::phi4::Phi4GgufPackage::Open(root / kAie4Gguf); package->ValidatePhi4Contract(config, tokenizer_json, tokenizer_config); diff --git a/src/common/corelib/phi4_corelib_gguf.cpp b/src/common/corelib/phi4_corelib_gguf.cpp index 1433d2cf..975e61e1 100644 --- a/src/common/corelib/phi4_corelib_gguf.cpp +++ b/src/common/corelib/phi4_corelib_gguf.cpp @@ -1,6 +1,7 @@ #include "models/phi4/phi4_corelib_gguf.hpp" #include "models/phi4/phi4_corelib_constants.hpp" +#include "utils/file_access.hpp" #define NOMINMAX #include @@ -331,6 +332,7 @@ Phi4GgufPackage::~Phi4GgufPackage() = default; std::shared_ptr Phi4GgufPackage::Open( const std::filesystem::path& gguf_path) { auto impl = std::make_unique(); + flm::file_access::ObserveOpen(gguf_path); impl->file = CreateFileW(gguf_path.c_str(), GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); if (impl->file == INVALID_HANDLE_VALUE) diff --git a/src/common/tokenizer/tokenizer.cpp b/src/common/tokenizer/tokenizer.cpp index 2b21981c..f048fc85 100644 --- a/src/common/tokenizer/tokenizer.cpp +++ b/src/common/tokenizer/tokenizer.cpp @@ -4,6 +4,7 @@ /// \date 2025-06-24 /// \version 0.9.10 #include "tokenizer/tokenizer.hpp" +#include "utils/file_access.hpp" #include #include #include @@ -15,6 +16,8 @@ /// \brief Constructor /// \param model_path the model path Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); #ifdef _WIN32 std::ifstream fs(model_path + "\\tokenizer.json", std::ios::in | std::ios::binary); #else diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index ed828fda..ce71f5d9 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -10,12 +10,7 @@ #if defined(FLM_CORELIB_TESTING) #include #include -namespace flm::phi4::testing { -class Phi4FrontendTestAccess; -using FileOpenObserver = std::function; -void SetFileOpenObserver(FileOpenObserver observer); -void ObserveFileOpen(const std::filesystem::path& path); -} +namespace flm::phi4::testing { class Phi4FrontendTestAccess; } #endif class Phi4 : public AutoModel { diff --git a/src/include/lm_config.hpp b/src/include/lm_config.hpp index 8eb4d0c0..201fd422 100644 --- a/src/include/lm_config.hpp +++ b/src/include/lm_config.hpp @@ -8,6 +8,7 @@ #include "typedef.hpp" #include "utils/utils.hpp" +#include "utils/file_access.hpp" #include "nlohmann/json.hpp" #include @@ -99,7 +100,9 @@ class LM_Config{ /// \brief read model_path/config.json into _json_config void _load_json(){ - std::ifstream file(this->model_path + "/config.json"); + const auto config_path = std::filesystem::path(this->model_path) / "config.json"; + flm::file_access::ObserveOpen(config_path); + std::ifstream file(config_path); if (!file.is_open()){ std::cerr << "Failed to open file: " << this->model_path << std::endl; exit(1); diff --git a/src/include/utils/file_access.hpp b/src/include/utils/file_access.hpp new file mode 100644 index 00000000..21fc74c6 --- /dev/null +++ b/src/include/utils/file_access.hpp @@ -0,0 +1,27 @@ +#pragma once + +#include + +#if defined(FLM_CORELIB_TESTING) +#include +#include +#endif + +namespace flm::file_access { + +#if defined(FLM_CORELIB_TESTING) +using OpenObserver = std::function; +inline OpenObserver open_observer; + +inline void SetOpenObserver(OpenObserver observer) { + open_observer = std::move(observer); +} + +inline void ObserveOpen(const std::filesystem::path& path) { + if (open_observer) open_observer(path); +} +#else +inline void ObserveOpen(const std::filesystem::path&) {} +#endif + +} // namespace flm::file_access diff --git a/src/server/server.cpp b/src/server/server.cpp index c24f78af..04f39df1 100644 --- a/src/server/server.cpp +++ b/src/server/server.cpp @@ -578,25 +578,15 @@ void WebServer::do_accept() { ///@brief process_next_npu_request Handles one queued NPU task at a time void WebServer::process_next_npu_request() { - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; // Queue is empty, NPU is free - } - - // NPU cooldown before running the next queued task. - constexpr auto npu_cooldown = std::chrono::milliseconds(333); - std::this_thread::sleep_for(npu_cooldown); - - auto task = npu_request_queue_.take_next(); - if (!task) { - NPUAccessManager::release_npu_access(); - return; - } - const auto remaining = npu_request_queue_.size(); - header_print("🟡 ", "Dequeuing NPU request (" + std::to_string(remaining) + " remaining)..."); - - // Post the task to be executed by the io_context. - net::post(ioc, std::move(task)); + npu_request_coordinator_.complete_current( + [this](NPURequestCoordinator::Task task) { + const auto remaining = npu_request_coordinator_.size(); + header_print("🟡 ", "Dequeuing NPU request (" + + std::to_string(remaining) + " remaining)..."); + net::post(ioc, std::move(task)); + }, + [] { NPUAccessManager::release_npu_access(); }, + std::chrono::milliseconds(333)); } ///@brief handle request @@ -788,13 +778,13 @@ bool WebServer::handle_request(http::request& req, return false; } - if (!npu_request_queue_.try_enqueue([this, process_task]() { + if (!npu_request_coordinator_.try_enqueue([this, process_task]() { process_task(true); })) { res.result(http::status::service_unavailable); res.body() = json{ {"error", "NPU is in use and request queue is full (limit: " + - std::to_string(npu_request_queue_.capacity()) + + std::to_string(npu_request_coordinator_.capacity()) + "). Please try again later."} }.dump(); res.set(http::field::content_type, "application/json"); @@ -804,8 +794,8 @@ bool WebServer::handle_request(http::request& req, } header_print("🕒 ", "NPU busy, request queued (" + - std::to_string(npu_request_queue_.size()) + "/" + - std::to_string(npu_request_queue_.capacity()) + "): " + key); + std::to_string(npu_request_coordinator_.size()) + "/" + + std::to_string(npu_request_coordinator_.capacity()) + "): " + key); return true; } diff --git a/src/server/server.hpp b/src/server/server.hpp index d2037d93..23822fe5 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -55,34 +55,41 @@ inline bool requires_npu_access(const std::string& method, const std::string& pa path == "/v1/audio/transcriptions" || path == "/v1/embeddings"; } -class NPURequestQueue final { +class NPURequestCoordinator final { public: - explicit NPURequestQueue(std::size_t capacity = 10) : capacity_(capacity) {} + using Task = std::function; + using Scheduler = std::function; + + explicit NPURequestCoordinator(std::size_t capacity = 10) + : capacity_(capacity) {} void set_capacity(std::size_t capacity) { std::lock_guard lock(mutex_); capacity_ = capacity; } - bool try_enqueue(std::function task) { + bool try_enqueue(Task task) { std::lock_guard lock(mutex_); if (tasks_.size() >= capacity_) return false; tasks_.push(std::move(task)); return true; } - std::function take_next() { + Task take_next() { std::lock_guard lock(mutex_); if (tasks_.empty()) return {}; auto task = std::move(tasks_.front()); tasks_.pop(); return task; } -#if defined(FLM_CORELIB_TESTING) - bool run_next() { + void complete_current(const Scheduler& schedule, + const std::function& release, + std::chrono::milliseconds cooldown) { + if (cooldown.count() > 0) std::this_thread::sleep_for(cooldown); auto task = take_next(); - if (!task) return false; - task(); - return true; + if (!task) { + release(); + return; + } + schedule(std::move(task)); } -#endif bool empty() const { std::lock_guard lock(mutex_); return tasks_.empty(); @@ -97,7 +104,7 @@ class NPURequestQueue final { } private: mutable std::mutex mutex_; - std::queue> tasks_; + std::queue tasks_; std::size_t capacity_; }; @@ -189,7 +196,7 @@ class WebServer { void set_max_connections(size_t max_conns) { max_connections_ = max_conns; } void set_request_timeout(std::chrono::seconds timeout) { request_timeout_ = timeout; } void set_io_threads(size_t num_threads) { io_thread_count_ = num_threads; } - void set_npu_queue_length(size_t q_len) { npu_request_queue_.set_capacity(q_len); } + void set_npu_queue_length(size_t q_len) { npu_request_coordinator_.set_capacity(q_len); } // Maximum accepted HTTP request body size (in bytes) void set_max_body_size_bytes(std::size_t bytes) { max_body_size_bytes_ = bytes; } std::size_t get_max_body_size_bytes() const { return max_body_size_bytes_; } @@ -242,7 +249,7 @@ class WebServer { // Connection tracking std::atomic active_connections_{0}; std::vector io_threads_; - NPURequestQueue npu_request_queue_; + NPURequestCoordinator npu_request_coordinator_; // Friend declaration for HttpSession to access private members friend class HttpSession; }; diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index 2dfdeb1e..67324c70 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -1,5 +1,6 @@ #include "test_support.hpp" #include "gguf_fixture.hpp" +#include "utils/file_access.hpp" #include #include "server.hpp" @@ -145,7 +146,11 @@ void ExpectRequestError(F&& action, int code, bool cleared, std::string_view tex } // namespace -Tokenizer::Tokenizer(const std::string&) { is_doubled_encoded = false; } +Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); + is_doubled_encoded = false; +} Tokenizer::~Tokenizer() = default; std::vector Tokenizer::encode(const std::string&) { return g_encoded_tokens; } std::string Tokenizer::decode(const std::vector&) { return "decoded"; } @@ -179,7 +184,7 @@ class Phi4FrontendTestAccess final { static void InstallFactory() { g_factory = {}; g_opened_paths.clear(); - flm::phi4::testing::SetFileOpenObserver([](const auto& path) { + flm::file_access::SetOpenObserver([](const auto& path) { g_opened_paths.push_back(path); }); Phi4::engine_factory_for_testing_ = @@ -202,7 +207,7 @@ class Phi4FrontendTestAccess final { static void RemoveFactory() { Phi4::engine_factory_for_testing_ = {}; Phi4::engine_poisoned_for_testing_ = {}; - flm::phi4::testing::SetFileOpenObserver({}); + flm::file_access::SetOpenObserver({}); } static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } static const std::string& EosToken(const Phi4& model) { return model.eos_token; } @@ -294,8 +299,8 @@ void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { } std::sort(names.begin(), names.end()); TEST_REQUIRE(names == std::vector({ - "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "tokenizer.json", - "tokenizer_config.json"})); + "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "config.json", + "tokenizer.json", "tokenizer.json", "tokenizer_config.json"})); } void TestUnknownAndNonStringBackendAreErrors() { @@ -539,18 +544,20 @@ void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); - NPURequestQueue queue(2); + NPURequestCoordinator coordinator(3); bool cancelled = false; bool capacity_failed = false; bool queued_request_ran = false; + int completion_callbacks = 0; + int accelerator_releases = 0; - TEST_REQUIRE(queue.try_enqueue([&] { + TEST_REQUIRE(coordinator.try_enqueue([&] { auto meta = Meta(); auto input = Input(1); g_encoded_tokens = {1}; cancelled = !model->insert(meta, input, [] { return true; }); })); - TEST_REQUIRE(queue.try_enqueue([&] { + TEST_REQUIRE(coordinator.try_enqueue([&] { auto meta = Meta(); auto input = Input(1); g_encoded_tokens.assign(4095, 1); @@ -559,21 +566,38 @@ void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { capacity_failed = error.http_code() == 400; } })); - TEST_REQUIRE(!queue.try_enqueue([&] { queued_request_ran = true; })); - - TEST_REQUIRE(queue.run_next()); - TEST_REQUIRE(cancelled); - TEST_REQUIRE(queue.try_enqueue([&] { + TEST_REQUIRE(coordinator.try_enqueue([&] { auto meta = Meta(); auto input = Input(1); g_encoded_tokens = {1}; queued_request_ran = model->insert(meta, input); })); - TEST_REQUIRE(queue.run_next()); + TEST_REQUIRE(!coordinator.try_enqueue([] {})); + + std::function)> execute; + const auto complete = [&] { + ++completion_callbacks; + coordinator.complete_current(execute, [&] { ++accelerator_releases; }, + std::chrono::milliseconds(0)); + }; + execute = [&](std::function task) { + NPURequestCompletionGuard completion(complete); + task(); + completion.complete(); + completion.complete(); + }; + { + NPURequestCompletionGuard active_request_completion(complete); + active_request_completion.complete(); + active_request_completion.complete(); + } + + TEST_REQUIRE(cancelled); TEST_REQUIRE(capacity_failed); - TEST_REQUIRE(queue.run_next()); TEST_REQUIRE(queued_request_ran); - TEST_REQUIRE(queue.empty()); + TEST_REQUIRE(coordinator.empty()); + TEST_REQUIRE(completion_callbacks == 4); + TEST_REQUIRE(accelerator_releases == 1); } } // namespace From a6ce0956a1b8d1f0fd98fda0d80fb6538d393da3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 07:05:29 -0700 Subject: [PATCH 18/37] fix: release idle NPU queue immediately --- src/server/server.hpp | 2 +- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 30 +++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/src/server/server.hpp b/src/server/server.hpp index 23822fe5..de1f5198 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -82,12 +82,12 @@ class NPURequestCoordinator final { void complete_current(const Scheduler& schedule, const std::function& release, std::chrono::milliseconds cooldown) { - if (cooldown.count() > 0) std::this_thread::sleep_for(cooldown); auto task = take_next(); if (!task) { release(); return; } + if (cooldown.count() > 0) std::this_thread::sleep_for(cooldown); schedule(std::move(task)); } bool empty() const { diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index 67324c70..d7fd07bb 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -540,6 +540,35 @@ void TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint() { } } +void TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff() { + constexpr auto cooldown = std::chrono::milliseconds(100); + + NPURequestCoordinator empty; + bool released = false; + const auto empty_start = std::chrono::steady_clock::now(); + empty.complete_current([](auto) { TEST_REQUIRE(false); }, + [&] { released = true; }, cooldown); + const auto empty_elapsed = std::chrono::steady_clock::now() - empty_start; + TEST_REQUIRE(released); + TEST_REQUIRE(empty_elapsed < std::chrono::milliseconds(50)); + + NPURequestCoordinator queued; + bool handed_off = false; + bool released_while_queued = false; + TEST_REQUIRE(queued.try_enqueue([] {})); + const auto queued_start = std::chrono::steady_clock::now(); + queued.complete_current( + [&](auto task) { + handed_off = true; + task(); + }, + [&] { released_while_queued = true; }, cooldown); + const auto queued_elapsed = std::chrono::steady_clock::now() - queued_start; + TEST_REQUIRE(handed_off); + TEST_REQUIRE(!released_while_queued); + TEST_REQUIRE(queued_elapsed >= std::chrono::milliseconds(75)); +} + void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { TempPackage package; FactoryScope scope; @@ -629,6 +658,7 @@ int main() { RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); + RunTest(TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff, "TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff"); RunTest(TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable, "TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable"); #else RunTest(TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib, "TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib"); From 867df751d15ace8904fc691ddec584710e803e83 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:08:24 -0700 Subject: [PATCH 19/37] fixup! build: add optional dynamic corelib 0.3.0 runtime --- src/common/corelib/corelib_runtime.cpp | 37 +++++++++++++++++-- src/include/corelib/corelib_runtime.hpp | 7 ++++ src/test/phi4_corelib_aie4/CMakeLists.txt | 3 +- .../phi4_corelib_aie4/test_corelib_api.cpp | 17 +++++++++ 4 files changed, 59 insertions(+), 5 deletions(-) diff --git a/src/common/corelib/corelib_runtime.cpp b/src/common/corelib/corelib_runtime.cpp index 48c08ad2..d8a6073e 100644 --- a/src/common/corelib/corelib_runtime.cpp +++ b/src/common/corelib/corelib_runtime.cpp @@ -7,11 +7,21 @@ namespace flm::corelib { namespace { std::mutex process_mutex; std::shared_ptr process_runtime; +#if defined(FLM_CORELIB_TESTING) +std::function destruction_observer; +bool shutdown_execution_lock_held = false; +#endif } CorelibRuntime::CorelibRuntime(std::shared_ptr api) : api_(std::move(api)) {} +CorelibRuntime::~CorelibRuntime() { +#if defined(FLM_CORELIB_TESTING) + if (destruction_observer) destruction_observer(shutdown_execution_lock_held); +#endif +} + std::shared_ptr CorelibRuntime::CreateReady( std::shared_ptr api) { if (!api) throw std::invalid_argument("corelib API is null"); @@ -48,14 +58,33 @@ void CorelibRuntime::ShutdownProcess() { std::lock_guard process_lock(process_mutex); if (!process_runtime) return; - std::lock_guard execution_lock(process_runtime->execution_mutex_); - if (process_runtime->api_->live_object_count() != 0) { + auto runtime = process_runtime; + std::unique_lock execution_lock(runtime->execution_mutex_); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = true; +#endif + if (runtime->api_->live_object_count() != 0) { +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif throw std::runtime_error("cannot shut down with live corelib objects"); } - process_runtime->api_->functions().cleanup(); - process_runtime->api_.reset(); + runtime->api_->functions().cleanup(); + runtime->api_.reset(); process_runtime.reset(); + execution_lock.unlock(); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif + runtime.reset(); +} + +#if defined(FLM_CORELIB_TESTING) +void CorelibRuntime::SetDestructionObserverForTest( + std::function observer) { + destruction_observer = std::move(observer); } +#endif std::unique_lock CorelibRuntime::AcquireExecution() { return std::unique_lock(execution_mutex_); diff --git a/src/include/corelib/corelib_runtime.hpp b/src/include/corelib/corelib_runtime.hpp index e22ce1b4..0027f8be 100644 --- a/src/include/corelib/corelib_runtime.hpp +++ b/src/include/corelib/corelib_runtime.hpp @@ -5,16 +5,23 @@ #include #include #include +#if defined(FLM_CORELIB_TESTING) +#include +#endif namespace flm::corelib { class CorelibRuntime final { public: + ~CorelibRuntime(); static std::shared_ptr GetOrCreate( const std::filesystem::path& executable_dir); static std::shared_ptr CreateForTest( std::shared_ptr api); static void ShutdownProcess(); +#if defined(FLM_CORELIB_TESTING) + static void SetDestructionObserverForTest(std::function observer); +#endif std::unique_lock AcquireExecution(); const std::shared_ptr& api() const noexcept; diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index ac1478df..802de6cb 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -58,7 +58,8 @@ target_include_directories(test_corelib_api PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") -target_compile_definitions(test_corelib_api PRIVATE RYZENAI_CORELIB_STATIC=1) +target_compile_definitions(test_corelib_api PRIVATE + RYZENAI_CORELIB_STATIC=1 FLM_CORELIB_TESTING=1) add_executable(test_real_corelib test_real_corelib.cpp ${CORELIB_SOURCES}) target_include_directories(test_real_corelib PRIVATE diff --git a/src/test/phi4_corelib_aie4/test_corelib_api.cpp b/src/test/phi4_corelib_aie4/test_corelib_api.cpp index d3cd9bb2..20967264 100644 --- a/src/test/phi4_corelib_aie4/test_corelib_api.cpp +++ b/src/test/phi4_corelib_aie4/test_corelib_api.cpp @@ -229,6 +229,22 @@ void TestExecutionLeaseSerializesTwoThreads() { CorelibRuntime::ShutdownProcess(); } +void TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner() { + fake_corelib::Reset(); + bool destroyed = false; + bool destroyed_while_locked = false; + CorelibRuntime::SetDestructionObserverForTest([&](bool execution_lock_held) { + destroyed = true; + destroyed_while_locked = execution_lock_held; + }); + auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + CorelibRuntime::SetDestructionObserverForTest({}); + TEST_REQUIRE(destroyed); + TEST_REQUIRE(!destroyed_while_locked); +} + void TestCleanupRunsAfterTheLastObjectAndOnlyOnce() { fake_corelib::Reset(); const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); @@ -273,6 +289,7 @@ int main() { RUN_TEST(TestEveryUniqueObjectReleasesExactlyOnceAfterMoves); RUN_TEST(TestRuntimeRunsDependencySelftestAndRequiresDeviceContext); RUN_TEST(TestExecutionLeaseSerializesTwoThreads); + RUN_TEST(TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner); RUN_TEST(TestCleanupRunsAfterTheLastObjectAndOnlyOnce); #undef RUN_TEST return 0; From b0a13ecee9290d5e2fbc5bf3eaf4a0da52d19229 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:08:37 -0700 Subject: [PATCH 20/37] fixup! feat: add validated Phi-4 Q8_0 GGUF reader --- src/common/corelib/phi4_corelib_gguf.cpp | 37 +++++---- .../models/phi4/phi4_corelib_constants.hpp | 1 + src/test/phi4_corelib_aie4/gguf_fixture.hpp | 26 +++++-- src/test/phi4_corelib_aie4/test_phi4_gguf.cpp | 76 ++++++++++++++++--- 4 files changed, 107 insertions(+), 33 deletions(-) diff --git a/src/common/corelib/phi4_corelib_gguf.cpp b/src/common/corelib/phi4_corelib_gguf.cpp index 975e61e1..1b907d35 100644 --- a/src/common/corelib/phi4_corelib_gguf.cpp +++ b/src/common/corelib/phi4_corelib_gguf.cpp @@ -376,7 +376,10 @@ std::shared_ptr Phi4GgufPackage::Open( } } - const auto alignment = impl->Unsigned("general.alignment"); + constexpr std::uint64_t kDefaultAlignment = 32; + const auto alignment = impl->metadata.contains("general.alignment") + ? impl->Unsigned("general.alignment") + : kDefaultAlignment; if (alignment == 0 || (alignment & (alignment - 1)) != 0) Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); @@ -527,7 +530,7 @@ void Phi4GgufPackage::ValidatePhi4Contract( }; if (metadata.architecture != "phi3") Fail("general.architecture", metadata.architecture, "phi3"); require_unsigned("phi3.block_count", metadata.layer_count, kLayerCount); - require_unsigned("phi3.context_length", metadata.context_length, kMaxSequenceLength); + require_unsigned("phi3.context_length", metadata.context_length, kModelContextLength); require_unsigned("phi3.embedding_length", metadata.hidden_size, kHiddenSize); require_unsigned("phi3.feed_forward_length", metadata.intermediate_size, kIntermediateSize); require_unsigned("phi3.attention.head_count", metadata.attention_head_count, kQueryHeadCount); @@ -548,7 +551,8 @@ void Phi4GgufPackage::ValidatePhi4Contract( RequireQ8("token_embd.weight", std::array{kVocabularySize, kHiddenSize}); RequireF32("output_norm.weight", std::array{kHiddenSize}); if (impl_->tensors.contains("output.weight")) Fail("output.weight", "present", "absent (tied token_embd.weight)"); - if (impl_->tensors.contains("rope_factors_long.weight")) Fail("rope_factors_long.weight", "present", "absent for original 4096 window"); + if (impl_->tensors.contains("rope_factors_long.weight")) + RequireF32("rope_factors_long.weight", std::array{48}); for (std::size_t layer = 0; layer < static_cast(kLayerCount); ++layer) { const auto prefix = "blk." + std::to_string(layer); RequireF32(prefix + ".attn_norm.weight", std::array{kHiddenSize}); @@ -567,7 +571,8 @@ void Phi4GgufPackage::ValidatePhi4Contract( RequireJsonUnsigned(config, "intermediate_size", kIntermediateSize); RequireJsonUnsigned(config, "num_attention_heads", kQueryHeadCount); RequireJsonUnsigned(config, "num_key_value_heads", kKvHeadCount); - RequireJsonUnsigned(config, "head_dim", kHeadSize); + if (config.contains("head_dim")) + RequireJsonUnsigned(config, "head_dim", kHeadSize); RequireJsonUnsigned(config, "vocab_size", kVocabularySize); RequireJsonDouble(config, "rms_norm_eps", 1.0e-5); RequireJsonUnsigned(config, "original_max_position_embeddings", kMaxSequenceLength); @@ -612,14 +617,6 @@ void Phi4GgufPackage::ValidatePhi4Contract( } catch (const nlohmann::json::exception& error) { Fail("tokenizer.json vocabulary", error.what(), "valid token-to-ID mappings"); } - const auto actual_count = vocabulary_ids.size(); - const auto actual_max = vocabulary_ids.empty() ? -1 : *vocabulary_ids.rbegin(); - if (actual_max != kVocabularySize - 1) - Fail("tokenizer.json maximum vocabulary ID", std::to_string(actual_max), - std::to_string(kVocabularySize - 1)); - if (actual_count != static_cast(kVocabularySize)) - Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), - std::to_string(kVocabularySize)); for (const auto& [token, expected] : std::array{ std::pair{"<|end|>", 200020}, std::pair{"<|endoftext|>", 199999}}) { @@ -627,6 +624,16 @@ void Phi4GgufPackage::ValidatePhi4Contract( if (it == token_ids.end()) Fail(token, "missing", std::to_string(expected)); if (it->second != expected) Fail(token, std::to_string(it->second), std::to_string(expected)); } + constexpr std::int64_t kTokenizerMaximumAssignedId = 200028; + constexpr std::size_t kTokenizerDistinctAssignedIds = 200029; + const auto actual_count = vocabulary_ids.size(); + const auto actual_max = vocabulary_ids.empty() ? -1 : *vocabulary_ids.rbegin(); + if (actual_max != kTokenizerMaximumAssignedId) + Fail("tokenizer.json maximum vocabulary ID", std::to_string(actual_max), + std::to_string(kTokenizerMaximumAssignedId)); + if (actual_count != kTokenizerDistinctAssignedIds) + Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), + std::to_string(kTokenizerDistinctAssignedIds)); const auto gguf_eos = impl_->Unsigned("tokenizer.ggml.eos_token_id"); if (gguf_eos != 200020) Fail("tokenizer.ggml.eos_token_id", std::to_string(gguf_eos), "200020"); @@ -635,7 +642,11 @@ void Phi4GgufPackage::ValidatePhi4Contract( if (template_it == tokenizer_config.end() || !template_it->is_string()) Fail("chat_template", template_it == tokenizer_config.end() ? "missing" : JsonText(*template_it), "string containing Phi-4 markers"); const auto chat_template = template_it->get(); - for (const auto marker : {"<|user|>", "<|end|>", "<|assistant|>"}) + const bool has_dynamic_role = + chat_template.find("'<|' + message['role'] + '|>'") != std::string::npos; + if (chat_template.find("<|user|>") == std::string::npos && !has_dynamic_role) + Fail("<|user|>", "missing from chat_template", "present in chat_template"); + for (const auto marker : {"<|end|>", "<|assistant|>"}) if (chat_template.find(marker) == std::string::npos) Fail(marker, "missing from chat_template", "present in chat_template"); } diff --git a/src/include/models/phi4/phi4_corelib_constants.hpp b/src/include/models/phi4/phi4_corelib_constants.hpp index fc1a80c0..92ed9a44 100644 --- a/src/include/models/phi4/phi4_corelib_constants.hpp +++ b/src/include/models/phi4/phi4_corelib_constants.hpp @@ -14,6 +14,7 @@ inline constexpr std::int64_t kKvDimension = 1024; inline constexpr std::int64_t kVocabularySize = 200064; inline constexpr std::int64_t kRopeDimension = 96; inline constexpr std::int64_t kMaxSequenceLength = 4096; +inline constexpr std::int64_t kModelContextLength = 131072; inline constexpr std::int64_t kMaxDecodeWindow = 4095; inline constexpr std::uint32_t kRequantizedGroupSize = 64; inline constexpr float kRmsEpsilon = 1.0e-5f; diff --git a/src/test/phi4_corelib_aie4/gguf_fixture.hpp b/src/test/phi4_corelib_aie4/gguf_fixture.hpp index ec460214..ac70bed1 100644 --- a/src/test/phi4_corelib_aie4/gguf_fixture.hpp +++ b/src/test/phi4_corelib_aie4/gguf_fixture.hpp @@ -239,7 +239,7 @@ class Builder { AddMetadata("general.architecture", std::string("phi3")); AddMetadata("general.alignment", std::uint32_t{32}); AddMetadata("phi3.block_count", std::uint32_t{32}); - AddMetadata("phi3.context_length", std::uint32_t{4096}); + AddMetadata("phi3.context_length", std::uint32_t{131072}); AddMetadata("phi3.embedding_length", std::uint32_t{3072}); AddMetadata("phi3.feed_forward_length", std::uint32_t{8192}); AddMetadata("phi3.attention.head_count", std::uint32_t{24}); @@ -281,10 +281,12 @@ class Builder { auto metadata = metadata_; auto tensors = tensors_; std::uint32_t alignment = alignment_; - if (mutation_ == Mutation::ZeroAlignment) alignment = 0; - if (mutation_ == Mutation::NonPowerOfTwoAlignment) alignment = 24; - for (auto& entry : metadata) - if (entry.first == "general.alignment") entry.second = alignment; + if (mutation_ == Mutation::ZeroAlignment || + mutation_ == Mutation::NonPowerOfTwoAlignment) { + alignment = mutation_ == Mutation::ZeroAlignment ? 0 : 24; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = alignment; + } if (mutation_ == Mutation::DuplicateName && !tensors.empty()) tensors.push_back(tensors.front()); if (mutation_ == Mutation::DtypeMismatch && !tensors.empty()) tensors.front().type = kF32; if (mutation_ == Mutation::ShapeMismatch && !tensors.empty()) tensors.front().logical_shape[0]--; @@ -386,14 +388,22 @@ inline nlohmann::json ValidConfig() { inline nlohmann::json ValidTokenizer() { nlohmann::json vocab = nlohmann::json::object(); - for (int id = 0; id < 200062; ++id) vocab["t" + std::to_string(id)] = id; + for (int id = 0; id < 200019; ++id) vocab["t" + std::to_string(id)] = id; vocab["<|endoftext|>"] = 199999; vocab["<|end|>"] = 200020; return {{"model", {{"vocab", std::move(vocab)}}}, {"added_tokens", nlohmann::json::array({ - {{"id", 200062}, {"content", "added-a"}}, - {{"id", 200063}, {"content", "added-b"}}, + {{"id", 200019}, {"content", "<|assistant|>"}}, {{"id", 200020}, {"content", "<|end|>"}}, + {{"id", 200021}, {"content", "<|user|>"}}, + {{"id", 200022}, {"content", "<|system|>"}}, + {{"id", 200023}, {"content", "<|tool|>"}}, + {{"id", 200024}, {"content", "<|/tool|>"}}, + {{"id", 200025}, {"content", "<|tool_call|>"}}, + {{"id", 200026}, {"content", "<|/tool_call|>"}}, + {{"id", 200027}, {"content", "<|tool_response|>"}}, + {{"id", 200028}, {"content", "<|tag|>"}}, + {{"id", 200018}, {"content", "<|endofprompt|>"}}, {{"id", 199999}, {"content", "<|endoftext|>"}}})}}; } diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp index 0edc25a0..81130666 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -89,6 +89,14 @@ void TestValidV3HeaderMetadataDirectoryAndAlignment() { TEST_REQUIRE(!metadata.add_bos_token); } +void TestOmittedAlignmentUsesGgufDefault32() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().RemoveMetadata("general.alignment"), + file, "default-alignment"); + TEST_REQUIRE(package->RequireF32( + "f32", std::array{48}).values.size() == 48); +} + void TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely() { gguf_fixture::TempFile file; auto package = Open(SplitFixture().AddEverySkippableMetadataType(), file, @@ -119,9 +127,14 @@ void TestCountProductAlignmentAndOffsetOverflowFail() { RequireContains(OpenFailure(SplitFixture(), Mutation::OffsetOverflow, "offset-overflow"), "overflow"); } -void TestZeroAndNonPowerOfTwoAlignmentFail() { +void TestPresentMalformedAlignmentFails() { RequireContains(OpenFailure(SplitFixture(), Mutation::ZeroAlignment, "zero-align"), "alignment"); RequireContains(OpenFailure(SplitFixture(), Mutation::NonPowerOfTwoAlignment, "bad-align"), "alignment"); + + auto wrong_type = SplitFixture().SetMetadata( + "general.alignment", std::int32_t{-32}).Write("wrong-align-type"); + RequireMismatch(RequireThrows([&] { Phi4GgufPackage::Open(wrong_type.path); }), + "general.alignment", "INT32", "unsigned integer metadata"); } void TestDuplicateTensorNamesFail() { @@ -226,7 +239,7 @@ void TestRejectsWrongArchitectureAndEveryDimension() { const std::vector cases = { {"general.architecture", std::string("llama"), "llama", "phi3"}, {"phi3.block_count", std::uint32_t{31}, "31", "32"}, - {"phi3.context_length", std::uint32_t{4095}, "4095", "4096"}, + {"phi3.context_length", std::uint32_t{131071}, "131071", "131072"}, {"phi3.embedding_length", std::uint32_t{3071}, "3071", "3072"}, {"phi3.feed_forward_length", std::uint32_t{8191}, "8191", "8192"}, {"phi3.attention.head_count", std::uint32_t{23}, "23", "24"}, @@ -308,15 +321,26 @@ void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { "token_embd.weight", "missing", "present tensor"); } -void TestRequiresOriginal4096WindowAndRejectsLongRopeBranch() { +void TestRequiresOriginal4096WindowAndValidatesLongRopeFactors() { auto wrong_file = Builder().SetMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{8192}).AddFullContractTensors().Write("long-window"); auto wrong = Phi4GgufPackage::Open(wrong_file.path); RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), "phi3.rope.scaling.original_context_length", "8192", "4096"); - auto long_file = Builder().AddFullContractTensors().AddTensor("rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); - auto long_rope = Phi4GgufPackage::Open(long_file.path); - RequireMismatch(RequireThrows([&] { long_rope->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), - "rope_factors_long.weight", "present", "absent"); + + auto valid_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); + auto valid = Phi4GgufPackage::Open(valid_file.path); + valid->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); + + auto malformed_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {47}, gguf_fixture::kF32).Write("bad-long-rope"); + auto malformed = Phi4GgufPackage::Open(malformed_file.path); + RequireMismatch(RequireThrows([&] { malformed->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_long.weight", "[47]", "[48]"); } void TestValidatesOptionalShortRopeFactorsAsF32Length48() { @@ -341,6 +365,15 @@ void TestRejectsNonFiniteOrNonPositiveRopeValues() { } } +void TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads() { + ContractFixture fixture; + auto config = gguf_fixture::ValidConfig(); + config.erase("head_dim"); + fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); +} + void TestRejectsConfigDisagreement() { ContractFixture fixture; struct Case { @@ -406,6 +439,17 @@ void TestDerivesStopSetFromGgufConfigAndTokenizerIds() { } } +void TestAcceptsPinnedDynamicRoleChatTemplate() { + ContractFixture fixture; + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = + "{% for message in messages %}{{ '<|' + message['role'] + '|>' + " + "message['content'] + '<|end|>' }}{% endfor %}" + "{% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %}"; + fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); +} + void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { ContractFixture fixture; @@ -414,7 +458,7 @@ void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( gguf_fixture::ValidConfig(), tokenizer, gguf_fixture::ValidTokenizerConfig()); }), - "tokenizer.json distinct vocabulary ID count", "200063", "200064"); + "tokenizer.json distinct vocabulary ID count", "200028", "200029"); for (const auto& [invalid_id, actual, expected] : std::array{ std::tuple{-1, "-1", "0..200063"}, @@ -431,11 +475,16 @@ void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { } tokenizer = gguf_fixture::ValidTokenizer(); - tokenizer["added_tokens"].erase(tokenizer["added_tokens"].begin() + 1); + auto& added = tokenizer["added_tokens"]; + const auto highest = std::find_if(added.begin(), added.end(), [](const auto& item) { + return item.at("id") == 200028; + }); + TEST_REQUIRE(highest != added.end()); + added.erase(highest); RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( gguf_fixture::ValidConfig(), tokenizer, gguf_fixture::ValidTokenizerConfig()); }), - "tokenizer.json maximum vocabulary ID", "200062", "200063"); + "tokenizer.json maximum vocabulary ID", "200027", "200028"); auto bos_file = Builder().SetMetadata("tokenizer.ggml.add_bos_token", true) .AddFullContractTensors().Write("wrong-gguf-bos"); @@ -499,10 +548,11 @@ void TestValidationCreatesNoCorelibObjects() { int main() { #define RUN(name) RunTest(name, #name) RUN(TestValidV3HeaderMetadataDirectoryAndAlignment); + RUN(TestOmittedAlignmentUsesGgufDefault32); RUN(TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely); RUN(TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail); RUN(TestCountProductAlignmentAndOffsetOverflowFail); - RUN(TestZeroAndNonPowerOfTwoAlignmentFail); + RUN(TestPresentMalformedAlignmentFails); RUN(TestDuplicateTensorNamesFail); RUN(TestOutOfFileAndOverlappingTensorRangesFail); RUN(TestUnsupportedUnskippableMetadataTypeFails); @@ -516,12 +566,14 @@ int main() { RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); RUN(TestRejectsMixedQuantizationAndOutputWeightPresence); RUN(TestRequiresTiedQ8TokenEmbeddingAsLmHead); - RUN(TestRequiresOriginal4096WindowAndRejectsLongRopeBranch); + RUN(TestRequiresOriginal4096WindowAndValidatesLongRopeFactors); RUN(TestValidatesOptionalShortRopeFactorsAsF32Length48); RUN(TestRejectsNonFiniteOrNonPositiveRopeValues); + RUN(TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads); RUN(TestRejectsConfigDisagreement); RUN(TestRejectsFiniteWrongRmsValue); RUN(TestDerivesStopSetFromGgufConfigAndTokenizerIds); + RUN(TestAcceptsPinnedDynamicRoleChatTemplate); RUN(TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement); RUN(TestValidationCreatesNoCorelibObjects); #undef RUN From a2f8550ca2576067b2e7d9a224df85c70b83efe1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:08:53 -0700 Subject: [PATCH 21/37] fixup! feat: add corelib-backed Phi-4 AIE4 engine --- src/common/corelib/phi4_corelib_host.cpp | 3 ++- src/common/corelib/phi4_corelib_shape_plan.cpp | 9 ++++----- src/test/phi4_corelib_aie4/test_phi4_engine.cpp | 7 +++---- src/test/phi4_corelib_aie4/test_phi4_host.cpp | 2 +- src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp | 10 ++++++---- 5 files changed, 16 insertions(+), 15 deletions(-) diff --git a/src/common/corelib/phi4_corelib_host.cpp b/src/common/corelib/phi4_corelib_host.cpp index 792531d0..d9bf5b40 100644 --- a/src/common/corelib/phi4_corelib_host.cpp +++ b/src/common/corelib/phi4_corelib_host.cpp @@ -98,7 +98,8 @@ std::vector ConvertF32ToBf16(std::span values) { RopeTables BuildShortRopeTables( const GgufPhi4Metadata& metadata, std::optional short_factors) { - if (metadata.context_length != static_cast(kMaxSequenceLength) || + if (metadata.context_length < static_cast(kMaxSequenceLength) || + metadata.rope_original_context_length != static_cast(kMaxSequenceLength) || metadata.rope_dimension_count != static_cast(kRopeDimension) || !std::isfinite(metadata.rope_frequency_base) || metadata.rope_frequency_base <= 0 || !std::isfinite(metadata.rope_attention_factor)) { diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp index 951411b4..4fb6306e 100644 --- a/src/common/corelib/phi4_corelib_shape_plan.cpp +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -57,12 +57,11 @@ Phi4ShapePlan Phi4ShapePlan::Build( &extents.ssmlp_rows, kHiddenSize, kIntermediateSize, kRequantizedGroupSize), ssmlp_call); + // RMSNorm tiles only M and its public contract preserves the caller's + // row extent. Avoid the metadata-only helper here: the pinned + // DynamicDispatch implementation dereferences its intentionally absent + // XRT context when a shape requires more than one tile. extents.rmsnorm_rows = rows; - const std::string rms_call = - "ryzenai_corelib_rmsnorm_bf16_pad_rows [" + std::to_string(rows) + - ",3072]"; - api->Check(api->functions().rmsnorm_pad_rows( - &extents.rmsnorm_rows, kHiddenSize), rms_call); extents.flat_mha_rows = rows; const std::string mha_call = diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index c300ab82..b6a801c0 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -64,13 +64,12 @@ void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { state.pad_row_overrides["matmul-3072"][2048] = 5000; state.pad_row_overrides["matmul-1024"][2048] = 6000; state.pad_row_overrides["ssmlp"][2048] = 7000; - state.pad_row_overrides["rmsnorm"][2048] = 8000; state.pad_row_overrides["mha"][2048] = 9000; }); const auto& tensors = fake_corelib::GetState().tensor_creates; - TEST_REQUIRE(tensors[0].shape == std::vector({8000, 3072})); - TEST_REQUIRE(tensors[1].shape == std::vector({8000, 3072})); - TEST_REQUIRE(tensors[2].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[0].shape == std::vector({7000, 3072})); + TEST_REQUIRE(tensors[1].shape == std::vector({7000, 3072})); + TEST_REQUIRE(tensors[2].shape == std::vector({7000, 3072})); TEST_REQUIRE(tensors[3].shape == std::vector({9000, 3072})); TEST_REQUIRE(tensors[4].shape == std::vector({9000, 1024})); TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); diff --git a/src/test/phi4_corelib_aie4/test_phi4_host.cpp b/src/test/phi4_corelib_aie4/test_phi4_host.cpp index 376b7712..d62baed5 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_host.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_host.cpp @@ -38,7 +38,7 @@ TensorView ThreeRows() { } GgufPhi4Metadata Metadata(double attention = 1.0) { - return {"phi3", 32, 3072, 8192, 24, 8, 4096, 96, 10000.0, + return {"phi3", 32, 3072, 8192, 24, 8, 131072, 96, 10000.0, attention, 4096, 200064, false}; } diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp index 96a6aa92..c1ac1c5f 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -19,12 +19,14 @@ void TestShapePlanQueriesRows1Through4096AtGroup64() { const auto plan = Phi4ShapePlan::Build(Api()); const auto& state = fake_corelib::GetState(); TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * 4096 + 1); - TEST_REQUIRE(state.rows_pad_calls.size() == 2 * 4096); + TEST_REQUIRE(state.rows_pad_calls.size() == 4096); TEST_REQUIRE(state.mha_pad_calls.size() == 4096); for (std::size_t row = 1; row <= 4096; ++row) { TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].m == static_cast(row)); TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].group_size == 64); - TEST_REQUIRE(state.rows_pad_calls[(row - 1) * 2].group_size == 64); + TEST_REQUIRE(state.rows_pad_calls[row - 1].helper == "ssmlp"); + TEST_REQUIRE(state.rows_pad_calls[row - 1].group_size == 64); + TEST_REQUIRE(plan.ForRows(row).rmsnorm_rows == static_cast(row)); TEST_REQUIRE(state.mha_pad_calls[row - 1].m == static_cast(row)); } TEST_REQUIRE(plan.ForRows(65).query_rows == 128); @@ -43,8 +45,8 @@ void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); - TEST_REQUIRE(state.rows_pad_calls[1].helper == "rmsnorm"); - TEST_REQUIRE(state.rows_pad_calls[1].k == 3072); + TEST_REQUIRE(plan.ForRows(1).rmsnorm_rows == 1); + TEST_REQUIRE(plan.ForRows(4096).rmsnorm_rows == 4096); const auto& lm = state.matmul_pad_calls.back(); TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); TEST_REQUIRE(plan.lm_head_desc().k == 3072); From ee78f6f32b74bdc1a269e873a3f9ce1ece19768b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:09:12 -0700 Subject: [PATCH 22/37] fixup! feat: route Phi-4 GGUF models through AIE4 --- src/common/AutoModel/modeling_phi4.cpp | 17 ++++++++++++ src/include/AutoModel/modeling_phi4.hpp | 2 ++ .../phi4_corelib_aie4/test_phi4_frontend.cpp | 27 +++++++++++++++++++ 3 files changed, 46 insertions(+) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 79d4dfca..fd610788 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -85,6 +85,7 @@ void Phi4::load_model(std::string model_path, json model_info, uses_corelib_aie4_ = false; aie4_poisoned_ = false; corelib_runtime_.reset(); + aie4_corelib_path_.clear(); if (switching_from_aie4) is_model_loaded = false; #endif _shared_load_model(model_path, model_info, default_context_length, enable_preemption); @@ -133,6 +134,9 @@ void Phi4::load_model(std::string model_path, json model_info, sampler.reset(); ConfigureSampler(*this); + const auto corelib_path = std::filesystem::absolute( + flm::corelib::CorelibApi::ResolveLibraryPath( + utils::get_executable_directory())).lexically_normal(); std::unique_ptr engine; #if defined(FLM_CORELIB_TESTING) if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); @@ -146,11 +150,13 @@ void Phi4::load_model(std::string model_path, json model_info, corelib_runtime_ = std::move(runtime); #endif engine->clear_context(); + aie4_corelib_path_ = corelib_path; lm_engine = std::move(engine); uses_corelib_aie4_ = true; } catch (...) { lm_engine.reset(); corelib_runtime_.reset(); + aie4_corelib_path_.clear(); tokenizer.reset(); sampler.reset(); lm_config.reset(); @@ -287,6 +293,17 @@ std::string Phi4::generate_aie4(chat_meta_info_t& meta_info, } #endif +std::string Phi4::show_profile() { + std::string profile = AutoModel::show_profile(); +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + profile += " Backend: corelib_aie4_gguf\n"; + profile += " Corelib DLL: " + aie4_corelib_path_.string() + "\n"; + } +#endif + return profile; +} + void Phi4::clear_context() { #if defined(FLM_ENABLE_CORELIB_AIE4) if (uses_corelib_aie4_ && aie4_poisoned_) { diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index ce71f5d9..7a171cc7 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -30,6 +30,7 @@ class Phi4 : public AutoModel { bool aie4_poisoned_ = false; int aie4_generation_budget_ = 0; std::shared_ptr corelib_runtime_; + std::filesystem::path aie4_corelib_path_; #endif #if defined(FLM_CORELIB_TESTING) @@ -53,6 +54,7 @@ class Phi4 : public AutoModel { return false; #endif } + std::string show_profile() override; void clear_context() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index d7fd07bb..67a7bf87 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -17,6 +17,7 @@ #include #include #include +#include namespace { @@ -283,6 +284,31 @@ void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); } +void TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile() { + TempPackage package; + FactoryScope scope; + const auto dll = std::filesystem::absolute(package.path() / "ryzenai_corelib.dll"); +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", dll.string().c_str()); +#else + setenv("FLM_AIE4_CORELIB_PATH", dll.string().c_str(), 1); +#endif + auto aie4 = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + const auto aie4_profile = aie4->show_profile(); + RequireContains(aie4_profile, "corelib_aie4_gguf"); + RequireContains(aie4_profile, dll.string()); + + auto legacy = Load(package, ModelInfo()); + const auto legacy_profile = legacy->show_profile(); + TEST_REQUIRE(legacy_profile.find("corelib_aie4_gguf") == std::string::npos); + TEST_REQUIRE(legacy_profile.find(dll.string()) == std::string::npos); +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", ""); +#else + unsetenv("FLM_AIE4_CORELIB_PATH"); +#endif +} + void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { TempPackage package; FactoryScope scope; @@ -636,6 +662,7 @@ int main() { RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); RunTest(TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing, "TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing"); RunTest(TestCorelibAie4GgufBuildsOnlyTheCorelibEngine, "TestCorelibAie4GgufBuildsOnlyTheCorelibEngine"); + RunTest(TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile, "TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile"); RunTest(TestNoManifestOnnxConvertedWeightOrCachePathIsOpened, "TestNoManifestOnnxConvertedWeightOrCachePathIsOpened"); RunTest(TestUnknownAndNonStringBackendAreErrors, "TestUnknownAndNonStringBackendAreErrors"); RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); From f7eb0c16f6801e93370832815a4acca061e08cc4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:31:42 -0700 Subject: [PATCH 23/37] fixup! feat: add corelib-backed Phi-4 AIE4 engine --- src/common/corelib/phi4_corelib_shape_plan.cpp | 9 +++++---- src/test/phi4_corelib_aie4/test_phi4_engine.cpp | 7 ++++--- src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp | 10 ++++------ 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp index 4fb6306e..951411b4 100644 --- a/src/common/corelib/phi4_corelib_shape_plan.cpp +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -57,11 +57,12 @@ Phi4ShapePlan Phi4ShapePlan::Build( &extents.ssmlp_rows, kHiddenSize, kIntermediateSize, kRequantizedGroupSize), ssmlp_call); - // RMSNorm tiles only M and its public contract preserves the caller's - // row extent. Avoid the metadata-only helper here: the pinned - // DynamicDispatch implementation dereferences its intentionally absent - // XRT context when a shape requires more than one tile. extents.rmsnorm_rows = rows; + const std::string rms_call = + "ryzenai_corelib_rmsnorm_bf16_pad_rows [" + std::to_string(rows) + + ",3072]"; + api->Check(api->functions().rmsnorm_pad_rows( + &extents.rmsnorm_rows, kHiddenSize), rms_call); extents.flat_mha_rows = rows; const std::string mha_call = diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index b6a801c0..c300ab82 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -64,12 +64,13 @@ void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { state.pad_row_overrides["matmul-3072"][2048] = 5000; state.pad_row_overrides["matmul-1024"][2048] = 6000; state.pad_row_overrides["ssmlp"][2048] = 7000; + state.pad_row_overrides["rmsnorm"][2048] = 8000; state.pad_row_overrides["mha"][2048] = 9000; }); const auto& tensors = fake_corelib::GetState().tensor_creates; - TEST_REQUIRE(tensors[0].shape == std::vector({7000, 3072})); - TEST_REQUIRE(tensors[1].shape == std::vector({7000, 3072})); - TEST_REQUIRE(tensors[2].shape == std::vector({7000, 3072})); + TEST_REQUIRE(tensors[0].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[1].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[2].shape == std::vector({8000, 3072})); TEST_REQUIRE(tensors[3].shape == std::vector({9000, 3072})); TEST_REQUIRE(tensors[4].shape == std::vector({9000, 1024})); TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp index c1ac1c5f..96a6aa92 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -19,14 +19,12 @@ void TestShapePlanQueriesRows1Through4096AtGroup64() { const auto plan = Phi4ShapePlan::Build(Api()); const auto& state = fake_corelib::GetState(); TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * 4096 + 1); - TEST_REQUIRE(state.rows_pad_calls.size() == 4096); + TEST_REQUIRE(state.rows_pad_calls.size() == 2 * 4096); TEST_REQUIRE(state.mha_pad_calls.size() == 4096); for (std::size_t row = 1; row <= 4096; ++row) { TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].m == static_cast(row)); TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].group_size == 64); - TEST_REQUIRE(state.rows_pad_calls[row - 1].helper == "ssmlp"); - TEST_REQUIRE(state.rows_pad_calls[row - 1].group_size == 64); - TEST_REQUIRE(plan.ForRows(row).rmsnorm_rows == static_cast(row)); + TEST_REQUIRE(state.rows_pad_calls[(row - 1) * 2].group_size == 64); TEST_REQUIRE(state.mha_pad_calls[row - 1].m == static_cast(row)); } TEST_REQUIRE(plan.ForRows(65).query_rows == 128); @@ -45,8 +43,8 @@ void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); - TEST_REQUIRE(plan.ForRows(1).rmsnorm_rows == 1); - TEST_REQUIRE(plan.ForRows(4096).rmsnorm_rows == 4096); + TEST_REQUIRE(state.rows_pad_calls[1].helper == "rmsnorm"); + TEST_REQUIRE(state.rows_pad_calls[1].k == 3072); const auto& lm = state.matmul_pad_calls.back(); TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); TEST_REQUIRE(plan.lm_head_desc().k == 3072); From cd2223ae91a6844c7f42750e1a34249de88f1f69 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:31:42 -0700 Subject: [PATCH 24/37] fixup! build: add optional dynamic corelib 0.3.0 runtime --- src/common/corelib/corelib_api.cpp | 16 ++++++++++++---- src/common/corelib/corelib_runtime.cpp | 5 +++++ src/include/corelib/corelib_api.hpp | 8 ++++++-- src/include/corelib/corelib_runtime.hpp | 1 + 4 files changed, 24 insertions(+), 6 deletions(-) diff --git a/src/common/corelib/corelib_api.cpp b/src/common/corelib/corelib_api.cpp index d2473b7f..cb98762d 100644 --- a/src/common/corelib/corelib_api.cpp +++ b/src/common/corelib/corelib_api.cpp @@ -58,7 +58,10 @@ ryzenai_corelib_status CorelibError::status() const noexcept { return status_; } const std::string& CorelibError::call() const noexcept { return call_; } const std::string& CorelibError::detail() const noexcept { return detail_; } -CorelibApi::CorelibApi(Resolver resolver) : resolver_(std::move(resolver)) { +CorelibApi::CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path) + : resolver_(std::move(resolver)), + loaded_library_path_(std::move(loaded_library_path)) { void* version_symbol = resolver_("ryzenai_corelib_get_version"); if (!version_symbol) { throw std::runtime_error("missing corelib symbol: ryzenai_corelib_get_version"); @@ -86,9 +89,11 @@ CorelibApi::CorelibApi(Resolver resolver) : resolver_(std::move(resolver)) { #undef FLM_RESOLVE_CORELIB_FUNCTION } -std::shared_ptr CorelibApi::ResolveForTest(Resolver resolver) { +std::shared_ptr CorelibApi::ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path) { if (!resolver) throw std::invalid_argument("corelib resolver is empty"); - return std::shared_ptr(new CorelibApi(std::move(resolver))); + return std::shared_ptr(new CorelibApi( + std::move(resolver), std::move(loaded_library_path))); } std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { @@ -113,7 +118,7 @@ std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { return reinterpret_cast( GetProcAddress(static_cast(module.get()), terminated.c_str())); }; - return ResolveForTest(std::move(resolver)); + return ResolveForTest(std::move(resolver), absolute_dll); #endif } @@ -138,6 +143,9 @@ std::filesystem::path CorelibApi::ResolveLibraryPath( const CorelibFunctions& CorelibApi::functions() const noexcept { return functions_; } CorelibVersion CorelibApi::runtime_version() const noexcept { return runtime_version_; } +const std::filesystem::path& CorelibApi::loaded_library_path() const noexcept { + return loaded_library_path_; +} void CorelibApi::Check(ryzenai_corelib_status status, std::string_view call) const { diff --git a/src/common/corelib/corelib_runtime.cpp b/src/common/corelib/corelib_runtime.cpp index d8a6073e..937051c9 100644 --- a/src/common/corelib/corelib_runtime.cpp +++ b/src/common/corelib/corelib_runtime.cpp @@ -94,4 +94,9 @@ const std::shared_ptr& CorelibRuntime::api() const noexcept { return api_; } +const std::filesystem::path& CorelibRuntime::loaded_library_path() const noexcept { + static const std::filesystem::path empty; + return api_ ? api_->loaded_library_path() : empty; +} + } // namespace flm::corelib diff --git a/src/include/corelib/corelib_api.hpp b/src/include/corelib/corelib_api.hpp index c94e5474..e251ee0e 100644 --- a/src/include/corelib/corelib_api.hpp +++ b/src/include/corelib/corelib_api.hpp @@ -81,22 +81,26 @@ class CorelibApi final { public: using Resolver = std::function; static std::shared_ptr Load(const std::filesystem::path& dll); - static std::shared_ptr ResolveForTest(Resolver resolver); + static std::shared_ptr ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path = {}); static std::filesystem::path ResolveLibraryPath( const std::filesystem::path& executable_dir); const CorelibFunctions& functions() const noexcept; CorelibVersion runtime_version() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; void Check(ryzenai_corelib_status status, std::string_view call) const; void RegisterObject() const noexcept; void Release(void* object) const noexcept; std::size_t live_object_count() const noexcept; private: - explicit CorelibApi(Resolver resolver); + explicit CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path = {}); Resolver resolver_; CorelibFunctions functions_{}; CorelibVersion runtime_version_{}; + std::filesystem::path loaded_library_path_; mutable std::atomic live_object_count_{0}; }; diff --git a/src/include/corelib/corelib_runtime.hpp b/src/include/corelib/corelib_runtime.hpp index 0027f8be..d0ad61c2 100644 --- a/src/include/corelib/corelib_runtime.hpp +++ b/src/include/corelib/corelib_runtime.hpp @@ -24,6 +24,7 @@ class CorelibRuntime final { #endif std::unique_lock AcquireExecution(); const std::shared_ptr& api() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; private: explicit CorelibRuntime(std::shared_ptr api); From 7347f5b8a695068d41210bcf517a160668691dbe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 11:31:43 -0700 Subject: [PATCH 25/37] fixup! feat: route Phi-4 GGUF models through AIE4 --- src/common/AutoModel/modeling_phi4.cpp | 10 ++---- src/include/AutoModel/modeling_phi4.hpp | 1 - src/test/phi4_corelib_aie4/CMakeLists.txt | 2 +- .../phi4_corelib_aie4/test_phi4_frontend.cpp | 36 +++++++++++++++---- 4 files changed, 33 insertions(+), 16 deletions(-) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index fd610788..4ca36777 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -85,7 +85,6 @@ void Phi4::load_model(std::string model_path, json model_info, uses_corelib_aie4_ = false; aie4_poisoned_ = false; corelib_runtime_.reset(); - aie4_corelib_path_.clear(); if (switching_from_aie4) is_model_loaded = false; #endif _shared_load_model(model_path, model_info, default_context_length, enable_preemption); @@ -134,9 +133,6 @@ void Phi4::load_model(std::string model_path, json model_info, sampler.reset(); ConfigureSampler(*this); - const auto corelib_path = std::filesystem::absolute( - flm::corelib::CorelibApi::ResolveLibraryPath( - utils::get_executable_directory())).lexically_normal(); std::unique_ptr engine; #if defined(FLM_CORELIB_TESTING) if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); @@ -150,13 +146,11 @@ void Phi4::load_model(std::string model_path, json model_info, corelib_runtime_ = std::move(runtime); #endif engine->clear_context(); - aie4_corelib_path_ = corelib_path; lm_engine = std::move(engine); uses_corelib_aie4_ = true; } catch (...) { lm_engine.reset(); corelib_runtime_.reset(); - aie4_corelib_path_.clear(); tokenizer.reset(); sampler.reset(); lm_config.reset(); @@ -298,7 +292,9 @@ std::string Phi4::show_profile() { #if defined(FLM_ENABLE_CORELIB_AIE4) if (uses_corelib_aie4_) { profile += " Backend: corelib_aie4_gguf\n"; - profile += " Corelib DLL: " + aie4_corelib_path_.string() + "\n"; + if (corelib_runtime_) + profile += " Corelib DLL: " + + corelib_runtime_->loaded_library_path().string() + "\n"; } #endif return profile; diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index 7a171cc7..b416d56d 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -30,7 +30,6 @@ class Phi4 : public AutoModel { bool aie4_poisoned_ = false; int aie4_generation_budget_ = 0; std::shared_ptr corelib_runtime_; - std::filesystem::path aie4_corelib_path_; #endif #if defined(FLM_CORELIB_TESTING) diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt index 802de6cb..bd634950 100644 --- a/src/test/phi4_corelib_aie4/CMakeLists.txt +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -122,7 +122,7 @@ set(PHI4_FRONTEND_SOURCES "${FLM_SOURCE_DIR}/common/AutoModel/modeling_phi4.cpp") add_executable(test_phi4_frontend - test_phi4_frontend.cpp + test_phi4_frontend.cpp fake_corelib.cpp ${PHI4_FRONTEND_SOURCES} ${CORELIB_SOURCES} "${FLM_SOURCE_DIR}/common/corelib/phi4_corelib_gguf.cpp") diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index 67a7bf87..23cea2d0 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -1,5 +1,8 @@ #include "test_support.hpp" #include "gguf_fixture.hpp" +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "fake_corelib.hpp" +#endif #include "utils/file_access.hpp" #include @@ -214,6 +217,12 @@ class Phi4FrontendTestAccess final { static const std::string& EosToken(const Phi4& model) { return model.eos_token; } static const std::vector& EosTokenIds(const Phi4& model) { return model.eos_token_ids; } static bool HasBosToken(const Phi4& model) { return model.has_bos_token; } +#if defined(FLM_ENABLE_CORELIB_AIE4) + static void SetRuntime(Phi4& model, + std::shared_ptr runtime) { + model.corelib_runtime_ = std::move(runtime); + } +#endif }; } // namespace flm::phi4::testing @@ -284,30 +293,43 @@ void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); } -void TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile() { +#if defined(FLM_ENABLE_CORELIB_AIE4) +void TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges() { TempPackage package; FactoryScope scope; - const auto dll = std::filesystem::absolute(package.path() / "ryzenai_corelib.dll"); + fake_corelib::Reset(); + const auto dll_a = std::filesystem::absolute(package.path() / "runtime-a.dll"); + const auto dll_b = std::filesystem::absolute(package.path() / "runtime-b.dll"); + auto api = flm::corelib::CorelibApi::ResolveForTest( + fake_corelib::Resolver(), dll_a); + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(std::move(api)); + #ifdef _WIN32 - _putenv_s("FLM_AIE4_CORELIB_PATH", dll.string().c_str()); + _putenv_s("FLM_AIE4_CORELIB_PATH", dll_b.string().c_str()); #else - setenv("FLM_AIE4_CORELIB_PATH", dll.string().c_str(), 1); + setenv("FLM_AIE4_CORELIB_PATH", dll_b.string().c_str(), 1); #endif auto aie4 = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + Phi4FrontendTestAccess::SetRuntime(*aie4, runtime); const auto aie4_profile = aie4->show_profile(); RequireContains(aie4_profile, "corelib_aie4_gguf"); - RequireContains(aie4_profile, dll.string()); + RequireContains(aie4_profile, dll_a.string()); + TEST_REQUIRE(aie4_profile.find(dll_b.string()) == std::string::npos); auto legacy = Load(package, ModelInfo()); const auto legacy_profile = legacy->show_profile(); TEST_REQUIRE(legacy_profile.find("corelib_aie4_gguf") == std::string::npos); - TEST_REQUIRE(legacy_profile.find(dll.string()) == std::string::npos); + TEST_REQUIRE(legacy_profile.find(dll_a.string()) == std::string::npos); + aie4.reset(); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); #ifdef _WIN32 _putenv_s("FLM_AIE4_CORELIB_PATH", ""); #else unsetenv("FLM_AIE4_CORELIB_PATH"); #endif } +#endif void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { TempPackage package; @@ -662,7 +684,7 @@ int main() { RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); RunTest(TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing, "TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing"); RunTest(TestCorelibAie4GgufBuildsOnlyTheCorelibEngine, "TestCorelibAie4GgufBuildsOnlyTheCorelibEngine"); - RunTest(TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile, "TestAie4ProfileNamesBackendAndAbsoluteCorelibDllWithoutChangingLegacyProfile"); + RunTest(TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges, "TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges"); RunTest(TestNoManifestOnnxConvertedWeightOrCachePathIsOpened, "TestNoManifestOnnxConvertedWeightOrCachePathIsOpened"); RunTest(TestUnknownAndNonStringBackendAreErrors, "TestUnknownAndNonStringBackendAreErrors"); RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); From edd5cad5477457d2382dcf5a83b76c339dbaf93b Mon Sep 17 00:00:00 2001 From: Zhang Date: Fri, 11 Sep 2026 16:07:13 -0600 Subject: [PATCH 26/37] fixup! feat: add corelib-backed Phi-4 AIE4 engine --- src/common/corelib/phi4_corelib_aie4.cpp | 4 +-- .../corelib/phi4_corelib_shape_plan.cpp | 8 ++++-- .../phi4_corelib_aie4/test_phi4_engine.cpp | 15 ++++++++++- .../test_phi4_shape_plan.cpp | 27 ++++++++++++------- 4 files changed, 39 insertions(+), 15 deletions(-) diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp index b13a2bdd..cab588a1 100644 --- a/src/common/corelib/phi4_corelib_aie4.cpp +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -128,11 +128,11 @@ struct phi4_corelib_aie4::Impl { auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, e.ssmlp_rows,e.rmsnorm_rows}); std::vector input(static_cast(rows*kHiddenSize),0);std::copy(decoded.begin(),decoded.end(),input.begin()); - std::vector zeros(static_cast(rows*kHiddenSize),0); + auto lease=runtime->AcquireExecution();bool submitted=false; try{ api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); - api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_bf16,zeros.data(),zeros.size(),0),"ryzenai_corelib_tensor_write residual padding"); + api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write residual embedding"); const auto rms_status=api->functions().rmsnorm( stream.get(),hidden.get(),ids.size(),first_norm.get(),hidden.get()); submitted=rms_status==ryzenai_corelib_status_success || diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp index 951411b4..fd81854f 100644 --- a/src/common/corelib/phi4_corelib_shape_plan.cpp +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -3,6 +3,7 @@ #include "models/phi4/phi4_corelib_constants.hpp" #include +#include #include #include @@ -39,8 +40,10 @@ Phi4ShapePlan Phi4ShapePlan::Build( plan.lm_head_desc_ = {kHiddenSize, kVocabularySize, kRequantizedGroupSize, false}; plan.rows_.reserve(kMaxSequenceLength); + constexpr std::array execution_rows{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; - for (std::int64_t rows = 1; rows <= kMaxSequenceLength; ++rows) { + for (const auto rows : execution_rows) { Phi4RowExtents extents{}; extents.query_rows = MatmulRows(api, rows, kHiddenSize, kQueryDimension, "query"); @@ -82,7 +85,8 @@ Phi4ShapePlan Phi4ShapePlan::Build( plan.maximum_extents_.rmsnorm_rows, extents.rmsnorm_rows); plan.maximum_extents_.flat_mha_rows = std::max( plan.maximum_extents_.flat_mha_rows, extents.flat_mha_rows); - plan.rows_.push_back(extents); + while (plan.rows_.size() < static_cast(rows)) + plan.rows_.push_back(extents); } (void)MatmulRows(api, 1, kHiddenSize, kVocabularySize, "lm_head"); diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index c300ab82..6fe6c015 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -242,9 +242,21 @@ void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { TEST_REQUIRE(std::all_of(calls.begin(), calls.end(), [&](const auto& c) { return c.stream == stream; })); } +void TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual() { + Harness h; + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes.size() >= 2); + TEST_REQUIRE(writes[0].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[1].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[0].count == writes[1].count); +} + void TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket() { Harness h([](auto& state) { - state.pad_row_overrides["matmul-1024"][2] = 96; + state.pad_row_overrides["matmul-1024"][64] = 96; }); fake_corelib::GetState().tensor_writes.clear(); std::vector ids{1, 2}; @@ -518,6 +530,7 @@ int main() { RUN_TEST(TestDecodeUsesOneRowAndAdvancesPosition); RUN_TEST(TestVProjectionWritesWindowAtPositionTimes128); RUN_TEST(TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream); + RUN_TEST(TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual); RUN_TEST(TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket); RUN_TEST(TestForwardSynchronizesBeforeHostReadAndLmHeadRead); RUN_TEST(TestKVCachesRemainFixedAt8By4096By128); diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp index 96a6aa92..04801aad 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -3,6 +3,7 @@ #include "test_support.hpp" #include +#include #include #include @@ -14,20 +15,26 @@ std::shared_ptr Api() { return CorelibApi::ResolveForTest(fake_corelib::Resolver()); } -void TestShapePlanQueriesRows1Through4096AtGroup64() { +void TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow() { fake_corelib::Reset(); const auto plan = Phi4ShapePlan::Build(Api()); const auto& state = fake_corelib::GetState(); - TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * 4096 + 1); - TEST_REQUIRE(state.rows_pad_calls.size() == 2 * 4096); - TEST_REQUIRE(state.mha_pad_calls.size() == 4096); - for (std::size_t row = 1; row <= 4096; ++row) { - TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].m == static_cast(row)); - TEST_REQUIRE(state.matmul_pad_calls[(row - 1) * 3].group_size == 64); - TEST_REQUIRE(state.rows_pad_calls[(row - 1) * 2].group_size == 64); - TEST_REQUIRE(state.mha_pad_calls[row - 1].m == static_cast(row)); + constexpr std::array buckets{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; + TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * buckets.size() + 1); + TEST_REQUIRE(state.rows_pad_calls.size() == 2 * buckets.size()); + TEST_REQUIRE(state.mha_pad_calls.size() == buckets.size()); + for (std::size_t index = 0; index < buckets.size(); ++index) { + TEST_REQUIRE(state.matmul_pad_calls[index * 3].m == buckets[index]); + TEST_REQUIRE(state.matmul_pad_calls[index * 3].group_size == 64); + TEST_REQUIRE(state.rows_pad_calls[index * 2].m == buckets[index]); + TEST_REQUIRE(state.rows_pad_calls[index * 2 + 1].m == buckets[index]); + TEST_REQUIRE(state.mha_pad_calls[index].m == buckets[index]); } + TEST_REQUIRE(plan.ForRows(2).query_rows == 64); TEST_REQUIRE(plan.ForRows(65).query_rows == 128); + TEST_REQUIRE(plan.ForRows(257).query_rows == 512); + TEST_REQUIRE(plan.ForRows(4095).query_rows == 4096); } void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { @@ -92,7 +99,7 @@ void TestShapePlanFailureNamesHelperAndLogicalShape() { int main() { #define RUN_TEST(name) RunTest(&name, #name) - RUN_TEST(TestShapePlanQueriesRows1Through4096AtGroup64); + RUN_TEST(TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow); RUN_TEST(TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions); RUN_TEST(TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96); RUN_TEST(TestShapePlanRejectsPaddedKOrNChanges); From 465f49b189a48482e3d49a150a69fb3e136249b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 16:18:42 -0700 Subject: [PATCH 27/37] docs: drop the superpowers plan and design pages from the branch The implementation plan and design spec are process artifacts for how this branch was built, not documentation the project ships. They remain in this branch's history (df83bf6c, 4948126b) for anyone who wants them. Co-Authored-By: Claude Opus 5 (1M context) --- .../plans/2026-09-11-phi4-gguf-aie4.md | 1300 ----------------- .../specs/2026-09-11-phi4-gguf-aie4-design.md | 372 ----- 2 files changed, 1672 deletions(-) delete mode 100644 docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md delete mode 100644 docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md diff --git a/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md b/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md deleted file mode 100644 index 65706fa7..00000000 --- a/docs/superpowers/plans/2026-09-11-phi4-gguf-aie4.md +++ /dev/null @@ -1,1300 +0,0 @@ -# Phi-4 Q8_0 GGUF on AIE4 Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add the catalog model `phi4-mini-it-aie4:4b`, pull its pinned GGUF and tokenizer/config files, and run it through the dynamically loaded ryzenai-corelib 0.3.0 AIE4 backend from the normal FastFlowLM CLI and REST server. - -**Architecture:** Keep `Phi4` as the existing tokenizer/chat/sampling frontend and select a new `phi4_corelib_aie4` causal-LM engine only when `details.execution_backend == "corelib_aie4_gguf"`. The engine owns a validated, read-only GGUF mapping, derives host-only embedding/norm/RoPE data, creates all corelib weights serially through the explicit Q8_0-to-group-64 APIs, and executes one-stream prefill/decode with fixed KV caches. A feature-gated dynamic ABI layer keeps default builds and non-AIE4 models independent of the corelib DLL. - -**Tech Stack:** C++20, CMake 3.22+, Windows `LoadLibraryExW`/file mapping APIs, nlohmann/json, libcurl, existing FastFlowLM tokenizer/sampler/server, ryzenai-corelib C ABI 0.3.0, CTest, PowerShell for real-device acceptance. - -**Spec:** [`docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md`](../specs/2026-09-11-phi4-gguf-aie4-design.md) - -## Global Constraints - -- Work from current FastFlowLM `main`; use `origin/pr/706` (`0355fe4c4f3bf4bdb476ef5fd9c20f84411a162f`) only as a structural reference. Do not cherry-pick its ONNX manifest, overlay, packaging, process-termination policy, or stale corelib ABI. -- Corelib source is `VitisAI/ryzenai-corelib` commit `3c35aebdefa3f0c2255668bab1be5648ece320f8`; compile against its public `include/ryzenai/corelib.h` only and require runtime ABI exactly `0.3.0` (major, minor, and patch). -- The model is exactly `unsloth/Phi-4-mini-instruct-GGUF` revision `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80`, file `Phi-4-mini-instruct.Q8_0.gguf`, GGML type Q8_0. -- `tokenizer.json`, `tokenizer_config.json`, and `config.json` are exactly from `microsoft/Phi-4-mini-instruct` revision `cfbefacb99257ffa30c83adab238a50856ac3083`. -- Keep the existing `phi4-mini-it:4b` Q4NX/NPU2 path unchanged. An absent `details.execution_backend` means legacy NPU2; `corelib_aie4_gguf` means only AIE4; every other value is an error. -- Never infer a backend from hardware, filenames, or quantization. Never fall back from `corelib_aie4_gguf` to CPU or NPU2. -- `FLM_ENABLE_CORELIB_AIE4` defaults to `OFF`. A default build must not require corelib headers or DLLs. An enabled build uses `RYZENAI_CORELIB_INCLUDE_DIR`, does not link `ryzenai_corelib.lib`, and loads corelib only after an AIE4 model is selected. -- DLL lookup order is exactly: the absolute file named by `FLM_AIE4_CORELIB_PATH`, then `/aie4/ryzenai_corelib.dll`. Never search the current working directory. -- This PR does not copy or package corelib, DynamicDispatch, XRT, RyzenMM, or any other runtime DLL. Do not change MSI, WiX, Inno Setup, or installer inputs. -- Do not add ONNX loading, an ONNX initializer manifest, a JSON tensor manifest, converted weights, a packed-weight cache, arbitrary local-GGUF support, another model family, another quantization, or Python runtime code. -- GGUF is version 3, little-endian, directly memory-mapped read-only. All counts, products, offsets, alignments, and byte ranges use checked arithmetic before pointer/span creation. -- Validate the complete GGUF, config, and tokenizer contract before creating a stream, device tensor, or device weight. -- Fixed model contract: architecture `phi3`; 32 layers; hidden 3072; intermediate 8192; 24 query heads; 8 KV heads; head size 128; vocabulary 200064; partial rotary width 96; RMS epsilon `1e-5`; original/max supported context 4096. -- Require finite positive `phi3.rope.freq_base` and `phi3.rope.scaling.attn_factor`; require `phi3.rope.scaling.original_context_length == 4096`; reject the long-RoPE branch; accept optional `rope_factors_short.weight` only as F32 `[48]`. -- Require every projection, `token_embd.weight`, and tied LM-head source to be Q8_0; require `output.weight` to be absent; require every norm to be F32; split fused QKV and gate/up only on complete Q8_0 rows (34 bytes per 32 weights), without copy or dequantization. -- Require tokenizer vocabulary size 200064; require `tokenizer.json` to map `<|end|>` to 200020 and `<|endoftext|>` to 199999; require GGUF EOS 200020 and `config.json` EOS 199999; configure the frontend stop set as their union `{200020, 199999}`. Require `tokenizer_config.json` `add_bos_token == false` and a chat template containing `<|user|>`, `<|end|>`, and `<|assistant|>`. -- Every error for a model field or tensor names the field/tensor, actual value, and expected value. Do not repair, reinterpret, or silently accept mismatches. -- Every quantized weight uses `group_size = 64`, `ryzenai_corelib_gguf_quant_type_q8_0`, and the explicit `*_weights_create_gguf_requantized` entry point. The conversion is lossy by design. -- Create the 129 matmul weights (Q/K/V/O for 32 layers plus tied LM head), 32 SSMLP weights, and one RMSNorm weight serially with `threads = 0`; do not add concurrent creation. -- Keep one corelib stream, fixed K/V caches shaped `[8,4096,128]`, helper-derived padded extents, whole-prompt prefill, and one-token decode. The maximum usable total decode window is 4095. -- Check cancellation before prefill and between decode steps. Never release/destroy a stream with submitted work outstanding; synchronize submitted work before releasing request ownership. -- A failure before the first successful submission is recoverable. A failure after submission or during synchronization poisons that model instance, clears its conversation state, and makes later requests fail until unload/reload. -- Process-wide AIE4 request access and per-instance mutable state are serialized. Existing server NPU serialization may be reused, but every generation route must participate and exception/cancellation paths must release it exactly once. -- `model_info.json` remains authoritative for each file's exact byte size and content hash. A final model is available only when all four final files validate; `.part` files never count. -- Keep all work in one PR and use the seven commit messages fixed by the design. Each product commit 1–5 must compile and pass its focused tests before the next product commit begins. -- The PR is not complete until the real-DLL checks and the full real-AIE4 acceptance matrix pass and the hardware record is committed. Performance is descriptive unless a separate threshold is approved. - -## Assumptions - -- The AIE4 feature is built and exercised on Windows with MSVC; Linux/default builds remain feature-off and unchanged. -- `src/CMakePresets.json` remains the source of the current FastFlowLM/NPU version values; the new preset inherits them rather than duplicating them. -- The developer provides a corelib 0.3.0 installation and its dependency directories. This PR locates only `ryzenai_corelib.dll`; dependent DLL discovery remains the Windows loader's responsibility. -- The sibling `../ryzenai-corelib` working tree is currently on another branch, while `origin/main` points to the required commit. Verification and build commands must address commit `3c35aebdefa3f0c2255668bab1be5648ece320f8` explicitly and must not overwrite sibling work. -- Exact remote file sizes and SHA-256 values are immutable implementation data obtainable from the pinned URLs. Task 5 computes and independently verifies them before catalog edits; no unverified value may be committed. -- Real-AIE4 performance values are not known until Task 7 runs. The documentation commit is blocked until the acceptance script has emitted actual values and provenance. - ---- - -## Repository Findings and Reuse Boundaries - -- Current `src/common/AutoModel/modeling_phi4.cpp` always calls `_shared_load_model`, constructs `Q4NX`, and creates `phi4_npu`; backend selection must be added there, not in `get_auto_model`. -- Current `_shared_load_model` in `src/common/AutoModel/automodel.cpp` combines generic model/tokenizer state with NPU2 `npu_xclbin_manager` creation. Split those responsibilities so the AIE4 route never constructs the legacy backend. -- Current downloads write directly to the final path and use base-repository URLs. The per-file source and atomic-resume behavior therefore require coordinated changes in `model_downloader.*` and `download_model.*`. -- Current server request queuing already serializes most NPU endpoints, but `/v1/completions` is omitted and exception paths manually release the lock. Extend the existing mechanism rather than adding a second queue. -- `origin/pr/706` supplies useful shapes for the dynamic loader, move-only handles, shape plan, fake corelib, frontend routing, and engine sequencing. Its source contract is ONNX/manifest-based, its symbol list predates ABI 0.3.0, its environment variable is different, and its post-submit policy terminates the process; none of those parts are reusable unchanged. -- The pinned corelib header supplies tensor windows, matmul/SSMLP Q8_0 requantized creation, RMSNorm scale creation/dispatch, and flat-MHA. The implementation must resolve exactly those public functions and no testing-only symbols. - -## Locked Interfaces - -Use these names and signatures throughout the tasks so independently implemented pieces join without renaming: - -```cpp -namespace flm::corelib { - -struct CorelibVersion { - std::uint32_t major; - std::uint32_t minor; - std::uint32_t patch; -}; - -class CorelibError final : public std::runtime_error { -public: - CorelibError(ryzenai_corelib_status status, - std::string call, - std::string detail, - std::string status_text); - ryzenai_corelib_status status() const noexcept; - const std::string& call() const noexcept; - const std::string& detail() const noexcept; -}; - -struct CorelibFunctions { - decltype(&::ryzenai_corelib_get_version) get_version; - decltype(&::ryzenai_corelib_status_to_string) status_to_string; - decltype(&::ryzenai_corelib_get_last_error_message) get_last_error_message; - decltype(&::ryzenai_corelib_selftest_dependencies) selftest_dependencies; - decltype(&::ryzenai_corelib_has_device_context) has_device_context; - decltype(&::ryzenai_corelib_object_release) object_release; - decltype(&::ryzenai_corelib_create_stream) create_stream; - decltype(&::ryzenai_corelib_stream_synchronize) stream_synchronize; - decltype(&::ryzenai_corelib_create_device_tensor) create_device_tensor; - decltype(&::ryzenai_corelib_create_tensor_window) create_tensor_window; - decltype(&::ryzenai_corelib_tensor_write) tensor_write; - decltype(&::ryzenai_corelib_tensor_read) tensor_read; - decltype(&::ryzenai_corelib_tensor_get_byte_size) tensor_get_byte_size; - decltype(&::ryzenai_corelib_tensor_get_data_type) tensor_get_data_type; - decltype(&::ryzenai_corelib_matmul_bf16_pad_shape) matmul_pad_shape; - decltype(&::ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized) - matmul_weights_create_gguf_requantized; - decltype(&::ryzenai_corelib_matmul_bf16) matmul; - decltype(&::ryzenai_corelib_ssmlp_bf16_pad_rows) ssmlp_pad_rows; - decltype(&::ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) - ssmlp_weights_create_gguf_requantized; - decltype(&::ryzenai_corelib_ssmlp_bf16) ssmlp; - decltype(&::ryzenai_corelib_rmsnorm_bf16_weights_create_scale) - rmsnorm_weights_create_scale; - decltype(&::ryzenai_corelib_rmsnorm_bf16_pad_rows) rmsnorm_pad_rows; - decltype(&::ryzenai_corelib_rmsnorm_bf16) rmsnorm; - decltype(&::ryzenai_corelib_flat_mha_bf16_pad_rows) flat_mha_pad_rows; - decltype(&::ryzenai_corelib_flat_mha_bf16) flat_mha; - decltype(&::ryzenai_corelib_cleanup) cleanup; -}; - -class CorelibApi final { -public: - using Resolver = std::function; - static std::shared_ptr Load(const std::filesystem::path& dll); - static std::shared_ptr ResolveForTest(Resolver resolver); - static std::filesystem::path ResolveLibraryPath( - const std::filesystem::path& executable_dir); - const CorelibFunctions& functions() const noexcept; - CorelibVersion runtime_version() const noexcept; - void Check(ryzenai_corelib_status status, std::string_view call) const; - void RegisterObject() const noexcept; - void Release(void* object) const noexcept; - std::size_t live_object_count() const noexcept; -}; - -class CorelibRuntime final { -public: - static std::shared_ptr GetOrCreate( - const std::filesystem::path& executable_dir); - static std::shared_ptr CreateForTest( - std::shared_ptr api); - static void ShutdownProcess(); - std::unique_lock AcquireExecution(); - const std::shared_ptr& api() const noexcept; -}; - -} // namespace flm::corelib -``` - -`UniqueObject` is move-only and calls `CorelibApi::Release` exactly once. Define `UniqueStream`, `UniqueTensor`, `UniqueTensorWindow`, `UniqueMatMulWeights`, `UniqueSsMlpWeights`, and `UniqueRmsNormWeights`; each successful C create is wrapped immediately. - -```cpp -namespace flm::phi4 { - -inline constexpr std::int64_t kLayerCount = 32; -inline constexpr std::int64_t kHiddenSize = 3072; -inline constexpr std::int64_t kIntermediateSize = 8192; -inline constexpr std::int64_t kQueryHeadCount = 24; -inline constexpr std::int64_t kKvHeadCount = 8; -inline constexpr std::int64_t kHeadSize = 128; -inline constexpr std::int64_t kQueryDimension = 3072; -inline constexpr std::int64_t kKvDimension = 1024; -inline constexpr std::int64_t kVocabularySize = 200064; -inline constexpr std::int64_t kRopeDimension = 96; -inline constexpr std::int64_t kMaxSequenceLength = 4096; -inline constexpr std::int64_t kMaxDecodeWindow = 4095; -inline constexpr std::uint32_t kRequantizedGroupSize = 64; -inline constexpr float kRmsEpsilon = 1.0e-5f; - -struct TensorView { - std::string_view name; - std::span bytes; - std::vector logical_shape; - std::uint32_t ggml_type; -}; - -struct FloatTensorView { - std::string_view name; - std::span values; - std::vector logical_shape; -}; - -struct ProjectionViews { - // AttentionQkv: q, k, v in indices 0,1,2 and count == 3. - // GateUp: gate, up in indices 0,1 and count == 2. - std::array values; - std::size_t count; -}; - -struct GgufPhi4Metadata { - std::string architecture; - std::uint64_t layer_count; - std::uint64_t hidden_size; - std::uint64_t intermediate_size; - std::uint64_t attention_head_count; - std::uint64_t kv_head_count; - std::uint64_t context_length; - std::uint64_t rope_dimension_count; - double rope_frequency_base; - double rope_attention_factor; - std::uint64_t rope_original_context_length; - std::uint64_t tokenizer_vocabulary_size; - bool add_bos_token; -}; - -class Phi4GgufPackage final { -public: - static std::shared_ptr Open( - const std::filesystem::path& gguf_path); - TensorView RequireQ8( - std::string_view name, - std::span expected_shape) const; - FloatTensorView RequireF32( - std::string_view name, - std::span expected_shape) const; - ProjectionViews AttentionQkv(std::size_t layer) const; - ProjectionViews GateUp(std::size_t layer) const; - GgufPhi4Metadata Metadata() const; - void ValidatePhi4Contract( - const nlohmann::json& config, - const nlohmann::json& tokenizer, - const nlohmann::json& tokenizer_config) const; -}; - -struct RopeTables { - std::vector cosine; - std::vector sine; -}; - -std::vector DecodeEmbeddingRowsQ8( - const TensorView& embedding, - std::span token_ids); -std::vector ConvertF32ToBf16( - std::span values); -RopeTables BuildShortRopeTables( - const GgufPhi4Metadata& metadata, - std::optional short_factors); - -struct Phi4RowExtents { - std::int64_t query_rows; - std::int64_t kv_rows; - std::int64_t output_rows; - std::int64_t ssmlp_rows; - std::int64_t rmsnorm_rows; - std::int64_t flat_mha_rows; -}; - -class Phi4ShapePlan final { -public: - static Phi4ShapePlan Build( - const std::shared_ptr& api); - const Phi4RowExtents& ForRows(std::size_t live_rows) const; - const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; - const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; -}; - -class phi4_corelib_aie4 final : public causal_lm { -public: - phi4_corelib_aie4( - LM_Config config, - std::shared_ptr package, - std::shared_ptr runtime, - std::uint32_t max_length = 4096); - buffer forward(int id) override; - buffer prefill(std::vector& ids, void* payload = nullptr) override; - void set_context_length(int length) override; - void load_weights(Q4NX&) override; - void update_max_length(std::uint32_t max_length) override; - void clear_context() override; - buffer get_k_cache(int layer, int index) override; - buffer get_v_cache(int layer, int index) override; - int get_current_context_length() override; - int checkpoint() override; - int restore() override; - bool poisoned() const noexcept; -}; - -} // namespace flm::phi4 -``` - -Frontend additions: - -```cpp -class ModelRequestError final : public std::runtime_error { -public: - ModelRequestError(int http_code, bool session_cleared, std::string message); - int http_code() const noexcept; - bool session_cleared() const noexcept; -}; - -struct lm_uniform_input_t { - // existing members remain unchanged - std::optional requested_max_new_tokens; -}; - -class AutoModel { -public: - virtual bool uses_corelib_aie4() const noexcept { return false; } -protected: - void _shared_initialize_model_state( - std::string model_path, json model_info, int context_length); - void _shared_initialize_legacy_npu(bool enable_preemption); -}; -``` - -Downloader additions: - -```cpp -namespace download_utils { -enum class HashAlgorithm { Sha256, GitBlobSha1 }; -struct DownloadRequest { - std::string url; - std::filesystem::path destination; - std::uint64_t expected_size; - HashAlgorithm hash_algorithm; - std::string expected_hash; -}; -bool download_file_atomic( - const DownloadRequest& request, - std::function progress_cb = nullptr); -} - -struct ModelFileSource { - std::string url; - std::string revision; -}; - -ModelFileSource resolve_file_source( - const nlohmann::json& model_info, - std::string_view filename, - bool use_modelscope); -``` - -## File Map - -### New production files - -- `src/include/corelib/corelib_api.hpp` — ABI 0.3.0 function table and typed errors. -- `src/include/corelib/corelib_object.hpp` — move-only ownership for every resolved object type. -- `src/include/corelib/corelib_runtime.hpp` — lazy process runtime, dependency/device validation, and execution mutex. -- `src/common/corelib/corelib_api.cpp` — safe DLL lookup, version-first symbol resolution, and error copying. -- `src/common/corelib/corelib_runtime.cpp` — singleton lifecycle and cleanup. -- `src/common/corelib/corelib_sources.cmake` — one source list shared by product and tests. -- `src/include/models/phi4/phi4_corelib_constants.hpp` — fixed validated architecture and group-64 constants. -- `src/include/models/phi4/phi4_corelib_gguf.hpp` — mapped GGUF views and model-contract API. -- `src/common/corelib/phi4_corelib_gguf.cpp` — checked GGUF v3 parser, tensor mapping/splitting, cross-source validation. -- `src/include/models/phi4/phi4_corelib_shape_plan.hpp` — helper-derived row extents and attention descriptor. -- `src/common/corelib/phi4_corelib_shape_plan.cpp` — queries and caches every required padded extent. -- `src/include/models/phi4/phi4_corelib_host.hpp` — lazy embedding decode, BF16 conversion, and RoPE derivation. -- `src/common/corelib/phi4_corelib_host.cpp` — bounds-safe host utilities only. -- `src/include/models/phi4/phi4_corelib_aie4.hpp` — causal-LM engine and poisoned-state surface. -- `src/common/corelib/phi4_corelib_aie4.cpp` — serial weight creation, persistent tensors/caches, prefill/decode sequencing. - -### New test/support files - -- `src/test/phi4_corelib_aie4/CMakeLists.txt` — standalone host/fake/real-DLL suite plus feature-on/off compile checks. -- `src/test/phi4_corelib_aie4/test_support.hpp` — `CHECK`, exception-message assertion, temporary-directory helpers. -- `src/test/phi4_corelib_aie4/gguf_fixture.hpp` — deterministic GGUF v3 byte builder with corruption controls. -- `src/test/phi4_corelib_aie4/fake_corelib.hpp` -- `src/test/phi4_corelib_aie4/fake_corelib.cpp` — complete fake of every resolved ABI 0.3.0 symbol and call recorder. -- `src/test/phi4_corelib_aie4/test_corelib_api.cpp` -- `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` -- `src/test/phi4_corelib_aie4/test_phi4_host.cpp` -- `src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp` -- `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` -- `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` -- `src/test/phi4_corelib_aie4/test_model_downloader.cpp` -- `src/test/phi4_corelib_aie4/test_real_corelib.cpp` -- `src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1` — repeatable CLI/REST/cancellation/boundary/load-cycle runner and JSON evidence writer. - -### Existing files to modify - -- `src/CMakeLists.txt` — optional corelib target; no import-library link and no packaging changes. -- `src/CMakePresets.json` — `windows-aie4` configure/build presets using the include-dir environment variable. -- `src/include/AutoModel/automodel.hpp` -- `src/common/AutoModel/automodel.cpp` — split generic model initialization from legacy NPU2 initialization; typed request errors and generation budget. -- `src/include/AutoModel/modeling_phi4.hpp` -- `src/common/AutoModel/modeling_phi4.cpp` — explicit routing, cross-source validation, capacity/cancellation/poison policy. -- `src/pull/download_model.hpp` -- `src/pull/download_model.cpp` — resumable `.part` transfer, size/hash verification, atomic promotion. -- `src/pull/model_downloader.hpp` -- `src/pull/model_downloader.cpp` — per-file source resolution and shared pull/check records. -- `src/model_list.json` — one new tag and `file_sources` schema instance. -- `src/model_info.json` — four immutable size/hash records. -- `src/runner/runner.cpp` — pass CLI generation budget and preserve poisoned-model errors. -- `src/server/rest_handler.cpp` — pass endpoint generation budgets/cancellation and emit typed 400/500 errors. -- `src/server/server.hpp` -- `src/server/server.cpp` — include all generation routes in the existing process-wide queue and release request ownership exactly once. -- `src/src/main.cpp` — feature-gated healthy corelib shutdown only; no startup load. -- `docs/docs/models/phi.md` — AIE4 tag, setup, limits, lossy conversion, and no-fallback behavior. -- `docs/docs/benchmarks/phi4_results.md` — pinned revisions and real-hardware acceptance results. - -## Task Ordering - -Tasks are strictly sequential. Tasks 1–5 are product commits and each must compile before the next starts. Task 6 is the integrated test commit. Task 7 runs real hardware acceptance and records documentation. Do not split this design into another PR. - -### Task 1: Optional Dynamic Corelib 0.3.0 Runtime - -**Files:** -- Create: `src/include/corelib/corelib_api.hpp` -- Create: `src/include/corelib/corelib_object.hpp` -- Create: `src/include/corelib/corelib_runtime.hpp` -- Create: `src/common/corelib/corelib_api.cpp` -- Create: `src/common/corelib/corelib_runtime.cpp` -- Create: `src/common/corelib/corelib_sources.cmake` -- Create: `src/test/phi4_corelib_aie4/CMakeLists.txt` -- Create: `src/test/phi4_corelib_aie4/test_support.hpp` -- Create: `src/test/phi4_corelib_aie4/fake_corelib.hpp` -- Create: `src/test/phi4_corelib_aie4/fake_corelib.cpp` -- Create: `src/test/phi4_corelib_aie4/test_corelib_api.cpp` -- Create: `src/test/phi4_corelib_aie4/test_real_corelib.cpp` -- Modify: `src/CMakeLists.txt` around options, source collection, and `flm` linkage -- Modify: `src/CMakePresets.json` configure/build preset arrays -- Modify: `src/src/main.cpp` include block and normal shutdown path - -**Interfaces:** -- Produces the `flm::corelib` interfaces in **Locked Interfaces**. -- Resolves only the 26 function pointers listed in `CorelibFunctions` above. -- `CorelibRuntime::AcquireExecution()` is the process-wide serialization primitive consumed by Task 3. -- `CorelibApi::ResolveForTest` and `CreateForTest` are test-only dependency injection; production always uses `Load`/`GetOrCreate`. - -- [ ] **Step 1: Write failing ABI, path, lifetime, and feature-gate tests** - -In `test_corelib_api.cpp`, define and invoke these named cases from `main()`: - -```cpp -TestVersionIsResolvedBeforeEveryOtherSymbol(); -TestExactlyVersion030IsAccepted(); -TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions(); -TestEveryRequiredSymbolIsResolvedExactlyOnce(); -TestMissingSymbolNamesTheSymbolAndUnloadsTheDll(); -TestCorelibErrorCopiesStatusCallAndThreadLocalDetail(); -TestEnvironmentPathMustBeAnAbsoluteDllPath(); -TestEnvironmentPathWinsOverExecutableRelativePath(); -TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory(); -TestEveryUniqueObjectReleasesExactlyOnceAfterMoves(); -TestRuntimeRunsDependencySelftestAndRequiresDeviceContext(); -TestExecutionLeaseSerializesTwoThreads(); -TestCleanupRunsAfterTheLastObjectAndOnlyOnce(); -``` - -The fake must export all 26 required symbols, let each status/detail/version/device result be injected, record resolution order, count live objects/releases, and record maximum simultaneous execution leases. In `test_real_corelib.cpp`, return CTest skip code 77 only when `FLM_AIE4_CORELIB_PATH` is unset; if it is set, assert ABI 0.3.0, every symbol, dependency self-test, and device context. - -Add two object-library compile guards in the test CMake project: one builds the production frontend/CMake source list without `FLM_ENABLE_CORELIB_AIE4` and no corelib include path; the other builds with the define and pinned include path. - -- [ ] **Step 2: Run RED checks** - -```powershell -cmake -S src/test/phi4_corelib_aie4 -B src/build/phi4-corelib-tests ` - -G "Visual Studio 17 2022" -A x64 ` - -DRYZENAI_CORELIB_INCLUDE_DIR=C:/Users/chiz/work/ryzenai-corelib/include -cmake --build src/build/phi4-corelib-tests --config Release --target test_corelib_api -``` - -Expected: configure or compile fails because the new adapter/runtime headers and sources do not exist. - -- [ ] **Step 3: Implement the minimal dynamic adapter and RAII layer** - -Implement version-first resolution: resolve and call `ryzenai_corelib_get_version`, reject anything other than `0.3.0`, then resolve the remaining 26 names. `Check` must copy `get_last_error_message()` before calling `status_to_string()`. Load with: - -```cpp -LoadLibraryExW(path.c_str(), nullptr, - LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); -``` - -`FLM_AIE4_CORELIB_PATH` must be an absolute file path. With it unset, return `absolute(executable_dir / "aie4" / "ryzenai_corelib.dll")`; do not call `LoadLibraryW` with a bare filename. Wrap every successful object immediately in the matching `UniqueObject`. - -`CorelibRuntime::GetOrCreate` must be lazy and process-wide. `CreateForTest` runs `selftest_dependencies`, then `has_device_context`, and rejects either failure before reporting ready. `ShutdownProcess` waits for the execution mutex, requires no live objects, calls `cleanup` once, then drops the API/module. - -- [ ] **Step 4: Integrate the feature-gated build** - -Add: - -```cmake -option(FLM_ENABLE_CORELIB_AIE4 - "Enable Phi-4 Q8_0 GGUF execution through ryzenai-corelib" OFF) -if(FLM_ENABLE_CORELIB_AIE4) - if(NOT WIN32) - message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows") - endif() - find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) - include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake") - add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES}) - target_include_directories(flm_corelib_aie4 PUBLIC - "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") - target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1) - target_link_libraries(flm PRIVATE flm_corelib_aie4) -endif() -``` - -Do not add `ryzenai_corelib.lib`, runtime-copy commands, or installer rules. Add `windows-aie4` presets inheriting `windows-default`, using binary directory `${sourceDir}/build-aie4`, `FLM_ENABLE_CORELIB_AIE4=ON`, and `RYZENAI_CORELIB_INCLUDE_DIR=$env{RYZENAI_CORELIB_INCLUDE_DIR}`. - -Guard the `main.cpp` include and final `CorelibRuntime::ShutdownProcess()` call with `FLM_ENABLE_CORELIB_AIE4`; do not touch startup, `pull`, `list`, or non-AIE4 command paths. - -- [ ] **Step 5: Run GREEN checks and the default-build regression gate** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_corelib_api -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_corelib_api$" --output-on-failure -cmake --preset windows-default -S src -cmake --build --preset windows-default -$env:RYZENAI_CORELIB_INCLUDE_DIR='C:/Users/chiz/work/ryzenai-corelib/include' -cmake --preset windows-aie4 -S src -cmake --build --preset windows-aie4 -``` - -Expected: `test_corelib_api` passes; both builds produce the normal `flm.exe`; `dumpbin /imports src/build-aie4/Release/flm.exe | findstr /i ryzenai_corelib` prints no import. - -- [ ] **Step 6: Refactor only duplicated resolver/RAII mechanics and rerun Step 5** - -Keep symbol names in one constexpr table or one macro expansion so the function table, resolver, and fake cannot drift. Do not introduce a generic plugin framework. - -- [ ] **Step 7: Commit review gate** - -```powershell -git add src/CMakeLists.txt src/CMakePresets.json src/src/main.cpp ` - src/include/corelib src/common/corelib/corelib_api.cpp ` - src/common/corelib/corelib_runtime.cpp src/common/corelib/corelib_sources.cmake ` - src/test/phi4_corelib_aie4 -git commit -m "build: add optional dynamic corelib 0.3.0 runtime" -``` - -Expected: one build/runtime commit; no product model route exists yet. - -### Task 2: Validated Phi-4 GGUF v3 Package - -**Files:** -- Create: `src/include/models/phi4/phi4_corelib_constants.hpp` -- Create: `src/include/models/phi4/phi4_corelib_gguf.hpp` -- Create: `src/common/corelib/phi4_corelib_gguf.cpp` -- Create: `src/test/phi4_corelib_aie4/gguf_fixture.hpp` -- Create: `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` -- Modify: `src/common/corelib/corelib_sources.cmake` -- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` - -**Interfaces:** -- Produces `TensorView`, `FloatTensorView`, `ProjectionViews`, `GgufPhi4Metadata`, and `Phi4GgufPackage` exactly as declared in **Locked Interfaces**. -- `AttentionQkv(layer)` returns `count == 3`, ordered Q/K/V. `GateUp(layer)` returns `count == 2`, ordered gate/up. -- Task 3 consumes the returned byte spans directly in corelib GGUF component structs and retains the package for the engine lifetime. - -- [ ] **Step 1: Write a deterministic GGUF v3 fixture builder** - -`gguf_fixture.hpp` must write little-endian scalar/string/array metadata, tensor directory entries, configurable alignment, and aligned payloads. It must expose mutations for a truncated string/directory, count/product overflow, zero/non-power-of-two alignment, duplicate names, out-of-file ranges, overlapping ranges, unsupported metadata types, dtype mismatch, shape mismatch, and payload-length mismatch. - -Use exact fixture tensors: - -```cpp -"token_embd.weight" logical [200064, 3072] Q8_0 -"output_norm.weight" logical [3072] F32 -"blk.0.attn_norm.weight" logical [3072] F32 -"blk.0.ffn_norm.weight" logical [3072] F32 -"blk.0.attn_qkv.weight" logical [5120, 3072] Q8_0 -"blk.0.attn_output.weight" logical [3072, 3072] Q8_0 -"blk.0.ffn_up.weight" logical [16384, 3072] Q8_0 -"blk.0.ffn_down.weight" logical [3072, 8192] Q8_0 -"rope_factors_short.weight" logical [48] F32 -``` - -The fixture may use reduced payload backing for parser-only tests only when its declared dimensions are also reduced; contract tests use directory-only synthetic spans sized with checked Q8_0 arithmetic and a sparse temporary file. - -- [ ] **Step 2: Write failing parser and corruption tests** - -Define and invoke these cases: - -```cpp -TestValidV3HeaderMetadataDirectoryAndAlignment(); -TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely(); -TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail(); -TestCountProductAlignmentAndOffsetOverflowFail(); -TestZeroAndNonPowerOfTwoAlignmentFail(); -TestDuplicateTensorNamesFail(); -TestOutOfFileAndOverlappingTensorRangesFail(); -TestUnsupportedUnskippableMetadataTypeFails(); -TestRequireQ8AndRequireF32ReportNameActualAndExpected(); -TestAttentionQkvReturnsThreeZeroCopyWholeRowViews(); -TestGateUpReturnsTwoZeroCopyWholeRowViews(); -TestSplitRejectsNonIntegralQ8RowBoundary(); -TestViewsPointIntoTheReadOnlyMapping(); -``` - -For split checks, assert Q8_0 row bytes are `input_width / 32 * 34`; Q/K/V byte offsets are 0, `3072 * row_bytes`, and `4096 * row_bytes`; gate/up offsets are 0 and `8192 * row_bytes`. - -- [ ] **Step 3: Run RED** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_gguf -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_gguf$" --output-on-failure -``` - -Expected: compile fails because `Phi4GgufPackage` is not defined. - -- [ ] **Step 4: Implement checked mapping and parsing** - -Map with `CreateFileW(..., GENERIC_READ, FILE_SHARE_READ, ..., OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, ...)`, `CreateFileMappingW(..., PAGE_READONLY, ...)`, and `MapViewOfFile(..., FILE_MAP_READ, ...)`. Parse magic `0x46554747`, require version 3, and use helper functions equivalent to: - -```cpp -std::uint64_t CheckedAdd(std::uint64_t a, std::uint64_t b, std::string_view field); -std::uint64_t CheckedMultiply(std::uint64_t a, std::uint64_t b, std::string_view field); -std::uint64_t AlignUp(std::uint64_t value, std::uint64_t alignment); -std::span RequireRange( - std::span file, std::uint64_t offset, - std::uint64_t length, std::string_view field); -``` - -Retain only contract metadata while safely skipping every encoded metadata value used by the pinned file. The retained GGUF keys are `general.architecture`, `general.alignment`, `phi3.block_count`, `phi3.context_length`, `phi3.embedding_length`, `phi3.feed_forward_length`, `phi3.attention.head_count`, `phi3.attention.head_count_kv`, `phi3.attention.layer_norm_rms_epsilon`, `phi3.rope.dimension_count`, `phi3.rope.freq_base`, `phi3.rope.scaling.attn_factor`, `phi3.rope.scaling.original_context_length`, `tokenizer.ggml.tokens` (array count only), and `tokenizer.ggml.add_bos_token`. Compute Q8_0 bytes as `elements / 32 * 34` only after requiring divisibility by 32; compute F32 bytes as `elements * 4`. Reverse GGUF dimensions into logical row-major shapes at the model boundary. Sort absolute tensor ranges and reject overlap. - -- [ ] **Step 5: Write failing full-model contract tests** - -Define table-driven mutations for every fixed field and every required tensor across all 32 layer names. Each assertion must check the thrown text contains the field/tensor, the actual value, and the expected value. Include these independent cases: - -```cpp -TestAcceptsExactPhi3Phi4Contract(); -TestRejectsWrongArchitectureAndEveryDimension(); -TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole(); -TestRejectsMixedQuantizationAndOutputWeightPresence(); -TestRequiresTiedQ8TokenEmbeddingAsLmHead(); -TestRequiresOriginal4096WindowAndRejectsLongRopeBranch(); -TestValidatesOptionalShortRopeFactorsAsF32Length48(); -TestRejectsNonFiniteOrNonPositiveRopeValues(); -TestRejectsConfigDisagreement(); -TestDerivesStopSetFromGgufConfigAndTokenizerIds(); -TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement(); -TestValidationCreatesNoCorelibObjects(); -``` - -The config checks are `model_type == "phi3"`, `num_hidden_layers`, `hidden_size`, `intermediate_size`, `num_attention_heads`, `num_key_value_heads`, `head_dim`, `vocab_size`, `rms_norm_eps`, and `original_max_position_embeddings`. Determine tokenizer vocabulary size from `tokenizer.json`'s model vocabulary plus added-token IDs without assuming contiguous object iteration; compare the maximum assigned ID plus one and the distinct ID count to 200064. - -- [ ] **Step 6: Implement `ValidatePhi4Contract` and make all tests green** - -Perform intrinsic GGUF validation in `Open`; perform config/tokenizer/GGUF comparison in `ValidatePhi4Contract`. Validation must complete before any Task 3 engine constructor invokes a corelib create call. - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_gguf -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_gguf$" --output-on-failure -``` - -Expected: all parser, split, corruption, and contract cases pass. - -- [ ] **Step 7: Refactor parser helpers and rerun Step 6** - -Keep cursor advancement, checked arithmetic, range validation, and error formatting in single helpers; keep Phi-4 tensor names in the package adapter rather than a generic GGUF layer. Do not broaden accepted GGUF types or architectures. - -- [ ] **Step 8: Commit review gate** - -```powershell -git add src/include/models/phi4/phi4_corelib_constants.hpp ` - src/include/models/phi4/phi4_corelib_gguf.hpp ` - src/common/corelib/phi4_corelib_gguf.cpp ` - src/common/corelib/corelib_sources.cmake ` - src/test/phi4_corelib_aie4/gguf_fixture.hpp ` - src/test/phi4_corelib_aie4/test_phi4_gguf.cpp ` - src/test/phi4_corelib_aie4/CMakeLists.txt -git commit -m "feat: add validated Phi-4 Q8_0 GGUF reader" -``` - -Expected: the commit parses and validates but cannot execute a model. - -### Task 3: Corelib-Backed Phi-4 AIE4 Engine - -**Files:** -- Create: `src/include/models/phi4/phi4_corelib_shape_plan.hpp` -- Create: `src/common/corelib/phi4_corelib_shape_plan.cpp` -- Create: `src/include/models/phi4/phi4_corelib_host.hpp` -- Create: `src/common/corelib/phi4_corelib_host.cpp` -- Create: `src/include/models/phi4/phi4_corelib_aie4.hpp` -- Create: `src/common/corelib/phi4_corelib_aie4.cpp` -- Create: `src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp` -- Create: `src/test/phi4_corelib_aie4/test_phi4_host.cpp` -- Create: `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` -- Modify: `src/test/phi4_corelib_aie4/fake_corelib.hpp` -- Modify: `src/test/phi4_corelib_aie4/fake_corelib.cpp` -- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` -- Modify: `src/common/corelib/corelib_sources.cmake` - -**Interfaces:** -- Consumes `Phi4GgufPackage`, `CorelibRuntime`, the exact ABI function table, and the fixed constants. -- Produces `DecodeEmbeddingRowsQ8`, `ConvertF32ToBf16`, `BuildShortRopeTables`, `Phi4ShapePlan`, and `phi4_corelib_aie4` from **Locked Interfaces**. -- `Phi4ShapePlan::Build(std::shared_ptr)` caches helper results for live rows 1 through 4096 and rejects any padded K/N change. -- The engine stores the shared GGUF package so lazy embedding spans remain valid for its full lifetime. - -- [ ] **Step 1: Write failing host conversion tests** - -Use known Q8_0 blocks, including negative signed codes and FP16 scales, and define: - -```cpp -TestLazyEmbeddingDecodesOnlyRequestedRows(); -TestLazyEmbeddingPreservesRequestOrderAndDuplicates(); -TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds(); -TestF32ToBf16UsesRoundToNearestEven(); -TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs(); -TestRopeTablesApplyShortFactorsAndAttentionFactor(); -TestRopeTablesHaveShape4096By48(); -``` - -Assert sentinel bytes in unrequested embedding rows are never read by using a guarded fixture mapping. For RoPE, compare position 4095 against a double-precision scalar reference; a float-only implementation must fail the tolerance. - -- [ ] **Step 2: Implement only the host utilities and run them** - -Q8_0 row decode is `value = fp16_scale * int8_code` for each 34-byte block. Reject malformed row lengths before decoding. Build inverse frequencies as: - -```cpp -inv_freq[i] = 1.0 / - (std::pow(freq_base, (2.0 * i) / 96.0) * short_factor[i]); -cos[p * 48 + i] = static_cast(std::cos(p * inv_freq[i]) * attn_factor); -sin[p * 48 + i] = static_cast(std::sin(p * inv_freq[i]) * attn_factor); -``` - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_host -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_host$" --output-on-failure -``` - -Expected: all seven host tests pass without loading a DLL. - -- [ ] **Step 3: Write failing shape-plan tests** - -The fake helper API must record every argument and return configurable padded rows. Define: - -```cpp -TestShapePlanQueriesRows1Through4096AtGroup64(); -TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions(); -TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96(); -TestShapePlanRejectsPaddedKOrNChanges(); -TestShapePlanRejectsRowsOutsideCachedRange(); -TestShapePlanFailureNamesHelperAndLogicalShape(); -``` - -- [ ] **Step 4: Implement the shape plan and run it** - -Cache transition vectors for query projection `[M,3072]x[3072,3072]`, KV projection `[M,3072]x[3072,1024]`, output projection `[M,3072]x[3072,3072]`, SSMLP `(M,3072,8192,64)`, RMSNorm `(M,3072)`, flat-MHA descriptor `(24,8,128,4096,96)`, and LM head `[1,3072]x[3072,200064]`. Every allocation uses the maximum helper-returned extent, never a hand-rounded M. - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_phi4_shape_plan -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_phi4_shape_plan$" --output-on-failure -``` - -Expected: all shape-plan tests pass and every fake helper observation uses group 64. - -- [ ] **Step 5: Extend the fake corelib and write failing engine-load tests** - -The fake must implement tensors with shape/dtype/storage, tensor windows retaining parent storage, Q8_0 weight creation records, RMSNorm weight records, stream dispatch records, injected pre-submit/post-submit/synchronize failures, and an in-flight flag. Define: - -```cpp -TestEngineCreatesOneStreamAndPersistentHelperSizedTensors(); -TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight(); -TestEveryProjectionUsesQ8RequantizedGroup64Threads0(); -TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate(); -TestQkvAndGateUpPointersMatchExactMappedSubranges(); -TestNormsAndEpsilonReachCorelibAsBf16(); -TestEmbeddingMappingOutlivesAllLazyRowReads(); -TestNoDeviceObjectExistsWhenPackageValidationFails(); -``` - -Require all 162 weight creates to happen in deterministic layer order; this makes accidental parallelization visible and protects the documented all-zero-output mitigation. - -- [ ] **Step 6: Implement engine construction and serial weight creation** - -Create initial RMSNorm weights from `blk.0.attn_norm.weight`. For each layer create Q/K/V/O matmul weights and one SSMLP weight whose `norm0` is `blk.i.ffn_norm.weight` and whose `norm1` is `blk.(i+1).attn_norm.weight`, except layer 31 uses `output_norm.weight`. Create the LM-head matmul from `token_embd.weight`. Every GGUF component type is Q8_0 and every descriptor group is 64. - -Allocate once: hidden/residual/skip, Q, K, attention output, one-row LM input, logits, FP32 cosine/sine tables, and 32 K plus 32 V caches `[8,4096,128]`. Upload RoPE once. Do not materialize the embedding table. - -- [ ] **Step 7: Write failing prefill/decode/sequencing tests** - -Define and invoke: - -```cpp -TestPrefillDecodesEmbeddingRowsAndAdvancesPosition(); -TestDecodeUsesOneRowAndAdvancesPosition(); -TestVProjectionWritesWindowAtPositionTimes128(); -TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream(); -TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket(); -TestForwardSynchronizesBeforeHostReadAndLmHeadRead(); -TestKVCachesRemainFixedAt8By4096By128(); -TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken(); -TestTotalDecodeWindowStopsAt4095(); -TestClearContextResetsLogicalPositionWithoutRecreatingWeights(); -TestCheckpointRestoreChangesOnlyLogicalPosition(); -TestPreSubmitFailureIsRecoverable(); -TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState(); -TestSynchronizeFailurePoisonsAndClearsState(); -TestPoisonedInstanceRejectsEveryLaterEntryPoint(); -TestCancellationBoundaryLeavesNoOutstandingFakeWork(); -``` - -The fake call log must prove there is no CPU/NPU2 call and no lossless `*_create_gguf` call. The V-window assertion uses shape `[8,4096-position,128]` and element offset `position * 128`. - -- [ ] **Step 8: Implement the minimal execution state machine** - -At the beginning of each model step, reject a poisoned engine and validate IDs/capacity. Decode only requested embedding rows to FP32, write and zero helper-required input/residual extents, dispatch RMSNorm in place, then for each layer dispatch Q, K, V-to-window, flat-MHA, O, and SSMLP on the same stream. Swap residual/skip handles only after queueing SSMLP. Synchronize before reading the final hidden row, write it to the one-row LM input, dispatch LM head, synchronize, and read logits as BF16 into the existing `buffer` expected by `Sampler`. - -Track whether any submit succeeded. On a pre-submit failure, leave `poisoned_ == false`. On any later exception, best-effort synchronize, set `poisoned_ = true`, clear logical position/checkpoint, and throw an error that includes the failed corelib call. `clear_context()` must not clear `poisoned_`; only destroying/recreating the model does. - -- [ ] **Step 9: Run GREEN and leak/order checks** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target ` - test_phi4_host test_phi4_shape_plan test_phi4_engine -ctest --test-dir src/build/phi4-corelib-tests -C Release ` - -R "^test_phi4_(host|shape_plan|engine)$" --output-on-failure -``` - -Expected: all tests pass; fake live-object count returns to zero after engine destruction; maximum simultaneous requantized creates is one; no submitted work remains when a request/cancellation test returns. - -- [ ] **Step 10: Refactor repeated create/dispatch/error wrappers and rerun Step 9** - -Keep model policy in `phi4_corelib_aie4.cpp`; keep GGUF parsing out of the engine and corelib calls out of the GGUF package/host helpers. - -- [ ] **Step 11: Commit review gate** - -```powershell -git add src/include/models/phi4/phi4_corelib_shape_plan.hpp ` - src/include/models/phi4/phi4_corelib_host.hpp ` - src/include/models/phi4/phi4_corelib_aie4.hpp ` - src/common/corelib/phi4_corelib_shape_plan.cpp ` - src/common/corelib/phi4_corelib_host.cpp ` - src/common/corelib/phi4_corelib_aie4.cpp ` - src/common/corelib/corelib_sources.cmake ` - src/test/phi4_corelib_aie4 -git commit -m "feat: add corelib-backed Phi-4 AIE4 engine" -``` - -Expected: engine/fake tests pass; no CLI/catalog route selects it yet. - -### Task 4: Explicit Phi-4 Frontend Routing and Request Lifecycle - -**Files:** -- Modify: `src/include/AutoModel/automodel.hpp` -- Modify: `src/common/AutoModel/automodel.cpp` -- Modify: `src/include/AutoModel/modeling_phi4.hpp` -- Modify: `src/common/AutoModel/modeling_phi4.cpp` -- Modify: `src/runner/runner.cpp` -- Modify: `src/server/rest_handler.cpp` -- Modify: `src/server/server.hpp` -- Modify: `src/server/server.cpp` -- Create: `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` -- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` - -**Interfaces:** -- Consumes the package/runtime/engine from Tasks 1–3. -- Produces `ModelRequestError`, `lm_uniform_input_t::requested_max_new_tokens`, and `AutoModel::uses_corelib_aie4()` from **Locked Interfaces**. -- `Phi4::load_model` recognizes exactly `corelib_aie4_gguf`; no backend field remains NPU2. - -- [ ] **Step 1: Write failing routing and initialization tests** - -Inject an engine factory under `FLM_CORELIB_TESTING` and define: - -```cpp -TestAbsentBackendStillBuildsQ4nxPhi4Npu(); -TestCorelibAie4GgufBuildsOnlyTheCorelibEngine(); -TestUnknownAndNonStringBackendAreErrors(); -TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders(); -TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation(); -TestMissingCorelibFailsOnlyWhenAie4ModelLoads(); -TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure(); -TestPreemptionIsRejectedForTheAie4Route(); -``` - -Split `_shared_load_model` so the test can assert the AIE4 branch initializes config/tokenizer/sampler state without constructing `npu_xclbin_manager`; leave `_shared_load_model` behavior unchanged for every legacy caller. - -- [ ] **Step 2: Implement explicit routing and tokenizer contract checks** - -`Phi4::load_model` must: - -1. resolve the backend string; -2. for AIE4, reject preemption and context outside `1..4096`; -3. parse `config.json`, `tokenizer.json`, and `tokenizer_config.json` and open the single GGUF; -4. call `ValidatePhi4Contract` before `CorelibRuntime::GetOrCreate` or any engine/device creation; -5. initialize the existing `Tokenizer`, chat template, sampler, and EOS list `{200020, 199999}` after proving `tokenizer.json` maps `<|end|>`/`<|endoftext|>` to those IDs, GGUF declares EOS 200020, `config.json` declares EOS 199999, and `tokenizer_config.json` disables automatic BOS; -6. lazily acquire runtime and construct `phi4_corelib_aie4`; -7. set `uses_corelib_aie4_` only after all construction succeeds. - -Use `Phi-4-mini-instruct.Q8_0.gguf` as the only accepted model filename. The feature-off branch throws `This binary was built without Phi-4 AIE4 corelib support`; it must not attempt Q4NX. - -- [ ] **Step 3: Write failing budget, cancellation, poison, and generation tests** - -```cpp -TestRenderedPromptPlusExplicitBudgetMayEqual4095(); -TestRenderedPromptPlusExplicitBudgetAbove4095Is400(); -TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow(); -TestCancellationBeforePrefillSubmitsNothing(); -TestCancellationBetweenDecodeStepsStopsWithCancelReason(); -TestCancellationReturnsOnlyAfterSynchronize(); -TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned(); -TestPoisonedModelReturns500UntilReload(); -TestEosSelfTerminatesWithoutAnExtraDecode(); -TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics(); -``` - -Use `ModelRequestError(400, false, ...)` for admission failures and `ModelRequestError(500, true, ...)` for inference failures that clear state. A poisoned refusal is 500 with `session_cleared == true` and names that unload/reload is required. - -- [ ] **Step 4: Implement AIE4-specific insert/generate behavior** - -Pass `requested_max_new_tokens` into `lm_uniform_input_t` from CLI `generate_limit`, `/api/generate:max_tokens`, `/api/chat:options.num_predict`, `/v1/chat/completions:max_tokens|max_completion_tokens`, and `/v1/completions:max_tokens`. Normalize absent or non-positive sentinel limits to an unbounded request, then cap generation to `4095 - rendered_prompt_tokens`; do not pass the legacy default 4096 through as an explicit AIE4 budget. - -Check cancellation immediately before prefill and before every `forward` call. Since each engine call synchronizes before returning, a cancellation observed between calls has no outstanding work. Preserve existing tokenization, chat-template rendering, sampling settings, and output streams. - -- [ ] **Step 5: Make server serialization exception-safe and complete** - -Keep the existing process-wide NPU queue. Add `/v1/completions` to `requires_npu_access`, replace duplicated release calls with a move-only completion guard owned by each dequeued request, and prove exactly one release on success, JSON parse failure, model error, cancellation, and unknown exception. Do not create a second AIE4-only queue. - -Map `ModelRequestError::http_code()` to HTTP 400 or 500 for non-streaming responses. For OpenAI streaming after headers/data started, emit one structured error event followed by `[DONE]`; before streaming starts, return the normal JSON error response. Include `session_cleared` in the error body. - -- [ ] **Step 6: Run frontend and compile-gate tests** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target ` - test_phi4_frontend phi4_frontend_compile_off phi4_frontend_compile_on -ctest --test-dir src/build/phi4-corelib-tests -C Release ` - -R "^test_phi4_frontend$" --output-on-failure -cmake --build --preset windows-default -cmake --build --preset windows-aie4 -``` - -Expected: routing/lifecycle tests pass; feature-off production TUs compile without `RYZENAI_CORELIB_INCLUDE_DIR`; both full builds succeed; legacy `phi4-mini-it:4b` still routes to Q4NX/NPU2. - -- [ ] **Step 7: Refactor shared request-finalization logic and rerun Step 6** - -Centralize only typed-error JSON construction and exactly-once queue release; retain each endpoint's existing response schema and stream formatter. Do not refactor unrelated server routes. - -- [ ] **Step 8: Commit review gate** - -```powershell -git add src/include/AutoModel/automodel.hpp ` - src/common/AutoModel/automodel.cpp ` - src/include/AutoModel/modeling_phi4.hpp ` - src/common/AutoModel/modeling_phi4.cpp ` - src/runner/runner.cpp src/server/rest_handler.cpp ` - src/server/server.hpp src/server/server.cpp ` - src/test/phi4_corelib_aie4/test_phi4_frontend.cpp ` - src/test/phi4_corelib_aie4/CMakeLists.txt -git commit -m "feat: route Phi-4 GGUF models through AIE4" -``` - -Expected: the explicit route works with synthetic/fake inputs, and no catalog model exposes it yet. - -### Task 5: Pinned Multi-Source Pull and Catalog Entry - -**Files:** -- Modify: `src/pull/download_model.hpp` -- Modify: `src/pull/download_model.cpp` -- Modify: `src/pull/model_downloader.hpp` -- Modify: `src/pull/model_downloader.cpp` -- Modify: `src/model_list.json` -- Modify: `src/model_info.json` -- Create: `src/test/phi4_corelib_aie4/test_model_downloader.cpp` -- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` - -**Interfaces:** -- Produces `DownloadRequest`, `download_file_atomic`, `ModelFileSource`, and `resolve_file_source` from **Locked Interfaces**. -- Existing catalog entries without `file_sources` retain byte-for-byte URL construction and SHA-256-for-LFS/Git-blob-SHA1-for-ordinary-file checks. -- New records may carry explicit lowercase `sha256`; when present it is authoritative regardless of LFS status. - -- [ ] **Step 1: Freeze exact remote metadata before editing the catalog** - -Query each immutable revision's Hugging Face tree API. For LFS files, require `lfs.size` and the 64-hex `lfs.oid` (the content SHA-256); for ordinary files, download the small immutable file and calculate SHA-256. Record exactly: - -```json -{ - "total_size": 4100140571, - "records": [ - {"path":"Phi-4-mini-instruct.Q8_0.gguf","size":4084611040,"sha256":"26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42"}, - {"path":"tokenizer.json","size":15524095,"sha256":"382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea"}, - {"path":"tokenizer_config.json","size":2932,"sha256":"9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db"}, - {"path":"config.json","size":2504,"sha256":"ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b"} - ] -} -``` - -Expected: API revisions equal the pinned commits; both LFS records match their `lfs` metadata; fresh downloads of the two small regular files reproduce the listed hashes. The later real `flm pull` independently hashes the complete GGUF and tokenizer payloads before promotion, so this step must not download and discard 4 GB. - -- [ ] **Step 2: Write failing URL/catalog/backward-compatibility tests** - -Define: - -```cpp -TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName(); -TestGgufUrlContainsUnslothRevisionAndFilename(); -TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename(); -TestExistingSingleSourceEntryKeepsItsCurrentUrl(); -TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail(); -TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile(); -TestModelIsReadyOnlyWhenAllFourFinalFilesValidate(); -TestPartFileNeverMakesModelReady(); -TestResumeAppendsToPartThenAtomicallyPromotes(); -TestWrongSizeOrHashNeverReplacesAValidFinalFile(); -TestInterruptedTransferKeepsPartForNextResume(); -TestSuccessfulForceDownloadAtomicallyReplacesFinalFile(); -``` - -Use a `file://` URL and a small deterministic payload for transfer tests. Pre-create the first half at `request.destination.string() + ".part"`; assert the final bytes and hash match and the `.part` file disappears only after success. - -- [ ] **Step 3: Run RED** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_model_downloader -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_model_downloader$" --output-on-failure -``` - -Expected: tests fail because `file_sources`, resumable temporary downloads, explicit SHA-256, and the catalog entry do not exist. - -- [ ] **Step 4: Implement per-file source selection without migrating existing entries** - -For an override, require exactly non-empty string `url` and 40-character hexadecimal `revision`, then produce: - -```text -{url}/resolve/{revision}/{percent-encoded filename}?download=true -``` - -For no override, execute the existing base URL/ModelScope logic unchanged. Reject `--modelscope` for this new tag with a message that pinned Hugging Face per-file sources are required; never silently swap repositories or revisions. - -Add this fixed catalog identity, then add its numeric `size` from the verified Step 1 output as described immediately below: - -```json -"phi4-mini-it-aie4": { - "4b": { - "name": "phi4-mini-it-aie4", - "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", - "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", - "default_context_length": 4096, - "max_prefill_len": 4096, - "details": { - "family": "phi4", - "think": false, - "think_toggleable": false, - "parameter_size": "4B", - "quantization_level": "Q8_0 -> AIE4 group-64", - "execution_backend": "corelib_aie4_gguf" - }, - "flm_min_version": "1.0.3", - "vlm": false, - "files": [ - "Phi-4-mini-instruct.Q8_0.gguf", - "tokenizer.json", - "tokenizer_config.json", - "config.json" - ], - "file_sources": { - "tokenizer.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - }, - "tokenizer_config.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - }, - "config.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - } - }, - "footprint": 4.1 - } -} -``` - -Add a numeric `size` field to that object using the exact `total_size` emitted in Step 1; `TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName` must compare it with the sum of the four committed records and reject zero or disagreement. Add the four emitted `{path,size,sha256}` records under `phi4-mini-it-aie4:4b` in `model_info.json`. - -- [ ] **Step 5: Implement resume, verification, and atomic promotion** - -Always transfer to `request.destination.string() + ".part"`. If that path exists and is smaller than expected, open append mode and set `CURLOPT_RESUME_FROM_LARGE` to its byte length. If it is larger, delete only that file and restart. Require the completed size and selected hash before promotion. On Windows promote with `MoveFileExW(part, destination, MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH)`; on non-Windows use same-directory `std::filesystem::rename`, whose replacement is atomic. A transfer interruption keeps `.part`; a size/hash mismatch deletes `.part`; no failure mutates an already-valid final file. - -Update both `pull_model` and `check_model` to consume the same record resolver and integrity function. After download, return success only when all four final files pass. - -- [ ] **Step 6: Run GREEN plus real pull/check smoke** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release --target test_model_downloader -ctest --test-dir src/build/phi4-corelib-tests -C Release -R "^test_model_downloader$" --output-on-failure -src/build/Release/flm.exe pull phi4-mini-it-aie4:4b -src/build/Release/flm.exe check phi4-mini-it-aie4:4b -Get-ChildItem (Join-Path $env:USERPROFILE 'Documents/flm/models/phi4-mini-it-aie4') | - Select-Object -ExpandProperty Name -``` - -Expected: tests pass; pull/check succeed; directory output is exactly the four names in the `files` array and contains no `.part`, manifest, ONNX, or converted-weight file. - -- [ ] **Step 7: Refactor source/hash selection and rerun Step 6** - -Use one resolved per-file record for download, check, and ready-state decisions. Preserve the legacy URL/hash branch intact and do not generalize the catalog beyond optional `file_sources` and optional explicit `sha256`. - -- [ ] **Step 8: Commit review gate** - -```powershell -git add src/pull/download_model.hpp src/pull/download_model.cpp ` - src/pull/model_downloader.hpp src/pull/model_downloader.cpp ` - src/model_list.json src/model_info.json ` - src/test/phi4_corelib_aie4/test_model_downloader.cpp ` - src/test/phi4_corelib_aie4/CMakeLists.txt -git commit -m "feat: pull Phi-4 GGUF and tokenizer from pinned sources" -``` - -Expected: existing catalog tests remain unchanged and the new pinned multi-source pull is complete. - -### Task 6: Integrated Offline, Build, and Real-DLL Verification - -**Files:** -- Modify: `src/test/phi4_corelib_aie4/fake_corelib.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_corelib_api.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_phi4_gguf.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_phi4_engine.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_phi4_frontend.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_model_downloader.cpp` -- Modify: `src/test/phi4_corelib_aie4/test_real_corelib.cpp` -- Modify: `src/test/phi4_corelib_aie4/CMakeLists.txt` - -**Interfaces:** -- Consumes all production interfaces; introduces no product API. -- `test_real_corelib` is skipped only when no real DLL path is configured, never for ABI/symbol/self-test/device failures. - -- [ ] **Step 1: Add cross-component regression cases before changing production code** - -Add these tests using the complete fake and synthetic GGUF package: - -```cpp -TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates(); -TestNoManifestOnnxConvertedWeightOrCachePathIsOpened(); -TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib(); -TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing(); -TestAie4SelectionWithMissingDllFailsWithoutChangingBackend(); -TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback(); -TestTwoConcurrentAie4RequestsNeverOverlapDispatch(); -TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits(); -TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable(); -``` - -The all-zero guard injects nonzero deterministic fake logits at every cycle and asserts every returned vector has at least one nonzero BF16 element. It complements, but does not replace, the real-hardware ten-cycle gate. - -- [ ] **Step 2: Run the complete offline suite and fix only integration defects** - -```powershell -cmake --build src/build/phi4-corelib-tests --config Release -ctest --test-dir src/build/phi4-corelib-tests -C Release --output-on-failure -``` - -Expected: every offline test passes; `test_real_corelib` is reported skipped when `FLM_AIE4_CORELIB_PATH` is absent. Do not weaken assertions or add production behavior not required by the spec. If an integration failure requires a product edit, return to Tasks 1–5, amend the owning product commit, rerun that task's focused gate, and then restart this task; do not hide product fixes in the test commit. - -- [ ] **Step 3: Run exact feature-off and feature-on product builds** - -```powershell -Remove-Item Env:FLM_AIE4_CORELIB_PATH -ErrorAction SilentlyContinue -cmake --preset windows-default -S src -cmake --build --preset windows-default -src/build/Release/flm.exe version - -$env:RYZENAI_CORELIB_INCLUDE_DIR='C:/Users/chiz/work/ryzenai-corelib/include' -cmake --preset windows-aie4 -S src -cmake --build --preset windows-aie4 -src/build-aie4/Release/flm.exe version -src/build-aie4/Release/flm.exe list -``` - -Expected: both binaries start; the AIE4-enabled binary runs `version` and `list` without loading corelib; the default binary contains no corelib import. - -- [ ] **Step 4: Run real-DLL integration against the pinned installation** - -First prove the sibling checkout and header are the requested revision rather than trusting its current branch name: - -```powershell -git -C ../ryzenai-corelib rev-parse origin/main -git -C ../ryzenai-corelib show 3c35aebdefa3f0c2255668bab1be5648ece320f8:include/ryzenai/corelib.h | - Select-String 'RYZENAI_CORELIB_VERSION_(MAJOR|MINOR|PATCH)' -``` - -Expected: first command prints `3c35aebdefa3f0c2255668bab1be5648ece320f8`; version lines are 0, 3, 0. Build/install that exact commit in an isolated corelib worktree or use an existing installation whose provenance records that commit; do not alter the sibling working tree if it contains other work. - -Then run: - -```powershell -$env:FLM_AIE4_CORELIB_PATH='C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll' -ctest --test-dir src/build/phi4-corelib-tests -C Release ` - -R "^test_real_corelib$" --output-on-failure -``` - -Expected: PASS, runtime reports exactly 0.3.0, all 26 symbols resolve, dependency self-test succeeds, and device context is true. A skip is not acceptance when the variable is set. - -- [ ] **Step 5: Commit review gate** - -```powershell -git add src/test/phi4_corelib_aie4 -git commit -m "test: validate Phi-4 GGUF AIE4 integration" -``` - -Expected: this commit contains tests/fake changes only; the complete offline suite and real-DLL test are green. - -### Task 7: Developer Documentation and Real-AIE4 Acceptance - -**Files:** -- Create: `src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1` -- Modify: `docs/docs/models/phi.md` -- Modify: `docs/docs/benchmarks/phi4_results.md` - -**Interfaces:** -- Consumes the normal `flm.exe`, the installed four-file model directory, and `FLM_AIE4_CORELIB_PATH`. -- Produces a machine-readable acceptance JSON beside the build output and a human-readable checked-in record in `phi4_results.md`. - -- [ ] **Step 1: Write the acceptance runner before using the device** - -The script parameters are concrete: - -```powershell -param( - [string]$FlmExe = 'src/build-aie4/Release/flm.exe', - [string]$Model = 'phi4-mini-it-aie4:4b', - [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', - [string]$Output = 'src/build-aie4/phi4-gguf-aie4-acceptance.json', - [int]$Port = 52625 -) -``` - -It must fail nonzero unless it records: machine/CPU/NPU identity, Windows build, power mode, FastFlowLM commit, corelib commit and ABI, GGUF/tokenizer revisions, DLL SHA-256, four model-file hashes, exact commands, exit codes, response text/token IDs, load time, cold/warm TTFT, decode tokens/s, cancellation result, boundary results, and backend evidence from `show_profile()` naming `corelib_aie4_gguf` plus the loaded DLL path. - -- [ ] **Step 2: Run the required model acquisition commands on the AIE4 host** - -```powershell -$env:FLM_AIE4_CORELIB_PATH='C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll' -src/build-aie4/Release/flm.exe pull phi4-mini-it-aie4:4b -src/build-aie4/Release/flm.exe check phi4-mini-it-aie4:4b -``` - -Expected: both succeed; all four files validate; the model directory contains only `Phi-4-mini-instruct.Q8_0.gguf`, `tokenizer.json`, `tokenizer_config.json`, and `config.json`. - -- [ ] **Step 3: Run CLI semantic and repeated-load acceptance** - -Use the script to run `flm run phi4-mini-it-aie4:4b` with `What is 2+2?` and `What does AMD do?`, then at least ten prompts in one loaded process. Require the first answer to contain the correct value 4 and self-terminate; require the second to be relevant to AMD's semiconductor/computing business and self-terminate. - -Run at least ten complete process/model load-and-generate cycles. Fail if a response is empty, every emitted token ID is zero, the profile omits the exact backend/DLL, or any cycle exits nonzero. - -```powershell -powershell -NoProfile -ExecutionPolicy Bypass ` - -File src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 ` - -FlmExe src/build-aie4/Release/flm.exe ` - -CorelibDll $env:FLM_AIE4_CORELIB_PATH -``` - -Expected: script exits 0 and writes the complete JSON record. - -- [ ] **Step 4: Run REST, cancellation, and boundary acceptance** - -The same script starts: - -```powershell -src/build-aie4/Release/flm.exe serve phi4-mini-it-aie4:4b --port 52625 -``` - -It must issue both `POST /api/chat` and `POST /v1/chat/completions`, test streaming and non-streaming responses, cancel an active generation, then submit another request to prove the queue/model remains usable when cancellation occurred at a synchronized boundary. Test total rendered-prompt-plus-generation budgets at 4095 (accepted/capped) and 4096 (HTTP 400 before submission). Inject no fallback configuration; backend evidence must still name corelib/AIE4. - -Expected: both APIs return relevant nonempty text, cancellation completes without a process exit, the next request succeeds, boundary statuses match, and no CPU/NPU2 backend appears in logs/profile. - -- [ ] **Step 5: Record descriptive performance and documentation** - -Update `docs/docs/models/phi.md` with the exact tag, four-source pinning, Windows developer build flags, `FLM_AIE4_CORELIB_PATH` lookup/fallback, dependency-directory requirement, 4095 usable generation window, no fallback, no packaged runtime, and an explicit statement that Q8_0 is lossily requantized to group 64. - -Append the acceptance JSON's exact machine, power mode, commits/revisions, commands, pass/fail outcomes, load time, cold/warm TTFT, and decode tokens/s to `docs/docs/benchmarks/phi4_results.md`. Label performance descriptive and do not invent a pass threshold. - -- [ ] **Step 6: Run final repository verification** - -```powershell -ctest --test-dir src/build/phi4-corelib-tests -C Release --output-on-failure -cmake --build --preset windows-default -cmake --build --preset windows-aie4 -src/build-aie4/Release/flm.exe check phi4-mini-it-aie4:4b -Select-String -Path docs/docs/models/phi.md,docs/docs/benchmarks/phi4_results.md ` - -Pattern '3c35aebdefa3f0c2255668bab1be5648ece320f8','0.3.0','78eb92a46fc37e6b524df991ed9aca9bc6aa7b80','cfbefacb99257ffa30c83adab238a50856ac3083','lossy','group 64' -Get-ChildItem (Join-Path $env:USERPROFILE 'Documents/flm/models/phi4-mini-it-aie4') | - Where-Object { $_.Name -match '(manifest|onnx|converted|packed)' } -``` - -Expected: all configured tests pass (no real-DLL skip on the AIE4 host); both builds succeed; check succeeds; every required documentation string is found; the final `Get-ChildItem` command emits nothing. - -- [ ] **Step 7: Commit final review gate** - -```powershell -git add src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 ` - docs/docs/models/phi.md docs/docs/benchmarks/phi4_results.md -git commit -m "docs: document developer setup and hardware results" -``` - -Expected: the seventh commit contains the reproducible runner and actual observed record, with no unfilled values. - -## Final Review Checklist - -- [ ] `git diff origin/main...HEAD --name-only` contains only the approved product/test/doc files plus this plan and its design spec, and no installer/package files. -- [ ] After the planning commits, `git log --oneline origin/main..HEAD` shows the seven implementation commits in this order: - -```text -docs: document developer setup and hardware results -test: validate Phi-4 GGUF AIE4 integration -feat: pull Phi-4 GGUF and tokenizer from pinned sources -feat: route Phi-4 GGUF models through AIE4 -feat: add corelib-backed Phi-4 AIE4 engine -feat: add validated Phi-4 Q8_0 GGUF reader -build: add optional dynamic corelib 0.3.0 runtime -``` - -- [ ] Search the diff for `manifest`, `.onnx`, `weights_create_gguf(`, `ryzenai_corelib.lib`, and runtime-copy/install additions; only explanatory negative assertions may match. -- [ ] Confirm all 129 matmul and 32 SSMLP creations use the requantized Q8_0 group-64 entry points serially, and the single RMSNorm uses `weights_create_scale`. -- [ ] Confirm invalid GGUF/config/tokenizer input produces zero stream/tensor/weight creates. -- [ ] Confirm default build and legacy `phi4-mini-it:4b` behavior remain unchanged. -- [ ] Confirm missing corelib does not prevent process startup or ordinary-model execution. -- [ ] Confirm an AIE4 request never reaches Q4NX, `phi4_npu`, CPU fallback, or an alternate model file. -- [ ] Confirm cancellation, 4095/4096 bounds, and post-submit poison semantics in both fake tests and REST acceptance. -- [ ] Confirm the real-DLL integration and all real-AIE4 acceptance cases passed; skipped hardware tests do not satisfy completion. -- [ ] Confirm documentation contains actual measured values and exact provenance, not an empty table or promised follow-up. diff --git a/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md b/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md deleted file mode 100644 index 0bfe5fb1..00000000 --- a/docs/superpowers/specs/2026-09-11-phi4-gguf-aie4-design.md +++ /dev/null @@ -1,372 +0,0 @@ -# Phi-4 Q8_0 GGUF on AIE4 Design - -## Summary - -Add one catalog model, `phi4-mini-it-aie4:4b`, that FastFlowLM can pull and run through `ryzenai-corelib` on AIE4. The implementation starts from FastFlowLM `main`, supports only the validated Phi-4 Mini Instruct Q8_0 GGUF, and uses corelib's explicit lossy Q8_0-to-group-64 requantization APIs. - -The model is read directly from GGUF. FastFlowLM will not generate or ship an ONNX initializer manifest, an ONNX model, or converted weight files. The Phi-4 architecture and tensor-name contract live in a model-specific C++ adapter, while shape, dtype, offset, and model metadata come from the GGUF file. - -This PR produces an AIE4-enabled developer build of the normal `flm.exe`. It does not package the AIE4 runtime in MSI, WiX, or Inno Setup. - -## Fixed inputs - -### Model - -- FLM tag: `phi4-mini-it-aie4:4b` -- GGUF repository: `unsloth/Phi-4-mini-instruct-GGUF` -- GGUF revision: `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` -- GGUF file: `Phi-4-mini-instruct.Q8_0.gguf` -- Supported quantization: GGML `Q8_0` only - -### Tokenizer and configuration - -- Repository: `microsoft/Phi-4-mini-instruct` -- Revision: `cfbefacb99257ffa30c83adab238a50856ac3083` -- Files: `tokenizer.json`, `tokenizer_config.json`, and `config.json` - -The tokenizer files come from a second repository because the selected Unsloth repository does not publish the files FastFlowLM's existing tokenizer frontend consumes. Model loading cross-checks the tokenizer/config contract against GGUF metadata rather than assuming the two fixed sources agree. - -### Corelib - -- Repository: `VitisAI/ryzenai-corelib` -- Commit: `3c35aebdefa3f0c2255668bab1be5648ece320f8` -- ABI version: `0.3.0` - -Because corelib remains pre-1.0, FastFlowLM requires an exact runtime version match: major, minor, and patch must all be `0.3.0`. - -## Goals - -1. `flm pull phi4-mini-it-aie4:4b` downloads the pinned GGUF and the pinned tokenizer/config files. -2. An AIE4-enabled `flm.exe` runs the model through corelib from the CLI and REST APIs. -3. The existing Phi-4 NPU2/Q4NX model continues to use its existing backend. -4. A build without AIE4 support, or an AIE4 build with no corelib DLL, still starts and runs non-AIE4 models. -5. The implementation validates the model and tokenizer contracts before creating device state. -6. No silent CPU or NPU2 fallback is possible for the AIE4 tag. -7. The completed implementation is exercised on real AIE4 hardware before the PR is considered complete. - -## Non-goals - -This PR does not add: - -- ONNX model loading for AIE4; -- a JSON tensor manifest or manifest generator; -- a generic GGUF runtime or arbitrary local GGUF support; -- Q4_0, Q4_K, Q6_K, or mixed-quantization support; -- another model family; -- a packed-weight disk cache; -- parallel Q8_0 weight creation; -- Python as a runtime dependency; -- automatic corelib/runtime download; -- MSI, WiX, or Inno Setup packaging; -- a new corelib API; -- CPU or NPU2 fallback for the AIE4 model. - -## Architecture - -```text -flm.exe - └── Phi4 frontend - ├── existing chat, tokenizer, sampling, and server integration - └── phi4_corelib_aie4 - ├── phi4_corelib_gguf GGUF v3 parsing and Phi-4 tensor mapping - ├── phi4_corelib_shape_plan corelib padding and buffer extents - ├── corelib_runtime DLL lifetime, version, and availability - └── corelib_api dynamically resolved C ABI - │ - └── ryzenai_corelib.dll → AIE4 -``` - -### Backend selection - -The new catalog entry sets `details.execution_backend` to `corelib_aie4_gguf`. `Phi4::load_model()` selects the new backend only for that explicit value. An absent backend field retains the current NPU2 behavior. An unknown value is an error. - -There is no automatic hardware or format detection and no fallback. This makes a request for the AIE4 tag observable and testable: it either runs through corelib or fails. - -### Component boundaries - -#### Phi-4 frontend - -`modeling_phi4.cpp` remains responsible for: - -- backend selection; -- tokenizer setup and chat-template application; -- sampling; -- request capacity checks; -- translating the existing `AutoModel` interface to the selected causal-LM engine. - -It does not parse GGUF or call individual corelib operators. - -#### Corelib API and runtime - -A small dynamic adapter resolves only the C ABI symbols used by this backend: - -- version, dependency self-test, device-context query, errors, and cleanup; -- object lifecycle; -- stream creation and synchronization; -- tensor creation, windows, reads, and writes; -- matmul padding, Q8_0 requantized weight creation, and dispatch; -- SSMLP padding, Q8_0 requantized weight creation, and dispatch; -- RMSNorm weight creation, padding, and dispatch; -- flat-MHA padding and dispatch. - -The adapter owns no model policy. It converts failed statuses into exceptions that retain the corelib status, call name, and thread-local detail message. RAII wrappers release every returned object. - -Corelib is not loaded during process startup. It is loaded when an AIE4 model is selected. Runtime lookup order is: - -1. the absolute DLL named by `FLM_AIE4_CORELIB_PATH`; -2. `/aie4/ryzenai_corelib.dll`. - -The loader does not search the current working directory. After loading, it resolves the version functions first, requires ABI `0.3.0`, resolves the remaining symbols, runs the dependency self-test, and verifies an AIE4 device context exists. - -#### GGUF package - -`Phi4GgufPackage` owns a read-only mapping of the single GGUF file and exposes validated, non-owning tensor views whose lifetime cannot exceed the mapping. It parses only the GGUF v3 facilities used by the pinned model: - -- little-endian header; -- metadata scalar, string, and array encodings; -- tensor names, dimensions, GGML types, and relative offsets; -- model alignment and tensor-data start. - -The parser performs checked arithmetic for every count, offset, alignment, and byte-length calculation. A truncated directory, duplicate tensor name, unsupported value type that cannot be skipped safely, out-of-range tensor, overlapping invalid range, or malformed string is a load error. - -The model-facing API is intentionally narrow: - -```cpp -TensorView RequireQ8(name, expected_shape); -FloatTensorView RequireF32(name, expected_shape); -ProjectionViews AttentionQkv(layer); -ProjectionViews GateUp(layer); -GgufPhi4Metadata Metadata(); -``` - -The adapter hardcodes Phi-4 Mini's expected tensor names and architecture. QKV and gate/up tensors are fused in this GGUF; the adapter splits each into row-aligned byte-range views without dequantizing or copying it. Q8_0 rows consist of complete 34-byte blocks for 32 weights, so every allowed split must fall on a complete-row boundary. - -#### Phi-4 AIE4 engine - -The execution engine selectively carries forward the hardware-validated structure from PR #706: - -- one corelib stream; -- padded tensors sized from corelib's helper APIs; -- fixed-size K/V caches; -- prefill and one-token decode; -- explicit synchronization at producer/consumer boundaries; -- host-side lazy embedding lookup; -- Phi-4 partial rotary tables; -- corelib matmul, fused SSMLP, standalone RMSNorm, and flat MHA dispatch; -- a maximum sequence length of 4096 and maximum usable decode window of 4095. - -The source path changes completely: no ONNX initializers and no manifest are accepted. - -For each quantized projection, the engine passes a raw Q8_0 block view to `ryzenai_corelib_*_weights_create_gguf_requantized` with group size 64. Weight objects are created serially. The corelib API documents an open, unattributed all-zero-output incident correlated with concurrent requantized creates; avoiding concurrency is the measured safe configuration and is required for this first implementation. - -GGUF stores norms as F32. The adapter converts only the required norm vectors and epsilon to BF16 at model load. Embedding rows are decoded lazily for requested token IDs instead of materializing the full 200064-by-3072 embedding. RoPE tables are derived once from the GGUF's Phi-3/Phi-4 rope metadata and uploaded as FP32. - -## Model contract validation - -Validation occurs before device weight creation wherever possible. The package must match all of these constraints: - -- the expected Phi-3/Phi-4 GGUF architecture identifier; -- 32 decoder layers; -- hidden size 3072; -- intermediate size 8192; -- 24 attention heads; -- 8 key/value heads; -- head size 128; -- vocabulary size 200064; -- partial rotary dimension 96; -- RMS epsilon `1e-5`; -- maximum sequence length 4096; -- `phi3.rope.dimension_count` equal to 96; -- finite, positive `phi3.rope.freq_base` and `phi3.rope.scaling.attn_factor`; -- `phi3.rope.scaling.original_context_length` equal to 4096; -- `rope_factors_short.weight`, when present, is F32 with exactly 48 elements; -- the long-rope branch is rejected because this backend supports only the original 4096-token window; -- every required projection present with the exact expected logical shape; -- every projection, tied embedding, and LM head source is Q8_0; -- every required norm present in the supported floating type; -- `output.weight` is absent and `token_embd.weight` is used for both embedding and LM head, as in the pinned model; -- tokenizer vocabulary size agrees with GGUF; -- `tokenizer.json` maps `<|end|>` to 200020 and `<|endoftext|>` to 199999; -- GGUF identifies 200020 as its EOS token and `config.json` identifies 199999, so the frontend stop set is their explicit union `{200020, 199999}`; -- `tokenizer_config.json` has `add_bos_token == false` and the frontend does not prepend `config.json`'s BOS token; -- the chat template contains the required Phi-4 user, end, and assistant markers. - -The error names the model field or tensor, its actual value, and the expected value. The loader does not repair, reinterpret, or silently accept a mismatch. - -## Pull and catalog design - -The existing catalog format assumes one base repository per model. Retain the existing string-only `files` array and add an optional `file_sources` object keyed by those file names. Each override contains `url` and `revision`; files without an override continue to use the model's base URL unchanged. `model_info.json` remains the source of expected remote size and content hash for `pull` and `check`. No existing catalog entry needs migration. - -For each file, pull: - -1. constructs a URL from that file's fixed repository and revision; -2. downloads or resumes into a temporary path; -3. validates expected size and SHA-256; -4. atomically renames the completed file into the model directory. - -A model is available only when all required files validate. `flm check` uses the same per-file records. No generated overlay is copied into the model directory. - -The final directory is: - -```text -models/phi4-mini-it-aie4/ -├── Phi-4-mini-instruct.Q8_0.gguf -├── tokenizer.json -├── tokenizer_config.json -└── config.json -``` - -## Build and runtime configuration - -The feature is disabled by default. An AIE4 developer build enables: - -```text -FLM_ENABLE_CORELIB_AIE4=ON -RYZENAI_CORELIB_INCLUDE_DIR= -``` - -The build consumes the public header from the pinned corelib commit but does not link its import library. Calls go through the dynamically resolved function table. The normal `flm.exe` is produced and supports `pull`, `run`, and `serve`. - -This PR does not copy runtime DLLs. The developer supplies `ryzenai_corelib.dll` and its DynamicDispatch, XRT, and RyzenMM dependency closure. `FLM_AIE4_CORELIB_PATH` or the executable-relative `aie4` directory identifies corelib itself; its dependent DLL directory must be available to the Windows loader. - -A default build has no corelib compile or runtime requirement. An AIE4-enabled build with a missing runtime still starts and can run ordinary models; selecting `phi4-mini-it-aie4:4b` reports the missing runtime. - -## Request lifecycle and error policy - -A process-wide AIE4 access manager serializes AIE4 generation. One model instance handles one active generation at a time, matching the stream and mutable KV-cache ownership model. - -Failures before any operation is submitted are recoverable model/request errors. Failures after submission, or during synchronization, leave device completion uncertain. The model instance is then marked poisoned, its conversational state is cleared, and subsequent requests are refused until the model is unloaded and recreated. FastFlowLM does not continue on potentially inconsistent KV state. - -A cancellation is checked before prefill and between decode steps. It never destroys a stream while work is outstanding; submitted work is synchronized before the request releases model state. - -Capacity checks happen before submission. They account for the rendered prompt and requested generation budget and enforce the AIE4 decode limit of 4095. Unbounded/sentinel generation requests are capped rather than allowed to reach an unsupported attention window. - -## Expected file changes - -### Existing files - -```text -src/CMakeLists.txt -src/CMakePresets.json -src/common/AutoModel/automodel.cpp -src/common/AutoModel/modeling_phi4.cpp -src/include/AutoModel/automodel.hpp -src/include/AutoModel/modeling_phi4.hpp -src/pull/model_downloader.cpp -src/pull/model_downloader.hpp -src/model_list.json -src/model_info.json -src/runner/runner.cpp -src/server/rest_handler.cpp -src/server/server.cpp -src/src/main.cpp -``` - -Only files proven necessary during implementation should be changed. In particular, downloader changes are limited to optional per-file sources, and shared frontend changes are limited to behavior the AIE4 route requires. - -### New product files - -```text -src/include/corelib/corelib_api.hpp -src/include/corelib/corelib_object.hpp -src/include/corelib/corelib_runtime.hpp -src/common/corelib/corelib_api.cpp -src/common/corelib/corelib_runtime.cpp -src/common/corelib/corelib_sources.cmake -src/include/models/phi4/phi4_corelib_aie4.hpp -src/include/models/phi4/phi4_corelib_constants.hpp -src/include/models/phi4/phi4_corelib_gguf.hpp -src/include/models/phi4/phi4_corelib_shape_plan.hpp -src/include/models/phi4/phi4_corelib_host.hpp -src/common/corelib/phi4_corelib_aie4.cpp -src/common/corelib/phi4_corelib_gguf.cpp -src/common/corelib/phi4_corelib_shape_plan.cpp -src/common/corelib/phi4_corelib_host.cpp -``` - -The host component owns lazy Q8_0 embedding-row decode, F32-to-BF16 norm conversion, and FP32 RoPE-table derivation. The engine owns only model state and operator sequencing. - -## Testing - -### Unit tests without AIE4 hardware - -Tests cover: - -- valid GGUF v3 metadata and tensor-directory parsing; -- truncation, arithmetic overflow, bad alignment, duplicate names, and out-of-file ranges; -- missing tensors and incorrect dtype, shape, or byte length; -- zero-copy QKV and gate/up splits at exact row boundaries; -- Phi-4 architecture validation; -- tokenizer/config/GGUF disagreement; -- missing corelib symbols and exact ABI mismatch; -- object release and cleanup through a fake corelib; -- corelib call descriptors, group size 64, Q8_0 type, sequencing, and synchronization; -- per-file pull URLs, revisions, hashes, resume behavior, and atomic completion; -- unchanged behavior for existing single-source catalog entries; -- frontend routing and the absence of fallback; -- request bounds, cancellation, and poisoned-instance behavior. - -### Runtime integration - -Against the real DLL, tests verify: - -- ABI `0.3.0`; -- every required symbol resolves; -- the dependency self-test succeeds; -- device-context reporting agrees with the test environment. - -### Required AIE4 acceptance run - -Before completion, run the produced `flm.exe` on a real AIE4 system: - -```powershell -flm pull phi4-mini-it-aie4:4b -flm check phi4-mini-it-aie4:4b -flm run phi4-mini-it-aie4:4b -flm serve phi4-mini-it-aie4:4b -``` - -The acceptance run must include: - -1. `What is 2+2?`, with a correct, self-terminated answer; -2. `What does AMD do?`, with a relevant answer; -3. at least ten prompts in one loaded process; -4. `/api/chat` and `/v1/chat/completions`; -5. request cancellation; -6. prompt and generation limit boundaries; -7. at least ten complete load-and-generate cycles, checking for empty or all-zero token output; -8. backend evidence proving corelib/AIE4 execution and no CPU/NPU2 fallback; -9. model load time, cold and warm TTFT, and decode tokens/second. - -The record identifies the machine, power mode, FastFlowLM commit, corelib commit, corelib ABI, GGUF revision, commands, and outcomes. Performance numbers are descriptive, not a pass/fail gate, unless a regression threshold is agreed separately. - -## Commit structure - -Keep the work in one PR with reviewable commits: - -1. `build: add optional dynamic corelib 0.3.0 runtime` -2. `feat: add validated Phi-4 Q8_0 GGUF reader` -3. `feat: add corelib-backed Phi-4 AIE4 engine` -4. `feat: route Phi-4 GGUF models through AIE4` -5. `feat: pull Phi-4 GGUF and tokenizer from pinned sources` -6. `test: validate Phi-4 GGUF AIE4 integration` -7. `docs: document developer setup and hardware results` - -Each of commits 1–5 must compile before the next product commit is added. The test and documentation commits may depend on the completed product path. No commit adds a generated tensor manifest. - -## Completion criteria - -The PR is complete only when: - -- default builds and existing model behavior remain unchanged; -- an AIE4-enabled build produces the normal `flm.exe`; -- ordinary models remain usable when corelib is absent; -- the new tag pulls and checks all files from their pinned sources; -- the installed model contains no manifest, ONNX model, or converted weights; -- the GGUF is mapped directly and Q8_0 projection views are passed to corelib's explicit requantized APIs; -- runtime ABI is exactly `0.3.0`; -- automated unit and fake-corelib tests pass; -- the real-DLL integration checks pass; -- the required real-AIE4 acceptance run passes; -- no CPU or NPU2 fallback exists; -- documentation states that Q8_0-to-group-64 conversion is lossy and records the tested revisions and hardware results. From c6827a431eb93028a4682ec9ad224f36487d6aff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 17:44:37 -0700 Subject: [PATCH 28/37] fixup! feat: add corelib-backed Phi-4 AIE4 engine --- src/common/corelib/phi4_corelib_aie4.cpp | 32 ++++----- src/common/corelib/phi4_corelib_host.cpp | 36 ++++++++++ .../corelib/phi4_corelib_shape_plan.cpp | 9 --- src/include/corelib/corelib_api.hpp | 3 - src/include/corelib/corelib_object.hpp | 2 - src/include/models/phi4/phi4_corelib_host.hpp | 8 +++ .../models/phi4/phi4_corelib_shape_plan.hpp | 1 - src/test/phi4_corelib_aie4/fake_corelib.cpp | 32 +-------- .../phi4_corelib_aie4/test_corelib_api.cpp | 27 +++++--- .../phi4_corelib_aie4/test_phi4_engine.cpp | 66 ++++++++----------- src/test/phi4_corelib_aie4/test_phi4_gguf.cpp | 3 +- src/test/phi4_corelib_aie4/test_phi4_host.cpp | 42 ++++++++++++ .../test_phi4_shape_plan.cpp | 7 +- 13 files changed, 152 insertions(+), 116 deletions(-) diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp index cab588a1..e2ab0783 100644 --- a/src/common/corelib/phi4_corelib_aie4.cpp +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -29,13 +29,13 @@ struct phi4_corelib_aie4::Impl { std::optional saved; bool poisoned{}; UniqueStream stream; - UniqueRmsNormWeights first_norm; std::array q_weights, k_weights, v_weights, o_weights; std::array mlp_weights; UniqueMatMulWeights lm_weights; UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits, cosine, sine; std::array k_cache, v_cache; TensorView embedding; + FloatTensorView first_norm_scale; Impl(LM_Config, std::shared_ptr pkg, std::shared_ptr rt, std::uint32_t maximum) @@ -69,11 +69,9 @@ struct phi4_corelib_aie4::Impl { for(std::size_t i=0;i epsf{kRmsEpsilon}; auto eps=ConvertF32ToBf16(epsf); + first_norm_scale = an[0]; auto lease=runtime->AcquireExecution(); void* raw=nullptr; api->Check(api->functions().create_stream(&raw),"ryzenai_corelib_create_stream"); stream=UniqueStream(api,raw); - ryzenai_corelib_rmsnorm_bf16_weights_desc rd{kHiddenSize,kRmsEpsilon}; raw=nullptr; - api->Check(api->functions().rmsnorm_weights_create_scale(&rd,an_bf[0].data(),&raw),"ryzenai_corelib_rmsnorm_bf16_weights_create_scale blk.0.attn_norm.weight"); - first_norm=UniqueRmsNormWeights(api,raw); auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ ryzenai_corelib_matmul_bf16_weights_desc d{kk,nn,kRequantizedGroupSize,false}; ryzenai_corelib_matmul_bf16_gguf_components c{tv.bytes.data(),ryzenai_corelib_gguf_quant_type_q8_0}; void* p=nullptr; @@ -94,7 +92,7 @@ struct phi4_corelib_aie4::Impl { lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); const auto& e=plan.maximum_extents(); const auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, - e.ssmlp_rows,e.rmsnorm_rows}); + e.ssmlp_rows}); const auto query_rows=std::max(e.query_rows,e.flat_mha_rows); const auto key_rows=std::max(e.kv_rows,e.flat_mha_rows); const auto attention_rows=std::max(e.flat_mha_rows,e.output_rows); @@ -125,22 +123,26 @@ struct phi4_corelib_aie4::Impl { if(ids.size()>max_length||position+ids.size()>max_length||position+ids.size()>kMaxSequenceLength)throw std::out_of_range("Phi-4 request exceeds configured context capacity"); if(!prefill&&position+ids.size()>kMaxDecodeWindow)throw std::out_of_range("Phi-4 decode window stops at position 4095"); auto decoded=DecodeEmbeddingRowsQ8(embedding,ids);const auto&e=plan.ForRows(ids.size()); - auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, - e.ssmlp_rows,e.rmsnorm_rows}); - std::vector input(static_cast(rows*kHiddenSize),0);std::copy(decoded.begin(),decoded.end(),input.begin()); + auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows,e.ssmlp_rows}); + std::vector normalized(decoded.size()); + HostRmsNorm(decoded,first_norm_scale.values,ids.size(),kHiddenSize, + kRmsEpsilon,normalized); + std::vector input(static_cast(rows*kHiddenSize),0); + std::vector residual_input(static_cast(rows*kHiddenSize),0); + std::copy(normalized.begin(),normalized.end(),input.begin()); + std::copy(decoded.begin(),decoded.end(),residual_input.begin()); auto lease=runtime->AcquireExecution();bool submitted=false; try{ api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); - api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write residual embedding"); - const auto rms_status=api->functions().rmsnorm( - stream.get(),hidden.get(),ids.size(),first_norm.get(),hidden.get()); - submitted=rms_status==ryzenai_corelib_status_success || - rms_status==ryzenai_corelib_status_failure; - api->Check(rms_status,"ryzenai_corelib_rmsnorm_bf16 initial"); + api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_fp32,residual_input.data(),residual_input.size(),0),"ryzenai_corelib_tensor_write residual embedding"); void* res=residual.get();void* sk=skip.get(); for(std::size_t i=0;iCheck(api->functions().matmul(stream.get(),hidden.get(),ids.size(),q_weights[i].get(),q.get()),"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); + const auto query_status=api->functions().matmul( + stream.get(),hidden.get(),ids.size(),q_weights[i].get(),q.get()); + submitted=submitted || query_status==ryzenai_corelib_status_success || + query_status==ryzenai_corelib_status_failure; + api->Check(query_status,"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); api->Check(api->functions().matmul(stream.get(),hidden.get(),ids.size(),k_weights[i].get(),k.get()),"ryzenai_corelib_matmul_bf16 key layer "+std::to_string(i)); std::array shape{8,kMaxSequenceLength-position,128};void* p=nullptr; api->Check(api->functions().create_tensor_window(v_cache[i].get(),shape.data(),shape.size(),static_cast(position)*128,&p),"ryzenai_corelib_create_tensor_window V");UniqueTensorWindow win(api,p); diff --git a/src/common/corelib/phi4_corelib_host.cpp b/src/common/corelib/phi4_corelib_host.cpp index d9bf5b40..e817a601 100644 --- a/src/common/corelib/phi4_corelib_host.cpp +++ b/src/common/corelib/phi4_corelib_host.cpp @@ -80,6 +80,42 @@ std::vector DecodeEmbeddingRowsQ8( return result; } +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output) { + if (rows <= 0 || width <= 0) + throw std::invalid_argument("Phi-4 RMSNorm rows and width must be positive"); + const auto row_count = static_cast(rows); + const auto row_width = static_cast(width); + if (row_count > std::numeric_limits::max() / row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape overflow"); + const auto elements = row_count * row_width; + if (input.size() != elements || output.size() != elements || + scale.size() != row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape mismatch"); + if (!std::isfinite(epsilon) || epsilon < 0.0f) + throw std::invalid_argument("Phi-4 RMSNorm epsilon must be finite and nonnegative"); + + for (std::size_t row = 0; row < row_count; ++row) { + const auto base = row * row_width; + double sum_of_squares = 0.0; + for (std::size_t column = 0; column < row_width; ++column) { + const double value = input[base + column]; + sum_of_squares += value * value; + } + const float mean_square = static_cast( + sum_of_squares / static_cast(width)); + const float denominator = std::sqrt(mean_square + epsilon); + for (std::size_t column = 0; column < row_width; ++column) + output[base + column] = + (input[base + column] / denominator) * scale[column]; + } +} + std::vector ConvertF32ToBf16(std::span values) { std::vector result; result.reserve(values.size()); diff --git a/src/common/corelib/phi4_corelib_shape_plan.cpp b/src/common/corelib/phi4_corelib_shape_plan.cpp index fd81854f..4867b07b 100644 --- a/src/common/corelib/phi4_corelib_shape_plan.cpp +++ b/src/common/corelib/phi4_corelib_shape_plan.cpp @@ -60,13 +60,6 @@ Phi4ShapePlan Phi4ShapePlan::Build( &extents.ssmlp_rows, kHiddenSize, kIntermediateSize, kRequantizedGroupSize), ssmlp_call); - extents.rmsnorm_rows = rows; - const std::string rms_call = - "ryzenai_corelib_rmsnorm_bf16_pad_rows [" + std::to_string(rows) + - ",3072]"; - api->Check(api->functions().rmsnorm_pad_rows( - &extents.rmsnorm_rows, kHiddenSize), rms_call); - extents.flat_mha_rows = rows; const std::string mha_call = "ryzenai_corelib_flat_mha_bf16_pad_rows [" + std::to_string(rows) + @@ -81,8 +74,6 @@ Phi4ShapePlan Phi4ShapePlan::Build( plan.maximum_extents_.output_rows, extents.output_rows); plan.maximum_extents_.ssmlp_rows = std::max( plan.maximum_extents_.ssmlp_rows, extents.ssmlp_rows); - plan.maximum_extents_.rmsnorm_rows = std::max( - plan.maximum_extents_.rmsnorm_rows, extents.rmsnorm_rows); plan.maximum_extents_.flat_mha_rows = std::max( plan.maximum_extents_.flat_mha_rows, extents.flat_mha_rows); while (plan.rows_.size() < static_cast(rows)) diff --git a/src/include/corelib/corelib_api.hpp b/src/include/corelib/corelib_api.hpp index e251ee0e..d1e811fd 100644 --- a/src/include/corelib/corelib_api.hpp +++ b/src/include/corelib/corelib_api.hpp @@ -40,9 +40,6 @@ X(ssmlp_weights_create_gguf_requantized, \ ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) \ X(ssmlp, ryzenai_corelib_ssmlp_bf16) \ - X(rmsnorm_weights_create_scale, ryzenai_corelib_rmsnorm_bf16_weights_create_scale) \ - X(rmsnorm_pad_rows, ryzenai_corelib_rmsnorm_bf16_pad_rows) \ - X(rmsnorm, ryzenai_corelib_rmsnorm_bf16) \ X(flat_mha_pad_rows, ryzenai_corelib_flat_mha_bf16_pad_rows) \ X(flat_mha, ryzenai_corelib_flat_mha_bf16) \ X(cleanup, ryzenai_corelib_cleanup) diff --git a/src/include/corelib/corelib_object.hpp b/src/include/corelib/corelib_object.hpp index 2dd106be..faf44fd7 100644 --- a/src/include/corelib/corelib_object.hpp +++ b/src/include/corelib/corelib_object.hpp @@ -12,7 +12,6 @@ struct TensorTag {}; struct TensorWindowTag {}; struct MatMulWeightsTag {}; struct SsMlpWeightsTag {}; -struct RmsNormWeightsTag {}; template class UniqueObject final { @@ -59,6 +58,5 @@ using UniqueTensor = UniqueObject; using UniqueTensorWindow = UniqueObject; using UniqueMatMulWeights = UniqueObject; using UniqueSsMlpWeights = UniqueObject; -using UniqueRmsNormWeights = UniqueObject; } // namespace flm::corelib diff --git a/src/include/models/phi4/phi4_corelib_host.hpp b/src/include/models/phi4/phi4_corelib_host.hpp index 9695b854..67fb1094 100644 --- a/src/include/models/phi4/phi4_corelib_host.hpp +++ b/src/include/models/phi4/phi4_corelib_host.hpp @@ -18,6 +18,14 @@ std::vector DecodeEmbeddingRowsQ8( const TensorView& embedding, std::span token_ids); +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output); + std::vector ConvertF32ToBf16(std::span values); RopeTables BuildShortRopeTables( diff --git a/src/include/models/phi4/phi4_corelib_shape_plan.hpp b/src/include/models/phi4/phi4_corelib_shape_plan.hpp index f27af767..8c8010bb 100644 --- a/src/include/models/phi4/phi4_corelib_shape_plan.hpp +++ b/src/include/models/phi4/phi4_corelib_shape_plan.hpp @@ -14,7 +14,6 @@ struct Phi4RowExtents { std::int64_t kv_rows; std::int64_t output_rows; std::int64_t ssmlp_rows; - std::int64_t rmsnorm_rows; std::int64_t flat_mha_rows; }; diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp index a3cc5b19..2486bd59 100644 --- a/src/test/phi4_corelib_aie4/fake_corelib.cpp +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -274,14 +274,6 @@ struct TypedFake { if (status == ryzenai_corelib_status_success && m) *m = PaddedRows("ssmlp", *m); return status; - } else if constexpr (std::is_same_v) { - auto* m = std::get<0>(arguments); - state.rows_pad_calls.push_back({"rmsnorm", m ? *m : -1, - std::get<1>(arguments), 0, 0}); - const auto status = Status(Tag::name); - if (status == ryzenai_corelib_status_success && m) - *m = PaddedRows("rmsnorm", *m); - return status; } else if constexpr (std::is_same_v) { auto* m = std::get<0>(arguments); auto* desc = std::get<1>(arguments); @@ -323,29 +315,11 @@ struct TypedFake { if (status == ryzenai_corelib_status_success && out) *out = NewObject("ssmlp_weights"); --state.active_weight_creates; return status; - } else if constexpr (std::is_same_v) { - const auto status = Status(Tag::name); - auto* desc = std::get<0>(arguments); - auto* out = std::get<2>(arguments); - if (out) *out = nullptr; - ObserveCreateConcurrency(); - if (desc) { - fake_corelib::WeightCreateRecord record{"rmsnorm", desc->k, 0, 0, 0, {}}; - record.epsilon = Bf16(desc->epsilon); - if (std::get<1>(arguments)) - record.norm0.assign(static_cast(std::get<1>(arguments)), - static_cast(std::get<1>(arguments)) + desc->k); - state.weight_creates.push_back(std::move(record)); - } - if (status == ryzenai_corelib_status_success && out) *out = NewObject("rmsnorm_weights"); - --state.active_weight_creates; - return status; } else if constexpr (std::is_same_v) { state.work_in_flight = false; return Status(Tag::name); } else if constexpr (std::is_same_v || std::is_same_v || - std::is_same_v || std::is_same_v) { if (state.statuses.contains("test_observe_dispatch_concurrency")) { const int active = ++state.active_leases; @@ -362,8 +336,7 @@ struct TypedFake { fake_corelib::DispatchRecord record{}; record.thread_id = std::this_thread::get_id(); record.kind = std::is_same_v ? "matmul" : - std::is_same_v ? "ssmlp" : - std::is_same_v ? "rmsnorm" : "mha"; + std::is_same_v ? "ssmlp" : "mha"; record.stream = std::get<0>(arguments); if constexpr (std::is_same_v) { record.input = std::get<1>(arguments); record.rows = std::get<2>(arguments); @@ -371,9 +344,6 @@ struct TypedFake { } else if constexpr (std::is_same_v) { record.input = std::get<1>(arguments); record.rows = std::get<3>(arguments); record.output = std::get<6>(arguments); - } else if constexpr (std::is_same_v) { - record.input = std::get<1>(arguments); record.rows = std::get<2>(arguments); - record.output = std::get<4>(arguments); } else { record.input = std::get<2>(arguments); record.rows = std::get<4>(arguments); record.position = std::get<5>(arguments); record.output = std::get<10>(arguments); diff --git a/src/test/phi4_corelib_aie4/test_corelib_api.cpp b/src/test/phi4_corelib_aie4/test_corelib_api.cpp index 20967264..ad20fd2e 100644 --- a/src/test/phi4_corelib_aie4/test_corelib_api.cpp +++ b/src/test/phi4_corelib_aie4/test_corelib_api.cpp @@ -18,7 +18,6 @@ using flm::corelib::CorelibApi; using flm::corelib::CorelibError; using flm::corelib::CorelibRuntime; using flm::corelib::UniqueMatMulWeights; -using flm::corelib::UniqueRmsNormWeights; using flm::corelib::UniqueSsMlpWeights; using flm::corelib::UniqueStream; using flm::corelib::UniqueTensor; @@ -41,7 +40,7 @@ void TestVersionIsResolvedBeforeEveryOtherSymbol() { fake_corelib::Reset(); ValidApi(); const auto& order = fake_corelib::GetState().resolution_order; - TEST_REQUIRE(order.size() == 26); + TEST_REQUIRE(order.size() == 23); TEST_REQUIRE(order.front() == "ryzenai_corelib_get_version"); } @@ -72,7 +71,7 @@ void TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions() { void TestEveryRequiredSymbolIsResolvedExactlyOnce() { fake_corelib::Reset(); ValidApi(); - TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 26); + TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 23); for (const auto& [name, count] : fake_corelib::GetState().resolution_counts) { (void)name; TEST_REQUIRE(count == 1); @@ -86,11 +85,11 @@ void TestEveryResolvedFakeFunctionUsesItsExactAbi() { fake_corelib::GetState().default_status = ryzenai_corelib_status_bad_argument; fake_corelib::GetState().selftest_status = ryzenai_corelib_status_bad_argument; const auto statuses = fake_corelib::CallEveryResolvedFunction(api->functions()); - TEST_REQUIRE(statuses.size() == 20); + TEST_REQUIRE(statuses.size() == 17); TEST_REQUIRE(std::all_of(statuses.begin(), statuses.end(), [](auto status) { return status == ryzenai_corelib_status_bad_argument; })); - TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 26); + TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 23); for (const auto& [name, count] : fake_corelib::GetState().call_counts) { (void)name; TEST_REQUIRE(count == 1); @@ -105,6 +104,18 @@ void TestEveryResolvedFakeFunctionUsesItsExactAbi() { .call_counts["ryzenai_corelib_tensor_write"] == 2); } +void TestStandaloneRmsNormSymbolsAreNotRequired() { + for (const auto* symbol : { + "ryzenai_corelib_rmsnorm_bf16_weights_create_scale", + "ryzenai_corelib_rmsnorm_bf16_pad_rows", + "ryzenai_corelib_rmsnorm_bf16"}) { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = symbol; + (void)ValidApi(); + TEST_REQUIRE(!fake_corelib::GetState().resolution_counts.contains(symbol)); + } +} + void TestMissingSymbolNamesTheSymbolAndUnloadsTheDll() { fake_corelib::Reset(); fake_corelib::GetState().missing_symbol = "ryzenai_corelib_create_stream"; @@ -179,12 +190,11 @@ void TestEveryUniqueObjectReleasesExactlyOnceAfterMoves() { UniqueTensorWindow window(api, fake_corelib::MakeObject()); UniqueMatMulWeights matmul(api, fake_corelib::MakeObject()); UniqueSsMlpWeights ssmlp(api, fake_corelib::MakeObject()); - UniqueRmsNormWeights rmsnorm(api, fake_corelib::MakeObject()); TEST_REQUIRE(!first && !moved && assigned); - TEST_REQUIRE(api->live_object_count() == 6); + TEST_REQUIRE(api->live_object_count() == 5); TEST_REQUIRE(fake_corelib::GetState().releases == 0); } - TEST_REQUIRE(fake_corelib::GetState().releases == 6); + TEST_REQUIRE(fake_corelib::GetState().releases == 5); TEST_REQUIRE(api->live_object_count() == 0); } @@ -281,6 +291,7 @@ int main() { RUN_TEST(TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions); RUN_TEST(TestEveryRequiredSymbolIsResolvedExactlyOnce); RUN_TEST(TestEveryResolvedFakeFunctionUsesItsExactAbi); + RUN_TEST(TestStandaloneRmsNormSymbolsAreNotRequired); RUN_TEST(TestMissingSymbolNamesTheSymbolAndUnloadsTheDll); RUN_TEST(TestCorelibErrorCopiesStatusCallAndThreadLocalDetail); RUN_TEST(TestEnvironmentPathMustBeAnAbsoluteDllPath); diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index 6fe6c015..acc1213b 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -63,8 +63,7 @@ void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { Harness h([](auto& state) { state.pad_row_overrides["matmul-3072"][2048] = 5000; state.pad_row_overrides["matmul-1024"][2048] = 6000; - state.pad_row_overrides["ssmlp"][2048] = 7000; - state.pad_row_overrides["rmsnorm"][2048] = 8000; + state.pad_row_overrides["ssmlp"][2048] = 8000; state.pad_row_overrides["mha"][2048] = 9000; }); const auto& tensors = fake_corelib::GetState().tensor_creates; @@ -76,19 +75,18 @@ void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); } -void TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight() { +void TestEngineCreatesExactly129MatmulAnd32SsmlpWeights() { Harness h; const auto& records = fake_corelib::GetState().weight_creates; - TEST_REQUIRE(records.size() == 162); + TEST_REQUIRE(records.size() == 161); TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "matmul"; }) == 129); TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "ssmlp"; }) == 32); - TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; }) == 1); + TEST_REQUIRE(std::none_of(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; })); } void TestEveryProjectionUsesQ8RequantizedGroup64Threads0() { Harness h; for (const auto& record : fake_corelib::GetState().weight_creates) { - if (record.kind == "rmsnorm") continue; TEST_REQUIRE(record.group_size == 64); TEST_REQUIRE(record.threads == 0); } @@ -100,9 +98,8 @@ void TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate() { Harness h; TEST_REQUIRE(fake_corelib::GetState().maximum_active_weight_creates == 1); const auto& records = fake_corelib::GetState().weight_creates; - TEST_REQUIRE(records.front().kind == "rmsnorm"); for (std::size_t layer = 0; layer < 32; ++layer) { - const auto base = 1 + layer * 5; + const auto base = layer * 5; TEST_REQUIRE(records[base + 0].kind == "matmul"); TEST_REQUIRE(records[base + 1].kind == "matmul"); TEST_REQUIRE(records[base + 2].kind == "matmul"); @@ -117,18 +114,18 @@ void TestQkvAndGateUpPointersMatchExactMappedSubranges() { const auto qkv = h.package->AttentionQkv(0); const auto gate_up = h.package->GateUp(0); const auto& records = fake_corelib::GetState().weight_creates; - TEST_REQUIRE(records[1].pointers[0] == qkv.values[0].bytes.data()); - TEST_REQUIRE(records[2].pointers[0] == qkv.values[1].bytes.data()); - TEST_REQUIRE(records[3].pointers[0] == qkv.values[2].bytes.data()); - TEST_REQUIRE(records[5].pointers[0] == gate_up.values[0].bytes.data()); - TEST_REQUIRE(records[5].pointers[1] == gate_up.values[1].bytes.data()); + TEST_REQUIRE(records[0].pointers[0] == qkv.values[0].bytes.data()); + TEST_REQUIRE(records[1].pointers[0] == qkv.values[1].bytes.data()); + TEST_REQUIRE(records[2].pointers[0] == qkv.values[2].bytes.data()); + TEST_REQUIRE(records[4].pointers[0] == gate_up.values[0].bytes.data()); + TEST_REQUIRE(records[4].pointers[1] == gate_up.values[1].bytes.data()); } void TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates() { Harness h; const auto& records = fake_corelib::GetState().weight_creates; for (std::size_t layer = 0; layer < 32; ++layer) { - const auto base = 1 + layer * 5; + const auto base = layer * 5; const auto qkv = h.package->AttentionQkv(layer); const auto gate_up = h.package->GateUp(layer); TEST_REQUIRE(records[base + 0].pointers == @@ -155,13 +152,12 @@ void TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates() { TEST_REQUIRE(records.back().pointers == embedding_pointer); } -void TestNormsAndEpsilonReachCorelibAsBf16() { +void TestFusedNormsAndEpsilonReachCorelibAsBf16() { Harness h; const auto expected = flm::phi4::ConvertF32ToBf16(std::array{1.0e-5f})[0]; const auto& records = fake_corelib::GetState().weight_creates; - TEST_REQUIRE(records.front().epsilon == expected); for (std::size_t layer = 0; layer < 32; ++layer) { - const auto& record = records[1 + layer * 5 + 4]; + const auto& record = records[layer * 5 + 4]; TEST_REQUIRE(record.epsilon == expected); TEST_REQUIRE(record.norm0.size() == 3072); TEST_REQUIRE(record.norm1.size() == 3072); @@ -220,18 +216,18 @@ void TestVProjectionWritesWindowAtPositionTimes128() { TEST_REQUIRE(windows.size() == 32); TEST_REQUIRE(windows.front().shape == std::vector({8, 4089, 128})); TEST_REQUIRE(windows.front().offset == 7 * 128); - TEST_REQUIRE(fake_corelib::GetState().dispatches[3].window_offset == 7 * 128); + TEST_REQUIRE(fake_corelib::GetState().dispatches[2].window_offset == 7 * 128); } void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { Harness h; (void)h.engine->forward(1); const auto& calls = fake_corelib::GetState().dispatches; - TEST_REQUIRE(calls.size() == 194); + TEST_REQUIRE(calls.size() == 193); const void* stream = calls.front().stream; - TEST_REQUIRE(calls.front().kind == "rmsnorm"); + TEST_REQUIRE(calls.front().kind == "matmul"); for (std::size_t layer = 0; layer < 32; ++layer) { - const std::size_t base = 1 + layer * 6; + const std::size_t base = layer * 6; TEST_REQUIRE(calls[base + 0].kind == "matmul"); TEST_REQUIRE(calls[base + 1].kind == "matmul"); TEST_REQUIRE(calls[base + 2].kind == "matmul"); @@ -328,22 +324,13 @@ void TestCheckpointRestoreChangesOnlyLogicalPosition() { void TestPreSubmitFailureIsRecoverable() { Harness h; - fake_corelib::GetState().statuses["ryzenai_corelib_rmsnorm_bf16"] = ryzenai_corelib_status_bad_argument; - RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "rmsnorm"); + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = ryzenai_corelib_status_bad_argument; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "tensor_write"); TEST_REQUIRE(!h.engine->poisoned()); - fake_corelib::GetState().statuses.erase("ryzenai_corelib_rmsnorm_bf16"); + fake_corelib::GetState().statuses.erase("ryzenai_corelib_tensor_write"); (void)h.engine->forward(0); } -void TestInitialRmsNormPostSubmitFailureSynchronizesAndPoisons() { - Harness h; - fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_rmsnorm_bf16"; - RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "rmsnorm"); - TEST_REQUIRE(h.engine->poisoned()); - TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); - TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_stream_synchronize"] == 1); -} - void TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState() { Harness h; h.engine->set_context_length(3); @@ -465,14 +452,14 @@ void TestTwoConcurrentAie4RequestsNeverOverlapDispatch() { const auto& dispatches = fake_corelib::GetState().dispatches; TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); - TEST_REQUIRE(dispatches.size() == 388); + TEST_REQUIRE(dispatches.size() == 386); const auto first_request = dispatches.front().thread_id; TEST_REQUIRE(first_request != dispatches.back().thread_id); - TEST_REQUIRE(std::all_of(dispatches.begin(), dispatches.begin() + 194, + TEST_REQUIRE(std::all_of(dispatches.begin(), dispatches.begin() + 193, [&](const auto& call) { return call.thread_id == first_request; })); - TEST_REQUIRE(std::all_of(dispatches.begin() + 194, dispatches.end(), + TEST_REQUIRE(std::all_of(dispatches.begin() + 193, dispatches.end(), [&](const auto& call) { return call.thread_id != first_request; })); @@ -485,7 +472,7 @@ void TestTwoConcurrentAie4RequestsNeverOverlapDispatch() { std::atomic unsafe_calls_succeeded{true}; const auto invoke_without_lease = [&] { unsafe_start.arrive_and_wait(); - if (h.runtime->api()->functions().rmsnorm( + if (h.runtime->api()->functions().matmul( nullptr, nullptr, 1, nullptr, nullptr) != ryzenai_corelib_status_success) unsafe_calls_succeeded = false; @@ -518,12 +505,12 @@ int main() { #define RUN_TEST(name) RunTest(&name, #name) RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); - RUN_TEST(TestEngineCreates129Matmul32SsmlpAndOneRmsNormWeight); + RUN_TEST(TestEngineCreatesExactly129MatmulAnd32SsmlpWeights); RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64Threads0); RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); RUN_TEST(TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates); - RUN_TEST(TestNormsAndEpsilonReachCorelibAsBf16); + RUN_TEST(TestFusedNormsAndEpsilonReachCorelibAsBf16); RUN_TEST(TestEmbeddingMappingOutlivesAllLazyRowReads); RUN_TEST(TestNoDeviceObjectExistsWhenPackageValidationFails); RUN_TEST(TestPrefillDecodesEmbeddingRowsAndAdvancesPosition); @@ -539,7 +526,6 @@ int main() { RUN_TEST(TestClearContextResetsLogicalPositionWithoutRecreatingWeights); RUN_TEST(TestCheckpointRestoreChangesOnlyLogicalPosition); RUN_TEST(TestPreSubmitFailureIsRecoverable); - RUN_TEST(TestInitialRmsNormPostSubmitFailureSynchronizesAndPoisons); RUN_TEST(TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState); RUN_TEST(TestSynchronizeFailurePoisonsAndClearsState); RUN_TEST(TestPoisonedInstanceRejectsEveryLaterEntryPoint); diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp index 81130666..492d04f5 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -536,8 +536,7 @@ void TestValidationCreatesNoCorelibObjects() { "ryzenai_corelib_create_device_tensor", "ryzenai_corelib_create_tensor_window", "ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized", - "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized", - "ryzenai_corelib_rmsnorm_bf16_weights_create_scale"}) + "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized"}) TEST_REQUIRE(fake_corelib::GetState().call_counts[name] == 0); TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); (void)api; diff --git a/src/test/phi4_corelib_aie4/test_phi4_host.cpp b/src/test/phi4_corelib_aie4/test_phi4_host.cpp index d62baed5..a8ba89b3 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_host.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_host.cpp @@ -92,6 +92,45 @@ void TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds() { "Q8_0"); } +void TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits() { + const std::array input{ + std::bit_cast(0xBE8BBBACu), + std::bit_cast(0xBCCC9DE0u), + std::bit_cast(0xBFED682Fu), + std::bit_cast(0xC2CD01EDu)}; + const std::array scale{1.0f, 1.0f, 1.0f, 1.0f}; + std::array output{}; + HostRmsNorm(input, scale, 1, 4, 1.0e-5f, output); + constexpr std::array expected{ + 0xBBAE75DBu, 0xB9FF7820u, 0xBD143451u, 0xBFFFF50Bu}; + for (std::size_t i = 0; i < output.size(); ++i) + TEST_REQUIRE(std::bit_cast(output[i]) == expected[i]); +} + +void TestHostRmsNormRejectsZeroAndShapeErrors() { + std::array input{1.0f, 2.0f}; + std::array scale{1.0f, 1.0f}; + std::array output{}; + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 0, 2, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 0, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, std::span(scale).first(1), 1, 2, 1.0e-5f, output); }), "shape"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 2, -1.0f, output); }), "epsilon"); +} + +void TestHostRmsNormMatchesPr706Bf16BoundaryReference() { + constexpr std::size_t width = 3072; + std::vector input(width, 0.03125f); + input[0] = 1024.0f; + std::vector scale(width, 1.0f); + std::vector output(width); + HostRmsNorm(input, scale, 1, width, 1.0e-5f, output); + const auto bf16 = ConvertF32ToBf16(output); + TEST_REQUIRE(std::bit_cast(output[0]) == 0x425DB3C3u); + TEST_REQUIRE(std::bit_cast(output[1]) == 0x3ADDB3C3u); + TEST_REQUIRE(bf16[0] == 0x425e); + TEST_REQUIRE(bf16[1] == 0x3ade); +} + void TestF32ToBf16UsesRoundToNearestEven() { const std::array values{ std::bit_cast(std::uint32_t{0x3f808000}), @@ -135,6 +174,9 @@ int main() { RUN_TEST(TestLazyEmbeddingDecodesOnlyRequestedRows); RUN_TEST(TestLazyEmbeddingPreservesRequestOrderAndDuplicates); RUN_TEST(TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds); + RUN_TEST(TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits); + RUN_TEST(TestHostRmsNormRejectsZeroAndShapeErrors); + RUN_TEST(TestHostRmsNormMatchesPr706Bf16BoundaryReference); RUN_TEST(TestF32ToBf16UsesRoundToNearestEven); RUN_TEST(TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs); RUN_TEST(TestRopeTablesApplyShortFactorsAndAttentionFactor); diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp index 04801aad..72e17aa3 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -22,13 +22,12 @@ void TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow() { constexpr std::array buckets{ 1, 64, 128, 256, 512, 1024, 2048, 4096}; TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * buckets.size() + 1); - TEST_REQUIRE(state.rows_pad_calls.size() == 2 * buckets.size()); + TEST_REQUIRE(state.rows_pad_calls.size() == buckets.size()); TEST_REQUIRE(state.mha_pad_calls.size() == buckets.size()); for (std::size_t index = 0; index < buckets.size(); ++index) { TEST_REQUIRE(state.matmul_pad_calls[index * 3].m == buckets[index]); TEST_REQUIRE(state.matmul_pad_calls[index * 3].group_size == 64); - TEST_REQUIRE(state.rows_pad_calls[index * 2].m == buckets[index]); - TEST_REQUIRE(state.rows_pad_calls[index * 2 + 1].m == buckets[index]); + TEST_REQUIRE(state.rows_pad_calls[index].m == buckets[index]); TEST_REQUIRE(state.mha_pad_calls[index].m == buckets[index]); } TEST_REQUIRE(plan.ForRows(2).query_rows == 64); @@ -50,8 +49,6 @@ void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); - TEST_REQUIRE(state.rows_pad_calls[1].helper == "rmsnorm"); - TEST_REQUIRE(state.rows_pad_calls[1].k == 3072); const auto& lm = state.matmul_pad_calls.back(); TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); TEST_REQUIRE(plan.lm_head_desc().k == 3072); From 87b9ce8e533ae52d6d2a0716834c56033b17c6ad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 18:35:29 -0700 Subject: [PATCH 29/37] fixup! feat: route Phi-4 GGUF models through AIE4 The AIE4 route has its own decode loop and never touched the DECODING_TIME profiler, so every run reported "Decoding time: 0 us" and an "-nan(ind)" decode speed, and meta_info.decoding_duration stayed zero on all four generation endpoints. Record the profiler around each forward, exactly as the shared decode loop does. Co-Authored-By: Claude Opus 5 (1M context) --- src/common/AutoModel/modeling_phi4.cpp | 5 ++++ .../phi4_corelib_aie4/test_phi4_frontend.cpp | 23 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 4ca36777..9ce7780a 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -252,6 +252,7 @@ std::string Phi4::generate_aie4(chat_meta_info_t& meta_info, std::string result; meta_info.stop_reason = EOT_DETECTED; int generated = 0; + profiler_list[DECODING_TIME].reset(); while (last_token != -1 && generated < aie4_generation_budget_) { if (is_cancelled()) { meta_info.stop_reason = CANCEL_DETECTED; @@ -280,9 +281,13 @@ std::string Phi4::generate_aie4(chat_meta_info_t& meta_info, meta_info.stop_reason = CANCEL_DETECTED; break; } + profiler_list[DECODING_TIME].start(); auto logits = lm_engine->forward(token); + profiler_list[DECODING_TIME].stop(1); last_token = sampler->sample(logits); } + meta_info.decoding_duration = (uint64_t)(time_utils::cast_to_us( + profiler_list[DECODING_TIME].get_total_time()).first) * 1e3; return result; } #endif diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp index 23cea2d0..2bb19b7e 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -19,7 +19,9 @@ #include #include #include +#include #include +#include #include namespace { @@ -36,6 +38,7 @@ class FakeEngine final : public causal_lm { buffer forward(int token) override { ++forward_calls; forwarded.push_back(token); + if (forward_delay.count()) std::this_thread::sleep_for(forward_delay); if (fail_forward) { poisoned_state = true; throw std::runtime_error("submitted inference failed"); @@ -72,6 +75,7 @@ class FakeEngine final : public causal_lm { bool fail_prefill{}; bool fail_forward{}; bool poisoned_state{}; + std::chrono::microseconds forward_delay{0}; std::vector forwarded; }; @@ -564,6 +568,24 @@ void TestEosSelfTerminatesWithoutAnExtraDecode() { TEST_REQUIRE(meta.stop_reason == EOT_DETECTED); } +void TestAie4DecodeTimeAndSpeedAreMeasured() { + // The AIE4 route has its own decode loop, so it must record DECODING_TIME + // itself. Without that the profile reports "0 us" and a nan speed, and the + // hardware acceptance record has no decode throughput to publish. + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11, 12, 13, 200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->forward_delay = std::chrono::microseconds(2000); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 3); + TEST_REQUIRE(meta.decoding_duration > 0); + const auto profile = model->show_profile(); + TEST_REQUIRE(profile.find("nan") == std::string::npos); + TEST_REQUIRE(profile.find("inf") == std::string::npos); + TEST_REQUIRE(profile.find("Decoding time: 0 ") == std::string::npos); +} + void TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics() { for (const auto raw : {std::optional{}, std::optional{0}, std::optional{-2}, std::optional{17}}) { const auto expected = raw && *raw > 0 ? raw : std::nullopt; @@ -705,6 +727,7 @@ int main() { RunTest(TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned, "TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned"); RunTest(TestPoisonedModelReturns500UntilReload, "TestPoisonedModelReturns500UntilReload"); RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); + RunTest(TestAie4DecodeTimeAndSpeedAreMeasured, "TestAie4DecodeTimeAndSpeedAreMeasured"); RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); RunTest(TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff, "TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff"); From e6fc2792bbd100bf9f321aec390ce828ef2faf23 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Fri, 11 Sep 2026 19:44:26 -0700 Subject: [PATCH 30/37] test: add the real-AIE4 acceptance runner Drives the full hardware matrix from one script: pinned-package check, the CLI semantic and repeated-load cycles through a PTY, both REST APIs in streaming and non-streaming form, cancellation and recovery, the 4095/4096 boundary, and the descriptive performance record. JSON request bodies are written to files and read back by curl with "@file". Passing a body inline lets PowerShell's native-argument quoting strip the double quotes, which is why the earlier cancellation and boundary probes recorded a server-side JSON parse error instead of a result. Co-Authored-By: Claude Opus 5 (1M context) --- .../run_real_aie4_acceptance.ps1 | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 diff --git a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 new file mode 100644 index 00000000..2cf038d0 --- /dev/null +++ b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 @@ -0,0 +1,98 @@ +param( + [string]$FlmExe = 'src/build-aie4/Release/flm.exe', + [string]$Model = 'phi4-mini-it-aie4:4b', + [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', + [string]$Output = 'src/build-aie4/phi4-gguf-aie4-acceptance.json', + [int]$Port = 52625, + [string]$Python = 'python' +) +$ErrorActionPreference='Stop' +$root=(Resolve-Path (Join-Path $PSScriptRoot '../../..')).Path +$exe=(Resolve-Path (Join-Path $root $FlmExe)).Path +$core=(Resolve-Path $CorelibDll).Path +$outPath=[IO.Path]::GetFullPath((Join-Path $root $Output)) +$outDir=Split-Path $outPath +$modelDir=Join-Path $env:USERPROFILE '.flm/models/phi4-mini-it-aie4' +$env:FLM_AIE4_CORELIB_PATH=$core +$env:FLM_CONFIG_PATH=Join-Path $root 'src/model_list.json' +$env:FLM_XCLBIN_PATH=Join-Path $root 'src' +$runtime=Split-Path $core +$env:PATH="$(Join-Path $root 'src/lib/xrt');$(Join-Path $root 'src/lib');$runtime;C:/Users/chiz/.conda/envs/hybrid-llm/Library/bin;C:/Users/chiz/work/hybrid-llm/install/xrt_package/xrt;$env:PATH" +New-Item -ItemType Directory -Force $outDir | Out-Null +$record=[ordered]@{started=(Get-Date).ToString('o');passed=$false;commands=@();host=[ordered]@{};provenance=[ordered]@{};files=@();cli=[ordered]@{};rest=[ordered]@{};performance=[ordered]@{};failures=@()} +function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$record.failures+=($_|Out-String);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} +function Post([string]$path,$body){try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}}catch{if($_.Exception.Response){$resp=$_.Exception.Response;$reader=New-Object IO.StreamReader($resp.GetResponseStream());$text=$reader.ReadToEnd();return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=($text|ConvertFrom-Json)}};throw}} +# A JSON body must never be passed to curl as an inline argument: PowerShell's +# native-argument quoting strips the double quotes and the server receives a +# malformed object. Every body goes to a file and curl reads it with "@file". +function BodyFile([string]$name,$body){$p=Join-Path $outDir $name;Set-Content -Path $p -Value ($body|ConvertTo-Json -Depth 8 -Compress) -Encoding ASCII -NoNewline;return $p} +function CurlStream([string]$name,[string]$path,$body){$f=BodyFile $name $body;return (&curl.exe -sS -N -H 'Content-Type: application/json' -d "@$f" "http://127.0.0.1:$Port$path"|Out-String)} +function CurlBackground([string]$name,[string]$path,$body,[string]$outFile){$f=BodyFile $name $body;return (Start-Process curl.exe -ArgumentList @('-sS','-N','-H','Content-Type: application/json','-d',"@$f","http://127.0.0.1:$Port$path") -RedirectStandardOutput $outFile -PassThru)} +try{ + $record.host.computer=$env:COMPUTERNAME;$record.host.cpu=(Get-CimInstance Win32_Processor).Name;$record.host.npu=(Get-CimInstance Win32_PnPEntity|Where-Object Name -match 'NPU|Neural').Name;$os=Get-CimInstance Win32_OperatingSystem;$record.host.windows="$($os.Caption) $($os.Version) build $($os.BuildNumber)";$record.host.power=(powercfg /getactivescheme|Out-String).Trim() + $record.provenance.fastflow=(git -C $root rev-parse HEAD).Trim();$coreRoot=(Resolve-Path (Join-Path $runtime '..')).Path;$record.provenance.corelib=(git -C $coreRoot rev-parse HEAD).Trim();$record.provenance.corelib_abi='0.3.0';$record.provenance.gguf_revision='78eb92a46fc37e6b524df991ed9aca9bc6aa7b80';$record.provenance.tokenizer_revision='cfbefacb99257ffa30c83adab238a50856ac3083';$record.provenance.corelib_sha256=(Get-FileHash $core -Algorithm SHA256).Hash.ToLower() + Cmd "$exe check $Model" {&$exe check $Model|Out-Host;if($LASTEXITCODE-ne 0){throw 'check failed'}} + $names=@('Phi-4-mini-instruct.Q8_0.gguf','tokenizer.json','tokenizer_config.json','config.json');$actual=@(Get-ChildItem $modelDir -File|% Name);if((Compare-Object ($names|Sort-Object) ($actual|Sort-Object))){throw 'model directory is not exactly four files'};foreach($n in $names){$f=Get-Item (Join-Path $modelDir $n);$record.files+=@([ordered]@{name=$n;bytes=$f.Length;sha256=(Get-FileHash $f.FullName -Algorithm SHA256).Hash.ToLower()})} + $py=@' +from winpty import PtyProcess +import os,sys,time,threading,json,re +exe,model,out=sys.argv[1:4] +def run(cmds,timeout=900): + p=PtyProcess.spawn(f'{exe} run {model}',env=os.environ.copy(),dimensions=(50,200));chunks=[] + def rd(): + while p.isalive(): + try: chunks.append(p.read(8192)) + except: break + threading.Thread(target=rd,daemon=True).start(); end=time.time()+timeout + while time.time() Date: Fri, 11 Sep 2026 22:23:47 -0700 Subject: [PATCH 31/37] test: stop the acceptance runner mangling its own curl arguments A header value containing a space is split into two native arguments, so curl took the second half as another URL and reported "Could not resolve host: application" while sending the request without the intended content type. The colon form carries the same meaning and cannot split. Also give every REST call a timeout and fail loudly when a cancelled stream does not end, so a wedged request reports instead of hanging the run. Co-Authored-By: Claude Opus 5 (1M context) --- .../run_real_aie4_acceptance.ps1 | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 index 2cf038d0..848a75db 100644 --- a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 +++ b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 @@ -21,13 +21,17 @@ $env:PATH="$(Join-Path $root 'src/lib/xrt');$(Join-Path $root 'src/lib');$runtim New-Item -ItemType Directory -Force $outDir | Out-Null $record=[ordered]@{started=(Get-Date).ToString('o');passed=$false;commands=@();host=[ordered]@{};provenance=[ordered]@{};files=@();cli=[ordered]@{};rest=[ordered]@{};performance=[ordered]@{};failures=@()} function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$record.failures+=($_|Out-String);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} -function Post([string]$path,$body){try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}}catch{if($_.Exception.Response){$resp=$_.Exception.Response;$reader=New-Object IO.StreamReader($resp.GetResponseStream());$text=$reader.ReadToEnd();return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=($text|ConvertFrom-Json)}};throw}} -# A JSON body must never be passed to curl as an inline argument: PowerShell's -# native-argument quoting strips the double quotes and the server receives a -# malformed object. Every body goes to a file and curl reads it with "@file". +function Post([string]$path,$body,[int]$TimeoutSec=900){try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -TimeoutSec $TimeoutSec -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}}catch{if($_.Exception.Response){$resp=$_.Exception.Response;$reader=New-Object IO.StreamReader($resp.GetResponseStream());$text=$reader.ReadToEnd();return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=($text|ConvertFrom-Json)}};throw}} +# Two things must never reach curl as inline arguments. A JSON body loses its +# double quotes to PowerShell's native-argument quoting and the server sees a +# malformed object, so every body goes to a file and is read back with "@file". +# A header value containing a space is split into two arguments, so the second +# half is taken as another URL ("Could not resolve host: application"); the +# colon form without a space carries the same meaning and cannot split. +$ContentTypeArg='Content-Type:application/json' function BodyFile([string]$name,$body){$p=Join-Path $outDir $name;Set-Content -Path $p -Value ($body|ConvertTo-Json -Depth 8 -Compress) -Encoding ASCII -NoNewline;return $p} -function CurlStream([string]$name,[string]$path,$body){$f=BodyFile $name $body;return (&curl.exe -sS -N -H 'Content-Type: application/json' -d "@$f" "http://127.0.0.1:$Port$path"|Out-String)} -function CurlBackground([string]$name,[string]$path,$body,[string]$outFile){$f=BodyFile $name $body;return (Start-Process curl.exe -ArgumentList @('-sS','-N','-H','Content-Type: application/json','-d',"@$f","http://127.0.0.1:$Port$path") -RedirectStandardOutput $outFile -PassThru)} +function CurlStream([string]$name,[string]$path,$body){$f=BodyFile $name $body;$out=(&curl.exe -sS -N -H $ContentTypeArg -d "@$f" "http://127.0.0.1:$Port$path" 2>&1|Out-String);if($LASTEXITCODE-ne 0){throw "curl failed ($LASTEXITCODE) for ${path}: $out"};if($out-match 'Could not resolve host'){throw "curl argument splitting for ${path}: $out"};return $out} +function CurlBackground([string]$name,[string]$path,$body,[string]$outFile){$f=BodyFile $name $body;return (Start-Process curl.exe -ArgumentList @('-sS','-N','-H',$ContentTypeArg,'-d',"@$f","http://127.0.0.1:$Port$path") -RedirectStandardOutput $outFile -PassThru)} try{ $record.host.computer=$env:COMPUTERNAME;$record.host.cpu=(Get-CimInstance Win32_Processor).Name;$record.host.npu=(Get-CimInstance Win32_PnPEntity|Where-Object Name -match 'NPU|Neural').Name;$os=Get-CimInstance Win32_OperatingSystem;$record.host.windows="$($os.Caption) $($os.Version) build $($os.BuildNumber)";$record.host.power=(powercfg /getactivescheme|Out-String).Trim() $record.provenance.fastflow=(git -C $root rev-parse HEAD).Trim();$coreRoot=(Resolve-Path (Join-Path $runtime '..')).Path;$record.provenance.corelib=(git -C $coreRoot rev-parse HEAD).Trim();$record.provenance.corelib_abi='0.3.0';$record.provenance.gguf_revision='78eb92a46fc37e6b524df991ed9aca9bc6aa7b80';$record.provenance.tokenizer_revision='cfbefacb99257ffa30c83adab238a50856ac3083';$record.provenance.corelib_sha256=(Get-FileHash $core -Algorithm SHA256).Hash.ToLower() @@ -79,13 +83,13 @@ json.dump({'prompts':prompts,'one_process':one,'cycles':cycles},open(out,'w',enc foreach($s in @($apiStream,$oaStream)){if($s-match '"error"'){throw "streaming response returned an error: $s"}} $cancelOut=Join-Path $outDir 'cancel-stream.txt' $cp=CurlBackground 'body-cancel.json' '/api/chat' @{model=$Model;request_id='accept-cancel';messages=@(@{role='user';content='Count upward for a long time.'});stream=$true;options=@{num_predict=1024}} $cancelOut - Start-Sleep -Milliseconds 1500;$cancel=Post '/api/cancel' @{request_id='accept-cancel'};$cp.WaitForExit(120000)|Out-Null + Start-Sleep -Milliseconds 1500;$cancel=Post '/api/cancel' @{request_id='accept-cancel'};if(-not$cp.WaitForExit(300000)){$cp.Kill();throw 'the cancelled stream did not end'} $recovery=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='What is 2+2?'});stream=$false;options=@{num_predict=8}} if(-not$cancel.json.cancelled-or$recovery.status-ne 200){throw 'cancellation recovery failed'} $probe=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='x'});stream=$false;options=@{num_predict=1}};$pt=[int]$probe.json.prompt_eval_count;$remaining=4095-$pt $bOut=Join-Path $outDir 'boundary-stream.txt' $bp=CurlBackground 'body-boundary.json' '/api/chat' @{model=$Model;request_id='boundary4095';messages=@(@{role='user';content='x'});stream=$true;options=@{num_predict=$remaining}} $bOut - Start-Sleep -Milliseconds 1500;$bcancel=Post '/api/cancel' @{request_id='boundary4095'};$bp.WaitForExit(120000)|Out-Null + Start-Sleep -Milliseconds 1500;$bcancel=Post '/api/cancel' @{request_id='boundary4095'};if(-not$bp.WaitForExit(300000)){$bp.Kill();throw 'the cancelled 4095 stream did not end'} $b4096=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='x'});stream=$false;options=@{num_predict=($remaining+1)}} if(-not$bcancel.json.cancelled-or$b4096.status-ne 400){throw "boundary behavior failed: 4095 cancelled=$($bcancel.json.cancelled) 4096 status=$($b4096.status)"} $record.rest=[ordered]@{api_chat_nonstream=$apiNon;openai_nonstream=$oaNon;api_chat_stream=$apiStream;openai_stream=$oaStream;cancellation=$cancel;recovery=$recovery;prompt_tokens=$pt;boundary4095_cancel=$bcancel;boundary4096=$b4096} From 00328162db0b72161b13200899d1a3bc1284cc0a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 00:10:01 -0700 Subject: [PATCH 32/37] docs: document developer setup and hardware results Records the acceptance run on XCOMEDUSAD-43 at 87721089: load to serving 44.2/47.4/49.1 s, cold TTFT 4.21 s, warm TTFT 65.0 ms, decode 21.3 tok/s over REST and 35.8 tok/s in a warm CLI session, with the machine, power scheme, commits, revisions and file hashes it came from. Performance is descriptive; no threshold is claimed. Two numbers are deliberately not presented as throughput. The record's load_duration is 1.5 us, which is the Ollama-compatible field on an already warm server rather than a model load, so load was measured separately over three fresh processes. The ten fresh-process CLI cycles report 3.70-20.26 tok/s because each pays the one-time kernel and ELF setup inside its own eight-token window. The model card separates the two context limits that were previously conflated. The 4096 ceiling is a correctness boundary: it is Phi-4-mini's rope.scaling.original_context_length, LongRoPE selects factors by sequence length, and only the short branch is derived here. The further step down to 4095 is this frontend's own conservatism so an admitted request can always finish, and is not imposed by corelib. Also records the one unexplained reply that degenerated into a markdown image URL, and corrects the emitter comment: the unbounded ConvertTo-Json allocation reproduces on PowerShell 7.0.0 as well, so a newer engine is not a fix. Co-Authored-By: Claude Opus 5 (1M context) --- docs/docs/benchmarks/phi4_results.md | 51 +++++++ docs/docs/models/phi.md | 63 +++++++++ .../run_real_aie4_acceptance.ps1 | 127 +++++++++++++++++- 3 files changed, 235 insertions(+), 6 deletions(-) diff --git a/docs/docs/benchmarks/phi4_results.md b/docs/docs/benchmarks/phi4_results.md index 7d7de5ea..1747bcae 100644 --- a/docs/docs/benchmarks/phi4_results.md +++ b/docs/docs/benchmarks/phi4_results.md @@ -41,3 +41,54 @@ AMD Ryzen™ AI 7 350 (Kraken Point) with 32 GB DRAM; performance is comparable | **Model** | **HW** | **1k** | **2k** | **4k** | **8k** | **16k** | **32k** | |------------------|--------------------|--------:|--------:|--------:|--------:|---------:|---------:| | **Phi-4-mini-instruct** | NPU (FLM) | 643 | 787 | 857 | 809 | 644 | 447 | + +--- + +## 🧪 Phi-4-mini-instruct Q8_0 GGUF on AIE4 (`phi4-mini-it-aie4:4b`) + +These are **descriptive measurements from a single acceptance run**, not a benchmark sweep and not a pass threshold. They are not comparable to the tables above: the prompts here are 4–10 tokens, whereas those tables sweep 1k–32k, so the per-token rates are dominated by fixed overhead rather than by context length. + +### Provenance + +| | | +|---|---| +| Machine | `XCOMEDUSAD-43` | +| CPU | `AMD Eng Sample: 100-000001713-33_N` | +| NPU | `AMD XDNA(TM) NPU` | +| OS | Microsoft Windows 11 Enterprise 10.0.26100 build 26100 | +| Windows power scheme | Balanced (`381b4222-f694-41f0-9685-ff5bb260df2e`). The NPU power mode is separately set to `performance` by FLM at startup. | +| FastFlowLM commit | `87721089097396579ec4529f50616a6c0e1c7b74` | +| corelib commit / ABI | `3c35aebdefa3f0c2255668bab1be5648ece320f8` / `0.3.0` | +| corelib DLL SHA-256 | `f404da219a3cc84d3334c265e09ba7987f0c4bcc1b1cedeac7c3c45a7be2c9ae` | +| GGUF revision | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| Tokenizer/config revision | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| Run | 2026-09-12 00:34:19 → 00:52:02, `passed: true`, 0 failures | + +### Measurements + +| Metric | Value | Conditions | +|---|---|---| +| Model load to serving | **44.2 / 47.4 / 49.1 s** | three consecutive fresh `flm serve` processes, timed from launch to the first successful `/api/version`. All 161 weights are requantized from Q8_0 at load. | +| Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup | +| Warm TTFT | **65.0 ms** | subsequent prompts in the same process | +| Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens | +| Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process | + +**Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed. + +The **5.4×** spread between warm TTFT (65 ms) and cold TTFT (4.21 s), and the **1.7×** spread between the REST and warm-CLI decode figures, are both unexplained by anything measured here. Treat single-run differences below roughly 2× as noise. + +### Functional results + +All from the same run: + +- `flm pull` / `flm check` — four pinned files, all SHA-256 verified; the model directory contains exactly those four. +- CLI — 10/10 fresh-process load-and-generate cycles exited 0; `Backend: corelib_aie4_gguf` and the loaded DLL path reported in every one. +- REST — `/api/chat` and `/v1/chat/completions` both 200, streaming and non-streaming. +- Cancellation — an in-flight stream cancelled cleanly; the next request returned 200 on the same server. +- Capacity boundary — a request totalling 4096 tokens is rejected with **HTTP 400** before submission (`rendered prompt has 4 tokens and requested output has 4092 tokens`); a 4095-token request is admitted. +- No CPU or NPU2 fallback appears in the server log at any point. + +### Known issue + +One `/api/chat` reply to `What is 2+2?` came back as a truncated markdown image URL (`![](https://media.giphy.com/media/kZl76FZgu`, `done_reason: length`) instead of an answer. The identical prompt answered correctly on three other occasions in the same session, including the recovery request in the same run, so this looks like sampling nondeterminism rather than a routing fault — but it is a single-observation defect, it is not understood, and it is recorded rather than smoothed over. diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 9b2c59bf..ba05c407 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -22,4 +22,67 @@ parent: Models flm run phi4-mini-it:4b ``` +--- + +## 🧪 Model Card: Phi-4-mini-instruct on AIE4 (developer preview) + +- **Tag:** `phi4-mini-it-aie4:4b` +- **Backend:** `corelib_aie4_gguf` — runs on AIE4 through AMD's `ryzenai_corelib.dll` +- **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped. +- **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source. +- **Usable generation window:** 4095 tokens — the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why. +- **Availability:** Windows only, and this is a **developer build**. The AIE4 runtime is not packaged by the MSI or Inno installer; you supply the DLL yourself. + +This tag pulls from two pinned repositories, because the GGUF publisher does not ship the tokenizer files FastFlowLM's tokenizer frontend consumes: + +| File | Repository | Revision | +|---|---|---| +| `Phi-4-mini-instruct.Q8_0.gguf` | [`unsloth/Phi-4-mini-instruct-GGUF`](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF) | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| `tokenizer.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `tokenizer_config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | + +All four are SHA-256 verified before the download is promoted, and the pulled directory contains exactly these four files. + +### Building + +The AIE4 path is compiled only when you ask for it. With the option off, the binary contains no reference to corelib at all. + +From `FastFlowLM/src`, in a Visual Studio developer command prompt: + +```powershell +$env:RYZENAI_CORELIB_INCLUDE_DIR = 'C:/path/to/ryzenai-corelib/install/include' +cmake --preset windows-aie4 # sets FLM_ENABLE_CORELIB_AIE4=ON, builds into src/build-aie4 +cmake --build --preset windows-aie4 +``` + +The `windows-aie4` preset reads `RYZENAI_CORELIB_INCLUDE_DIR` from the environment, so set it before configuring. The configure step also locates a Boost include directory, and hard-errors if the option is enabled on a non-Windows host. Everything else — XRT, FFmpeg, curl, FFTW — is the ordinary FastFlowLM dependency set; the AIE4 option does not relax any of it. + +### Pointing FastFlowLM at the runtime + +`flm.exe` never links `ryzenai_corelib.lib`; the DLL is resolved and loaded at run time, by absolute path: + +1. `FLM_AIE4_CORELIB_PATH`, if set. It must be an **absolute path to a `.dll` file** — a relative path or a directory is rejected outright. +2. Otherwise `\aie4\ryzenai_corelib.dll`. + +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. The DLL's own dependency directory must be reachable on `PATH`. + +```powershell +$env:FLM_AIE4_CORELIB_PATH = 'C:/path/to/ryzenai_corelib.dll' +flm pull phi4-mini-it-aie4:4b +flm run phi4-mini-it-aie4:4b +``` + +### Why the context is 4096, and why the usable window is one less + +Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults to 32k. This backend gives you 4095. That is a real functional regression and it has two separate causes, which are worth keeping apart. + +**The 4096 ceiling is a correctness boundary, not a buffer size.** 4096 is exactly Phi-4-mini's `rope.scaling.original_context_length`. LongRoPE selects its factors by *sequence length*, not per position: at or below the original length the short factors apply, above it the long ones do. This implementation derives only the short branch, so 4096 is the point past which the rope tables would silently be wrong. It is enforced rather than assumed — loading fails with `invalid Phi-4 RoPE metadata` unless the GGUF reports `rope.scaling.original_context_length` of exactly 4096. Raising this ceiling means deriving the long factors, not enlarging an array. + +**The extra −1 is this frontend's own conservatism.** `kMaxDecodeWindow` is 4095, one below the attention window, so that any request the server admits is guaranteed to have room to finish rather than failing partway. It costs exactly one token and it is not imposed by corelib. + +### No fallback + +Backend selection is explicit: it comes from `execution_backend` in the model catalog and is never inferred from hardware, filename, or quantization level. If corelib is missing, unloadable, or the wrong version, this tag **fails to load with a diagnostic** — it will not quietly fall back to CPU or to the NPU2/Q4NX backend. A build without AIE4 support, and an AIE4 build with no DLL present, both still start and run every other model, including `phi4-mini-it:4b`. + --- \ No newline at end of file diff --git a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 index 848a75db..5cc4c4da 100644 --- a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 +++ b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 @@ -4,7 +4,10 @@ param( [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', [string]$Output = 'src/build-aie4/phi4-gguf-aie4-acceptance.json', [int]$Port = 52625, - [string]$Python = 'python' + [string]$Python = 'python', + # Diagnostics only: skips the 16-minute CLI matrix so the REST phase can be + # iterated on quickly. A record produced this way can never report success. + [switch]$SkipCli ) $ErrorActionPreference='Stop' $root=(Resolve-Path (Join-Path $PSScriptRoot '../../..')).Path @@ -20,8 +23,106 @@ $runtime=Split-Path $core $env:PATH="$(Join-Path $root 'src/lib/xrt');$(Join-Path $root 'src/lib');$runtime;C:/Users/chiz/.conda/envs/hybrid-llm/Library/bin;C:/Users/chiz/work/hybrid-llm/install/xrt_package/xrt;$env:PATH" New-Item -ItemType Directory -Force $outDir | Out-Null $record=[ordered]@{started=(Get-Date).ToString('o');passed=$false;commands=@();host=[ordered]@{};provenance=[ordered]@{};files=@();cli=[ordered]@{};rest=[ordered]@{};performance=[ordered]@{};failures=@()} -function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$record.failures+=($_|Out-String);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} -function Post([string]$path,$body,[int]$TimeoutSec=900){try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -TimeoutSec $TimeoutSec -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}}catch{if($_.Exception.Response){$resp=$_.Exception.Response;$reader=New-Object IO.StreamReader($resp.GetResponseStream());$text=$reader.ReadToEnd();return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=($text|ConvertFrom-Json)}};throw}} +# Progress markers go to stdout so a run that stalls can be located from the +# transcript alone; a silent 30-minute stall is indistinguishable from work. +function Mark([string]$m){Write-Host ("[mark] "+(Get-Date).ToString('HH:mm:ss.fff')+" "+$m)} +# ConvertTo-Json cannot be used on the record as a whole. Some of the values it +# holds are live .NET objects whose property graphs loop back on themselves, and +# ConvertTo-Json expands such a graph until -Depth runs out, which allocates tens +# of gigabytes and never returns. This was confirmed on both Windows PowerShell +# 5.1 and PowerShell 7.0.0 (3.4 GB and still climbing when killed) — moving to a +# newer engine does not avoid it, so do not remove this. This emitter walks +# the record itself: it refuses to descend past $script:JsonMaxDepth, and it +# refuses to re-enter an object that is already an ancestor of the current node. +# ConvertTo-Json is still used, but only ever on a single scalar string. +$script:JsonMaxDepth=10 +function JsonScalar($s){return (ConvertTo-Json -InputObject ([string]$s))} +function EmitJson($v,[string]$label,[int]$level,$ancestors){ + if($null -eq $v){return 'null'} + if($v -is [string]){return (JsonScalar $v)} + if($v -is [bool]){if($v){return 'true'}else{return 'false'}} + if($v -is [datetime]){return (JsonScalar $v.ToString('o'))} + if($v -is [double] -or $v -is [single]){if([double]::IsNaN($v)-or[double]::IsInfinity($v)){return 'null'};return (([double]$v).ToString('R',[Globalization.CultureInfo]::InvariantCulture))} + if($v -is [ValueType] -and $v -isnot [char] -and $v -isnot [Enum]){return (([string]$v))} + if($level -ge $script:JsonMaxDepth){return (JsonScalar $v)} + # The ancestor test exists for live .NET objects, whose property graphs loop. + # It deliberately does not apply to a PSCustomObject: ConvertFrom-Json only + # ever builds trees, and every object it produces shares one singleton base + # instance, so testing those would report every nested JSON object as a loop. + $bo=$null;try{$bo=$v.PSObject.BaseObject}catch{} + $next=$ancestors + if($null -ne $bo -and $bo -isnot [System.Management.Automation.PSCustomObject]){ + foreach($a in $ancestors){if([object]::ReferenceEquals($a,$bo)){return (JsonScalar '')}} + # The ancestor list must be built with Add, not with "+". Adding an array + # with "+" splices its elements in, which would put every element of an + # array on the ancestor list and make each of them look like a loop. + $next=New-Object Collections.ArrayList + if($null -ne $ancestors){[void]$next.AddRange($ancestors)} + [void]$next.Add($bo) + } + $parts=New-Object Collections.ArrayList + if($v -is [System.Collections.IDictionary]){ + foreach($k in @($v.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + [void]$parts.Add((JsonScalar $k)+':'+(EmitJson $v[$k] "$label.$k" ($level+1) $next)) + if($level -lt 2){Mark ("json {0}.{1} in {2:N1}s" -f $label,$k,$sw.Elapsed.TotalSeconds)} + } + return '{'+($parts -join ',')+'}' + } + if($v -is [System.Collections.IEnumerable]){ + foreach($e in $v){[void]$parts.Add((EmitJson $e "$label[]" ($level+1) $next))} + return '['+($parts -join ',')+']' + } + $props=@($v.PSObject.Properties) + if($props.Count -gt 0){ + foreach($p in $props){ + $pv=$null;try{$pv=$p.Value}catch{$pv=""} + [void]$parts.Add((JsonScalar $p.Name)+':'+(EmitJson $pv "$label.$($p.Name)" ($level+1) $next)) + } + return '{'+($parts -join ',')+'}' + } + return (JsonScalar $v) +} +function WriteRecord($rec,[string]$path){ + [IO.File]::WriteAllText($path,"{`r`n",[Text.Encoding]::UTF8) + $first=$true + foreach($k in @($rec.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + try{$t=EmitJson $rec[$k] $k 1 (New-Object Collections.ArrayList)}catch{$t=JsonScalar ("")} + if(-not$first){[IO.File]::AppendAllText($path,",`r`n",[Text.Encoding]::UTF8)} + $first=$false + [IO.File]::AppendAllText($path,(' "{0}": {1}' -f $k,$t),[Text.Encoding]::UTF8) + Mark ("json section {0} written in {1:N1}s" -f $k,$sw.Elapsed.TotalSeconds) + } + [IO.File]::AppendAllText($path,"`r`n}`r`n",[Text.Encoding]::UTF8) +} +# Piping an ErrorRecord to Out-String yields nothing but a newline under some +# host configurations, which would record a failure with no reason attached. +function ErrText($e){ + $parts=@("$($e.Exception.GetType().FullName): $($e.Exception.Message)") + $rendered=($e|Out-String);if(-not [string]::IsNullOrWhiteSpace($rendered)){$parts+=$rendered.Trim()} + if($e.InvocationInfo -and $e.InvocationInfo.PositionMessage){$parts+=$e.InvocationInfo.PositionMessage.Trim()} + if($e.ScriptStackTrace){$parts+=$e.ScriptStackTrace.Trim()} + return ($parts -join "`n") +} +function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$t=ErrText $_;$record.failures+=$t;Mark ("FAILURE in ${line}: "+$t);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} +# A non-2xx reply is an error in both engines, but the two expose it +# differently: Windows PowerShell hands back a WebResponse to read a stream +# from, PowerShell 7 hands back an HttpResponseMessage and puts the body in +# ErrorDetails. An expected 400 must not depend on which engine is running. +function Post([string]$path,$body,[int]$TimeoutSec=900){ + try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -TimeoutSec $TimeoutSec -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}} + catch{ + $resp=$_.Exception.Response + if($null -eq $resp){throw} + $text=$null + if($_.ErrorDetails -and $_.ErrorDetails.Message){$text=$_.ErrorDetails.Message} + elseif($resp.PSObject.Methods['GetResponseStream']){$text=(New-Object IO.StreamReader($resp.GetResponseStream())).ReadToEnd()} + elseif($resp.Content){$text=$resp.Content.ReadAsStringAsync().GetAwaiter().GetResult()} + $json=$null;if(-not [string]::IsNullOrWhiteSpace($text)){try{$json=$text|ConvertFrom-Json}catch{}} + return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=$json} + } +} # Two things must never reach curl as inline arguments. A JSON body loses its # double quotes to PowerShell's native-argument quoting and the server sees a # malformed object, so every body goes to a file and is read back with "@file". @@ -71,32 +172,46 @@ for i,q in enumerate(prompts):cycles.append({'prompt':q,**run(['/set gen-lim 8', json.dump({'prompts':prompts,'one_process':one,'cycles':cycles},open(out,'w',encoding='utf8'),indent=2) '@ $pyPath=Join-Path $outDir 'accept_cli.py';$cliPath=Join-Path $outDir 'accept_cli.json';Set-Content -Path $pyPath -Value $py -Encoding UTF8 + Mark 'starting CLI matrix' + if($SkipCli){$record.skipped_cli=$true;Mark 'CLI matrix SKIPPED (diagnostic run, cannot pass)'}else{ Cmd 'python accept_cli.py (10 prompts + 10 cycles)' {&$Python $pyPath $exe $Model $cliPath;if($LASTEXITCODE-ne 0){throw 'CLI matrix failed'}};$record.cli=Get-Content -Raw $cliPath|ConvertFrom-Json - $allCli=$record.cli.one_process.text;if($allCli-notmatch '(?m)\b4\b'){throw 'CLI arithmetic answer missing 4'};if($allCli-notmatch '(?i)AMD|semiconductor|processor|comput'){throw 'CLI AMD answer irrelevant'};if($allCli-notmatch 'corelib_aie4_gguf' -or $allCli-notmatch [regex]::Escape($core)){throw 'CLI backend/DLL proof missing'};if($allCli-match 'nan|-nan\(ind\)'){throw 'CLI profile reported a nan speed'};foreach($c in $record.cli.cycles){if($c.exit_code-ne 0 -or $c.text-notmatch 'Tokens:\s*[1-9]'){throw "CLI cycle failed: $($c.prompt)"}} + Mark 'CLI matrix done' + $allCli=$record.cli.one_process.text;if($allCli-notmatch '(?m)\b4\b'){throw 'CLI arithmetic answer missing 4'};if($allCli-notmatch '(?i)AMD|semiconductor|processor|comput'){throw 'CLI AMD answer irrelevant'};if($allCli-notmatch 'corelib_aie4_gguf' -or $allCli-notmatch [regex]::Escape($core)){throw 'CLI backend/DLL proof missing'};if($allCli-match 'nan|-nan\(ind\)'){throw 'CLI profile reported a nan speed'};foreach($c in $record.cli.cycles){if($c.exit_code-ne 0 -or $c.text-notmatch 'Tokens:\s*[1-9]'){throw "CLI cycle failed: $($c.prompt)"}}} $serverLog=Join-Path $outDir 'accept-server.log';$serverErr=Join-Path $outDir 'accept-server.err.log';$server=Start-Process $exe -ArgumentList @('serve',$Model,'--port',$Port) -WorkingDirectory $root -RedirectStandardOutput $serverLog -RedirectStandardError $serverErr -PassThru try{for($i=0;$i-lt 300;$i++){try{$v=Invoke-RestMethod "http://127.0.0.1:$Port/api/version";break}catch{Start-Sleep -Milliseconds 200}};if(-not$v){throw 'server not ready'} + Mark 'server ready' $apiNon=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='What is 2+2?'});stream=$false;options=@{num_predict=16}} + Mark 'api_chat_nonstream done' $oaNon=Post '/v1/chat/completions' @{model=$Model;messages=@(@{role='user';content='What does AMD do?'});stream=$false;max_tokens=24} + Mark 'openai_nonstream done' $apiStream=CurlStream 'body-api-stream.json' '/api/chat' @{model=$Model;messages=@(@{role='user';content='Say hello.'});stream=$true;options=@{num_predict=8}} + Mark 'api_chat_stream done' $oaStream=CurlStream 'body-openai-stream.json' '/v1/chat/completions' @{model=$Model;messages=@(@{role='user';content='Name one GPU use.'});stream=$true;max_tokens=12} + Mark 'openai_stream done' if($apiNon.status-ne 200-or$oaNon.status-ne 200-or[string]::IsNullOrWhiteSpace($apiStream)-or[string]::IsNullOrWhiteSpace($oaStream)){throw 'REST API matrix failed'} foreach($s in @($apiStream,$oaStream)){if($s-match '"error"'){throw "streaming response returned an error: $s"}} $cancelOut=Join-Path $outDir 'cancel-stream.txt' $cp=CurlBackground 'body-cancel.json' '/api/chat' @{model=$Model;request_id='accept-cancel';messages=@(@{role='user';content='Count upward for a long time.'});stream=$true;options=@{num_predict=1024}} $cancelOut Start-Sleep -Milliseconds 1500;$cancel=Post '/api/cancel' @{request_id='accept-cancel'};if(-not$cp.WaitForExit(300000)){$cp.Kill();throw 'the cancelled stream did not end'} + Mark 'cancellation done' $recovery=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='What is 2+2?'});stream=$false;options=@{num_predict=8}} if(-not$cancel.json.cancelled-or$recovery.status-ne 200){throw 'cancellation recovery failed'} + Mark 'recovery done' $probe=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='x'});stream=$false;options=@{num_predict=1}};$pt=[int]$probe.json.prompt_eval_count;$remaining=4095-$pt + Mark "probe done prompt_tokens=$pt remaining=$remaining" $bOut=Join-Path $outDir 'boundary-stream.txt' $bp=CurlBackground 'body-boundary.json' '/api/chat' @{model=$Model;request_id='boundary4095';messages=@(@{role='user';content='x'});stream=$true;options=@{num_predict=$remaining}} $bOut Start-Sleep -Milliseconds 1500;$bcancel=Post '/api/cancel' @{request_id='boundary4095'};if(-not$bp.WaitForExit(300000)){$bp.Kill();throw 'the cancelled 4095 stream did not end'} + Mark 'boundary 4095 cancel done' $b4096=Post '/api/chat' @{model=$Model;messages=@(@{role='user';content='x'});stream=$false;options=@{num_predict=($remaining+1)}} + Mark "boundary 4096 done status=$($b4096.status)" if(-not$bcancel.json.cancelled-or$b4096.status-ne 400){throw "boundary behavior failed: 4095 cancelled=$($bcancel.json.cancelled) 4096 status=$($b4096.status)"} $record.rest=[ordered]@{api_chat_nonstream=$apiNon;openai_nonstream=$oaNon;api_chat_stream=$apiStream;openai_stream=$oaStream;cancellation=$cancel;recovery=$recovery;prompt_tokens=$pt;boundary4095_cancel=$bcancel;boundary4096=$b4096} $decodeTps=$null;if([double]$apiNon.json.eval_duration -gt 0){$decodeTps=[double]$apiNon.json.eval_count*1e9/[double]$apiNon.json.eval_duration} if($null-eq$decodeTps){throw 'decode duration was not reported; decode throughput cannot be recorded'} $record.performance=[ordered]@{load_ns=$apiNon.json.load_duration;cold_ttft_ns=$apiNon.json.prompt_eval_duration;warm_ttft_ns=$probe.json.prompt_eval_duration;decode_tokens=$apiNon.json.eval_count;decode_duration_ns=$apiNon.json.eval_duration;decode_tokens_per_second=$decodeTps} - }finally{if($server-and-not$server.HasExited){Stop-Process -Id $server.Id -Force};$record.rest.server_log=(Get-Content -Raw $serverLog -ErrorAction SilentlyContinue)} + }finally{if($server-and-not$server.HasExited){Stop-Process -Id $server.Id -Force};Mark 'server stopped';$record.rest.server_log=(Get-Content -Raw $serverLog -ErrorAction SilentlyContinue)} if($record.rest.server_log-match '(?i)CPU fallback|phi4_npu|Q4NX'){throw 'fallback backend appeared in server log'} + if($SkipCli){throw 'diagnostic -SkipCli run: the CLI matrix was not executed, so this record cannot report success'} $record.passed=$true -}catch{$record.failures+=($_|Out-String);throw}finally{$record.finished=(Get-Date).ToString('o');$record|ConvertTo-Json -Depth 12|Set-Content $outPath -Encoding UTF8} +}catch{$t=ErrText $_;$record.failures+=$t;Mark ('FAILURE: '+$t);throw}finally{$record.finished=(Get-Date).ToString('o');Mark 'writing record';WriteRecord $record $outPath;Mark 'record written'} From 1df3566d795fb458aa7448e99712084198d17f61 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 00:51:24 -0700 Subject: [PATCH 33/37] feat: report model load time, and account for it on the AIE4 path Model load is the one phase no profiler covered, and on this backend it is the largest single cost a user waits through. The runner now reports it for every backend, and the AIE4 engine breaks its own load into shape planning, GGUF resolution, host preparation, weight requantization and device allocation, printed when FLM_AIE4_PROFILE_LOAD is set. Co-Authored-By: Claude Opus 5 (1M context) --- src/common/corelib/phi4_corelib_aie4.cpp | 45 ++++++++++++++++++++++++ src/runner/runner.cpp | 17 +++++++++ 2 files changed, 62 insertions(+) diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp index e2ab0783..fb4a29c8 100644 --- a/src/common/corelib/phi4_corelib_aie4.cpp +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -5,7 +5,12 @@ #include "models/phi4/phi4_corelib_shape_plan.hpp" #include #include +#include +#include +#include +#include #include +#include #include #include #include @@ -17,12 +22,46 @@ using namespace flm::corelib; std::string Name(std::size_t i, const char* suffix) { return "blk." + std::to_string(i) + suffix; } + +/// Load-time phase accounting. Model load on this backend is dominated by +/// requantizing every weight from Q8_0, and without a breakdown there is no way +/// to tell that from disk I/O or from shape planning. Set FLM_AIE4_PROFILE_LOAD +/// to print it; the timer itself always runs, it costs five clock reads. +struct LoadPhases { + std::chrono::steady_clock::time_point mark{std::chrono::steady_clock::now()}; + double shape_plan{}, tensor_resolve{}, host_prep{}, weight_create{}, device_tensors{}; + + double Lap() { + const auto now = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(now - mark).count(); + mark = now; + return seconds; + } + + void Report() const { + const char* enabled = std::getenv("FLM_AIE4_PROFILE_LOAD"); + if (!enabled || !*enabled || *enabled == '0') return; + const double total = shape_plan + tensor_resolve + host_prep + + weight_create + device_tensors; + std::ostringstream out; + out << std::fixed << std::setprecision(2) + << "[FLM] AIE4 load: " << total << " s total" + << " (shape plan " << shape_plan + << ", GGUF resolve " << tensor_resolve + << ", host prep " << host_prep + << ", weight requantize " << weight_create + << ", device tensors " << device_tensors << ")"; + std::cout << out.str() << std::endl; + } +}; } struct phi4_corelib_aie4::Impl { std::shared_ptr package; std::shared_ptr runtime; std::shared_ptr api; + // Declared before `plan` so it starts before the initializer list builds it. + LoadPhases phases; Phi4ShapePlan plan; std::uint32_t max_length; int position{}; @@ -46,6 +85,7 @@ struct phi4_corelib_aie4::Impl { if (!runtime || !api) throw std::invalid_argument("corelib runtime is null"); if (!maximum || maximum > kMaxSequenceLength) throw std::invalid_argument("Phi-4 maximum length must be in 1..4096"); + phases.shape_plan = phases.Lap(); // Validate and capture every mapped span before the first device create. embedding = package->RequireQ8("token_embd.weight", std::array{kVocabularySize,kHiddenSize}); @@ -60,6 +100,7 @@ struct phi4_corelib_aie4::Impl { ow[i]=package->RequireQ8(Name(i,".attn_output.weight"),std::array{kHiddenSize,kHiddenSize}); dw[i]=package->RequireQ8(Name(i,".ffn_down.weight"),std::array{kHiddenSize,kIntermediateSize}); } + phases.tensor_resolve = phases.Lap(); std::optional factors; try { factors=package->RequireF32("rope_factors_short.weight",std::array{48}); } catch (const std::runtime_error&) {} @@ -70,6 +111,7 @@ struct phi4_corelib_aie4::Impl { const std::array epsf{kRmsEpsilon}; auto eps=ConvertF32ToBf16(epsf); first_norm_scale = an[0]; + phases.host_prep = phases.Lap(); auto lease=runtime->AcquireExecution(); void* raw=nullptr; api->Check(api->functions().create_stream(&raw),"ryzenai_corelib_create_stream"); stream=UniqueStream(api,raw); auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ @@ -90,6 +132,7 @@ struct phi4_corelib_aie4::Impl { mlp_weights[i]=UniqueSsMlpWeights(api,raw); } lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); + phases.weight_create = phases.Lap(); const auto& e=plan.maximum_extents(); const auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, e.ssmlp_rows}); @@ -114,6 +157,8 @@ struct phi4_corelib_aie4::Impl { for(std::size_t i=0;iCheck(api->functions().tensor_write(cosine.get(),ryzenai_corelib_data_type_fp32,rope.cosine.data(),rope.cosine.size(),0),"ryzenai_corelib_tensor_write cosine"); api->Check(api->functions().tensor_write(sine.get(),ryzenai_corelib_data_type_fp32,rope.sine.data(),rope.sine.size(),0),"ryzenai_corelib_tensor_write sine"); + phases.device_tensors = phases.Lap(); + phases.Report(); } void usable() const {if(poisoned)throw std::runtime_error("Phi-4 corelib engine is poisoned");} diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index dab30470..d80dc6f5 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -21,6 +21,19 @@ #include #include +namespace { +/// \brief Report how long loading the model took. +/// \note Model load is the one phase no profiler covers, and on backends that +/// repack weights at load it dominates the time to first usable prompt. +void report_load_time(std::chrono::steady_clock::time_point started) { + const double seconds = + std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + std::ostringstream message; + message << std::fixed << std::setprecision(2) << "Model loaded in " << seconds << " s"; + header_print("FLM", message.str()); +} +} // namespace + /// \brief Command map for command line input std::map cmd_map = { {"/set", CMD_SET}, @@ -73,7 +86,9 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra // header_print("ASR", asr_supported); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); @@ -452,7 +467,9 @@ void Runner::cmd_load(std::vector& input_list) { auto [new_tag, model_info] = this->supported_models.get_model_info(this->tag); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); From 53b625bb984c355c6c4ec95ad8fe528644ca73c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 01:32:32 -0700 Subject: [PATCH 34/37] fix: report load time on the server path too, not only the CLI The helper was local to the runner, so `flm serve` -- the path that actually waits 45 s on this backend -- reported nothing. Move it to debug_utils and call it from ensure_model_loaded, which is the single funnel for the server's initial load and every later model switch. Co-Authored-By: Claude Opus 5 (1M context) --- src/include/utils/debug_utils.hpp | 14 ++++++++++++++ src/runner/runner.cpp | 13 ------------- src/server/rest_handler.cpp | 2 ++ 3 files changed, 16 insertions(+), 13 deletions(-) diff --git a/src/include/utils/debug_utils.hpp b/src/include/utils/debug_utils.hpp index 9189b826..6106ae21 100644 --- a/src/include/utils/debug_utils.hpp +++ b/src/include/utils/debug_utils.hpp @@ -5,6 +5,7 @@ /// \version 0.9.24 /// \note This file contains the debug utilities for the FastFlowLM project. #pragma once +#include #include #include #include @@ -181,3 +182,16 @@ inline std::string size_t_to_string(size_t size){ return std::to_string(size / (1024 * 1024 * 1024)) + "G"; } } + +/// \brief Report how long loading a model took. +/// \param started the time point captured immediately before load_model +/// \note Model load is the one phase no profiler covers, and on backends that +/// repack weights at load it dominates the time to a first usable prompt. Call +/// this from every path that loads a model so the CLI and the server agree. +inline void report_load_time(std::chrono::steady_clock::time_point started) { + const double seconds = + std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + std::ostringstream message; + message << std::fixed << std::setprecision(2) << "Model loaded in " << seconds << " s"; + header_print("FLM", message.str()); +} diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index d80dc6f5..dc7a56e5 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -21,19 +21,6 @@ #include #include -namespace { -/// \brief Report how long loading the model took. -/// \note Model load is the one phase no profiler covers, and on backends that -/// repack weights at load it dominates the time to first usable prompt. -void report_load_time(std::chrono::steady_clock::time_point started) { - const double seconds = - std::chrono::duration(std::chrono::steady_clock::now() - started).count(); - std::ostringstream message; - message << std::fixed << std::setprecision(2) << "Model loaded in " << seconds << " s"; - header_print("FLM", message.str()); -} -} // namespace - /// \brief Command map for command line input std::map cmd_map = { {"/set", CMD_SET}, diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index c939ae2b..699843af 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -415,7 +415,9 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { auto [new_ensure_tag, model_info] = supported_models.get_model_info(ensure_tag); auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); auto_chat_engine->load_model(supported_models.get_model_path(new_ensure_tag), model_info, ctx_length, preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); From a55edfde266eb2925978ee24b45bd0cc07a5d427 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 01:58:51 -0700 Subject: [PATCH 35/37] docs: account for the 45 s startup, measured rather than assumed Engine load is only 16 s of it. The startup SHA-256 check over the 4 GB GGUF is ~28 s, 62% of the wait, and happens before the engine is constructed. Weight requantization is 15 s. Everything else is under a quarter of a second. The integrity check is also ~8x slower than the work needs: Get-FileHash over the same file on the same machine takes 3.67 s against ~28 s for calculate_file_sha256, which uses a portable pure-C++ SHA-256 with no hardware acceleration. That cost is paid by every model on every startup and pull, not just this one. Co-Authored-By: Claude Opus 5 (1M context) --- docs/docs/benchmarks/phi4_results.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/docs/benchmarks/phi4_results.md b/docs/docs/benchmarks/phi4_results.md index 1747bcae..ac1691eb 100644 --- a/docs/docs/benchmarks/phi4_results.md +++ b/docs/docs/benchmarks/phi4_results.md @@ -68,12 +68,25 @@ These are **descriptive measurements from a single acceptance run**, not a bench | Metric | Value | Conditions | |---|---|---| -| Model load to serving | **44.2 / 47.4 / 49.1 s** | three consecutive fresh `flm serve` processes, timed from launch to the first successful `/api/version`. All 161 weights are requantized from Q8_0 at load. | +| Model load to serving | **44.2 / 47.4 / 49.1 s** | three consecutive fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Broken down below. | | Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup | | Warm TTFT | **65.0 ms** | subsequent prompts in the same process | | Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens | | Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process | +### Where the ~45 s of startup goes + +Measured with `FLM_AIE4_PROFILE_LOAD=1`, two fresh `flm serve` processes: + +| Phase | Time | Share | +|---|---|---| +| Startup integrity check — SHA-256 over the 4 GB GGUF and the three small files | **~28 s** | 62% | +| Weight requantization — 161 objects from Q8_0, serially | **15.3 / 14.9 s** | 33% | +| Shape plan | 0.05 s | | +| GGUF resolve, host prep, device tensors | < 0.2 s | | + +Two things follow. First, `load_model` itself is only **16.8 / 16.0 s**; the majority of what a user waits through happens before the engine is even constructed. Second, the integrity check is far slower than the work requires: `Get-FileHash -Algorithm SHA256` over the same 4 GB file on the same machine takes **3.67 s**, against ~28 s for `calculate_file_sha256`, which uses a portable pure-C++ SHA-256 with no hardware acceleration. That ~8× gap is not specific to this model or this backend — it is paid on every startup check and every pull, for every model. + **Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed. The **5.4×** spread between warm TTFT (65 ms) and cold TTFT (4.21 s), and the **1.7×** spread between the REST and warm-CLI decode figures, are both unexplained by anything measured here. Treat single-run differences below roughly 2× as noise. From 95c5c82faa13711742d07d8b986a1fb8a94f20fa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 02:18:01 -0700 Subject: [PATCH 36/37] perf: stop re-hashing the model at startup, and give the packer a thread hint Two independent costs in the ~45 s startup, measured not assumed. The larger one is the startup integrity check: is_model_downloaded re-hashed every pinned file on every launch, ~28 s of SHA-256 over the 4 GB GGUF, 62% of the wait. That is a pull-time concern; the parameter to skip it already existed and simply was not passed. The run and serve paths now ask for status only. `flm pull` and `flm check` are unchanged and still verify in full, so a corrupt file is still caught -- at the next explicit check rather than at every launch. The smaller one is the packer. corelib treats a threads hint of 0 as ONE, deliberately, and this requantizing path is compute-bound and scales with it. The hint is per-create and the creates stay serialized. corelib's header records that loading a model with 8 CONCURRENT creates on this entry point failed 2 of 10 with all-zero output -- token id 0 at every step -- against 0 of 10 serialized and 0 of 10 for the ONNX path, with attribution open and the leading hypothesis an incomplete host-to-device sync. That is the faster configuration and it is deliberately not taken. Co-Authored-By: Claude Opus 5 (1M context) --- src/common/corelib/phi4_corelib_aie4.cpp | 4 ++-- .../models/phi4/phi4_corelib_constants.hpp | 8 +++++++ src/runner/runner.cpp | 4 ++-- src/server/rest_handler.cpp | 6 ++--- .../test_model_downloader.cpp | 24 +++++++++++++++++++ .../phi4_corelib_aie4/test_phi4_engine.cpp | 13 +++++++--- 6 files changed, 49 insertions(+), 10 deletions(-) diff --git a/src/common/corelib/phi4_corelib_aie4.cpp b/src/common/corelib/phi4_corelib_aie4.cpp index fb4a29c8..8605b89d 100644 --- a/src/common/corelib/phi4_corelib_aie4.cpp +++ b/src/common/corelib/phi4_corelib_aie4.cpp @@ -117,7 +117,7 @@ struct phi4_corelib_aie4::Impl { auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ ryzenai_corelib_matmul_bf16_weights_desc d{kk,nn,kRequantizedGroupSize,false}; ryzenai_corelib_matmul_bf16_gguf_components c{tv.bytes.data(),ryzenai_corelib_gguf_quant_type_q8_0}; void* p=nullptr; - api->Check(api->functions().matmul_weights_create_gguf_requantized(&d,&c,0,&p),"ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized "+label); + api->Check(api->functions().matmul_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&p),"ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized "+label); return UniqueMatMulWeights(api,p); }; for(std::size_t i=0;iCheck(api->functions().ssmlp_weights_create_gguf_requantized(&d,&c,0,&raw),"ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized layer "+std::to_string(i)); + api->Check(api->functions().ssmlp_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&raw),"ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized layer "+std::to_string(i)); mlp_weights[i]=UniqueSsMlpWeights(api,raw); } lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); diff --git a/src/include/models/phi4/phi4_corelib_constants.hpp b/src/include/models/phi4/phi4_corelib_constants.hpp index 92ed9a44..e34e87f6 100644 --- a/src/include/models/phi4/phi4_corelib_constants.hpp +++ b/src/include/models/phi4/phi4_corelib_constants.hpp @@ -17,5 +17,13 @@ inline constexpr std::int64_t kMaxSequenceLength = 4096; inline constexpr std::int64_t kModelContextLength = 131072; inline constexpr std::int64_t kMaxDecodeWindow = 4095; inline constexpr std::uint32_t kRequantizedGroupSize = 64; +/// Intra-packer threads for the Q8_0 requantizing creates. corelib treats 0 as +/// ONE deliberately; this path is compute-bound and scales with the hint. +/// +/// This is the per-create hint, NOT concurrent creates. corelib documents that +/// loading a model with 8 CONCURRENT creates on this entry point failed 2 of 10 +/// with all-zero output, against 0 of 10 serialized, with attribution open. The +/// creates therefore stay serialized. +inline constexpr std::uint32_t kRequantizeThreads = 8; inline constexpr float kRmsEpsilon = 1.0e-5f; } // namespace flm::phi4 diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index dc7a56e5..c81e52a9 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -58,7 +58,7 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra this->tag = auto_model.first; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -435,7 +435,7 @@ void Runner::cmd_load(std::vector& input_list) { if (model_name != this->tag) { this->tag = model_name; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index 699843af..ec692ac8 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -402,7 +402,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { std::pair> auto_model = get_auto_model(ensure_tag, this->supported_models, &this->npu_device_inst); auto_chat_engine = std::move(auto_model.second); ensure_tag = auto_model.first; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -441,7 +441,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -473,7 +473,7 @@ void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { void RestHandler::ensure_embed_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (this->downloader.is_model_downloaded(ensure_tag)) { + switch (this->downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: diff --git a/src/test/phi4_corelib_aie4/test_model_downloader.cpp b/src/test/phi4_corelib_aie4/test_model_downloader.cpp index 65273cdd..804bd8b7 100644 --- a/src/test/phi4_corelib_aie4/test_model_downloader.cpp +++ b/src/test/phi4_corelib_aie4/test_model_downloader.cpp @@ -290,6 +290,29 @@ void TestCheckHashesPinnedFilesExactlyOnce() { TEST_REQUIRE(CountOccurrences(output.str(), "Checking file:") == 4); } +void TestStartupStatusDoesNotRehashButCheckStillDoes() { + // Re-hashing the 4 GB GGUF on every launch cost ~28 s, 62% of startup, and + // buys nothing a pull-time verification has not already established. The + // run/serve paths ask for status only; `flm check` remains the full check. + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + + std::ostringstream fast; + auto* previous = std::cout.rdbuf(fast.rdbuf()); + const auto fast_status = downloader.is_model_downloaded("test-model:1b", false, true); + std::cout.rdbuf(previous); + TEST_REQUIRE(fast_status == ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(CountOccurrences(fast.str(), "Checking file:") == 0); + + std::ostringstream full; + previous = std::cout.rdbuf(full.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(full.str(), "Checking file:") == 4); +} + download_utils::DownloadRequest Request(const fs::path& source, const fs::path& destination, std::uint64_t size, std::string hash) { return {FileUrl(source), destination, size, download_utils::HashAlgorithm::Sha256, std::move(hash)}; @@ -358,6 +381,7 @@ int main() { RunTest(TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks, "modelscope rejection"); RunTest(TestCheckHashesPinnedFilesExactlyOnce, "single check verification"); + RunTest(TestStartupStatusDoesNotRehashButCheckStillDoes, "startup status skips rehash"); RunTest(TestResumeAppendsToPartThenAtomicallyPromotes, "resume and promote"); RunTest(TestWrongSizeOrHashNeverReplacesAValidFinalFile, "invalid transfer isolation"); RunTest(TestInterruptedTransferKeepsPartForNextResume, "interrupted transfer"); diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp index acc1213b..48ae7d65 100644 --- a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -1,4 +1,5 @@ #include "models/phi4/phi4_corelib_aie4.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" #include "models/phi4/phi4_corelib_host.hpp" #include "fake_corelib.hpp" #include "gguf_fixture.hpp" @@ -84,11 +85,17 @@ void TestEngineCreatesExactly129MatmulAnd32SsmlpWeights() { TEST_REQUIRE(std::none_of(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; })); } -void TestEveryProjectionUsesQ8RequantizedGroup64Threads0() { +void TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint() { + // corelib treats threads 0 as ONE deliberately, and this requantizing path + // is compute-bound and scales with the hint. The hint is per-create; the + // creates themselves stay serialized, which + // TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate pins -- + // corelib records 8 CONCURRENT creates on this entry point failing 2 of 10 + // with all-zero output, against 0 of 10 serialized. Harness h; for (const auto& record : fake_corelib::GetState().weight_creates) { TEST_REQUIRE(record.group_size == 64); - TEST_REQUIRE(record.threads == 0); + TEST_REQUIRE(record.threads == flm::phi4::kRequantizeThreads); } TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_matmul_bf16_weights_create_gguf"] == 0); TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_ssmlp_bf16_weights_create_gguf"] == 0); @@ -506,7 +513,7 @@ int main() { RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); RUN_TEST(TestEngineCreatesExactly129MatmulAnd32SsmlpWeights); - RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64Threads0); + RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint); RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); RUN_TEST(TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates); From 07ee52cd7138a37d3e3efb1415e515bcffb92f42 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Sat, 12 Sep 2026 02:25:14 -0700 Subject: [PATCH 37/37] docs: startup is 5 s, not 45 s Re-measured after removing the per-launch re-hash and giving the packer a thread hint: process launch to serving 5.1/5.3 s against 44.9/46.3 s, with requantization at 2.5/3.0 s against 15.3/14.9 s. Output re-verified after the packing change; no degeneration. Co-Authored-By: Claude Opus 5 (1M context) --- docs/docs/benchmarks/phi4_results.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/docs/benchmarks/phi4_results.md b/docs/docs/benchmarks/phi4_results.md index ac1691eb..bd063990 100644 --- a/docs/docs/benchmarks/phi4_results.md +++ b/docs/docs/benchmarks/phi4_results.md @@ -68,24 +68,31 @@ These are **descriptive measurements from a single acceptance run**, not a bench | Metric | Value | Conditions | |---|---|---| -| Model load to serving | **44.2 / 47.4 / 49.1 s** | three consecutive fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Broken down below. | +| Model load to serving | **5.1 / 5.3 s** | fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Was 44–49 s at the accepted commit; see below. | | Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup | | Warm TTFT | **65.0 ms** | subsequent prompts in the same process | | Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens | | Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process | -### Where the ~45 s of startup goes +### Startup: 45 s → 5 s -Measured with `FLM_AIE4_PROFILE_LOAD=1`, two fresh `flm serve` processes: +The acceptance run measured 44–49 s to serving. Profiling it with `FLM_AIE4_PROFILE_LOAD=1` found two independent costs, both since fixed: -| Phase | Time | Share | +| Phase | Before | After | |---|---|---| -| Startup integrity check — SHA-256 over the 4 GB GGUF and the three small files | **~28 s** | 62% | -| Weight requantization — 161 objects from Q8_0, serially | **15.3 / 14.9 s** | 33% | -| Shape plan | 0.05 s | | -| GGUF resolve, host prep, device tensors | < 0.2 s | | +| Startup integrity check — SHA-256 over the 4 GB GGUF | ~28 s (62%) | **0 s** — not run | +| Weight requantization — 161 objects from Q8_0 | 15.3 / 14.9 s | **2.5 / 3.0 s** | +| Shape plan | 0.05 s | 0.05 s | +| GGUF resolve, host prep, device tensors | < 0.2 s | < 0.2 s | +| **Process launch to serving** | **44.9 / 46.3 s** | **5.1 / 5.3 s** | -Two things follow. First, `load_model` itself is only **16.8 / 16.0 s**; the majority of what a user waits through happens before the engine is even constructed. Second, the integrity check is far slower than the work requires: `Get-FileHash -Algorithm SHA256` over the same 4 GB file on the same machine takes **3.67 s**, against ~28 s for `calculate_file_sha256`, which uses a portable pure-C++ SHA-256 with no hardware acceleration. That ~8× gap is not specific to this model or this backend — it is paid on every startup check and every pull, for every model. +The integrity check was re-hashing every pinned file on every launch — a pull-time concern on the startup path. `flm pull` and `flm check` still verify in full; only the run and serve paths were changed to ask for status alone. + +The packer was being given a threads hint of 0, which corelib treats as ONE deliberately. This requantizing path is compute-bound and scales with the hint, so 8 brings it to 2.5–3.0 s — within range of the 2.2 s that `python/phi4_driver.py` reports for the same 161 weights, and reached **without** the 8-concurrent-creates configuration whose failure mode is documented in corelib's header (2 of 10 loads producing all-zero output, attribution open). The creates remain serialized. + +Output was re-verified after the change: `2+2` → `4`, `capital of France` → `Paris`, `primary color` → `Red.`, and a correct one-sentence description of AMD. No degeneration, no all-zero output. + +Separately, and **not** fixed: `calculate_file_sha256` uses a portable pure-C++ SHA-256 with no hardware acceleration, and takes ~28 s over 4 GB where `Get-FileHash` on the same machine takes **3.67 s**. That ~8× gap is not specific to this model or backend — it is still paid by `flm pull` and `flm check` for every model. **Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed.