feat(phi4): run Phi-4-mini-instruct Q8_0 GGUF on AIE4 via ryzenai-corelib

docs/docs/benchmarks/phi4_results.md

@@ -41,3 +41,74 @@ AMD Ryzen™ AI 7 350 (Kraken Point) with 32 GB DRAM; performance is comparable
 | **Model**        | **HW**       | **1k** | **2k** | **4k** | **8k** | **16k** | **32k** |
 |------------------|--------------------|--------:|--------:|--------:|--------:|---------:|---------:|
 | **Phi-4-mini-instruct**  | NPU (FLM)    | 643	| 787	| 857	| 809	| 644	| 447 | 
+
+---
+
+## 🧪 Phi-4-mini-instruct Q8_0 GGUF on AIE4 (`phi4-mini-it-aie4:4b`)
+
+These are **descriptive measurements from a single acceptance run**, not a benchmark sweep and not a pass threshold. They are not comparable to the tables above: the prompts here are 4–10 tokens, whereas those tables sweep 1k–32k, so the per-token rates are dominated by fixed overhead rather than by context length.
+
+### Provenance
+
+| | |
+|---|---|
+| Machine | `XCOMEDUSAD-43` |
+| CPU | `AMD Eng Sample: 100-000001713-33_N` |
+| NPU | `AMD XDNA(TM) NPU` |
+| OS | Microsoft Windows 11 Enterprise 10.0.26100 build 26100 |
+| Windows power scheme | Balanced (`381b4222-f694-41f0-9685-ff5bb260df2e`). The NPU power mode is separately set to `performance` by FLM at startup. |
+| FastFlowLM commit | `87721089097396579ec4529f50616a6c0e1c7b74` |
+| corelib commit / ABI | `3c35aebdefa3f0c2255668bab1be5648ece320f8` / `0.3.0` |
+| corelib DLL SHA-256 | `f404da219a3cc84d3334c265e09ba7987f0c4bcc1b1cedeac7c3c45a7be2c9ae` |
+| GGUF revision | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` |
+| Tokenizer/config revision | `cfbefacb99257ffa30c83adab238a50856ac3083` |
+| Run | 2026-09-12 00:34:19 → 00:52:02, `passed: true`, 0 failures |
+
+### Measurements
+
+| Metric | Value | Conditions |
+|---|---|---|
+| Model load to serving | **5.1 / 5.3 s** | fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Was 44–49 s at the accepted commit; see below. |
+| Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup |
+| Warm TTFT | **65.0 ms** | subsequent prompts in the same process |
+| Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens |
+| Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process |
+
+### Startup: 45 s → 5 s
+
+The acceptance run measured 44–49 s to serving. Profiling it with `FLM_AIE4_PROFILE_LOAD=1` found two independent costs, both since fixed:
+
+| Phase | Before | After |
+|---|---|---|
+| Startup integrity check — SHA-256 over the 4 GB GGUF | ~28 s (62%) | **0 s** — not run |
+| Weight requantization — 161 objects from Q8_0 | 15.3 / 14.9 s | **2.5 / 3.0 s** |
+| Shape plan | 0.05 s | 0.05 s |
+| GGUF resolve, host prep, device tensors | < 0.2 s | < 0.2 s |
+| **Process launch to serving** | **44.9 / 46.3 s** | **5.1 / 5.3 s** |
+
+The integrity check was re-hashing every pinned file on every launch — a pull-time concern on the startup path. `flm pull` and `flm check` still verify in full; only the run and serve paths were changed to ask for status alone.
+
+The packer was being given a threads hint of 0, which corelib treats as ONE deliberately. This requantizing path is compute-bound and scales with the hint, so 8 brings it to 2.5–3.0 s — within range of the 2.2 s that `python/phi4_driver.py` reports for the same 161 weights, and reached **without** the 8-concurrent-creates configuration whose failure mode is documented in corelib's header (2 of 10 loads producing all-zero output, attribution open). The creates remain serialized.
+
+Output was re-verified after the change: `2+2` → `4`, `capital of France` → `Paris`, `primary color` → `Red.`, and a correct one-sentence description of AMD. No degeneration, no all-zero output.
+
+Separately, and **not** fixed: `calculate_file_sha256` uses a portable pure-C++ SHA-256 with no hardware acceleration, and takes ~28 s over 4 GB where `Get-FileHash` on the same machine takes **3.67 s**. That ~8× gap is not specific to this model or backend — it is still paid by `flm pull` and `flm check` for every model.
+
+**Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed.
+
+The **5.4×** spread between warm TTFT (65 ms) and cold TTFT (4.21 s), and the **1.7×** spread between the REST and warm-CLI decode figures, are both unexplained by anything measured here. Treat single-run differences below roughly 2× as noise.
+
+### Functional results
+
+All from the same run:
+
+- `flm pull` / `flm check` — four pinned files, all SHA-256 verified; the model directory contains exactly those four.
+- CLI — 10/10 fresh-process load-and-generate cycles exited 0; `Backend: corelib_aie4_gguf` and the loaded DLL path reported in every one.
+- REST — `/api/chat` and `/v1/chat/completions` both 200, streaming and non-streaming.
+- Cancellation — an in-flight stream cancelled cleanly; the next request returned 200 on the same server.
+- Capacity boundary — a request totalling 4096 tokens is rejected with **HTTP 400** before submission (`rendered prompt has 4 tokens and requested output has 4092 tokens`); a 4095-token request is admitted.
+- No CPU or NPU2 fallback appears in the server log at any point.
+
+### Known issue
+
+One `/api/chat` reply to `What is 2+2?` came back as a truncated markdown image URL (`![](https://media.giphy.com/media/kZl76FZgu`, `done_reason: length`) instead of an answer. The identical prompt answered correctly on three other occasions in the same session, including the recovery request in the same run, so this looks like sampling nondeterminism rather than a routing fault — but it is a single-observation defect, it is not understood, and it is recorded rather than smoothed over.

docs/docs/models/phi.md

@@ -22,4 +22,67 @@ parent: Models
 flm run phi4-mini-it:4b
 ```
 
+---
+
+## 🧪 Model Card: Phi-4-mini-instruct on AIE4 (developer preview)
+
+- **Tag:** `phi4-mini-it-aie4:4b`
+- **Backend:** `corelib_aie4_gguf` — runs on AIE4 through AMD's `ryzenai_corelib.dll`
+- **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped.
+- **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source.
+- **Usable generation window:** 4095 tokens — the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why.
+- **Availability:** Windows only, and this is a **developer build**. The AIE4 runtime is not packaged by the MSI or Inno installer; you supply the DLL yourself.
+
+This tag pulls from two pinned repositories, because the GGUF publisher does not ship the tokenizer files FastFlowLM's tokenizer frontend consumes:
+
+| File | Repository | Revision |
+|---|---|---|
+| `Phi-4-mini-instruct.Q8_0.gguf` | [`unsloth/Phi-4-mini-instruct-GGUF`](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF) | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` |
+| `tokenizer.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` |
+| `tokenizer_config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` |
+| `config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` |
+
+All four are SHA-256 verified before the download is promoted, and the pulled directory contains exactly these four files.
+
+### Building
+
+The AIE4 path is compiled only when you ask for it. With the option off, the binary contains no reference to corelib at all.
+
+From `FastFlowLM/src`, in a Visual Studio developer command prompt:
+
+```powershell
+$env:RYZENAI_CORELIB_INCLUDE_DIR = 'C:/path/to/ryzenai-corelib/install/include'
+cmake --preset windows-aie4          # sets FLM_ENABLE_CORELIB_AIE4=ON, builds into src/build-aie4
+cmake --build --preset windows-aie4
+```
+
+The `windows-aie4` preset reads `RYZENAI_CORELIB_INCLUDE_DIR` from the environment, so set it before configuring. The configure step also locates a Boost include directory, and hard-errors if the option is enabled on a non-Windows host. Everything else — XRT, FFmpeg, curl, FFTW — is the ordinary FastFlowLM dependency set; the AIE4 option does not relax any of it.
+
+### Pointing FastFlowLM at the runtime
+
+`flm.exe` never links `ryzenai_corelib.lib`; the DLL is resolved and loaded at run time, by absolute path:
+
+1. `FLM_AIE4_CORELIB_PATH`, if set. It must be an **absolute path to a `.dll` file** — a relative path or a directory is rejected outright.
+2. Otherwise `<directory containing flm.exe>\aie4\ryzenai_corelib.dll`.
+
+The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. The DLL's own dependency directory must be reachable on `PATH`.
+
+```powershell
+$env:FLM_AIE4_CORELIB_PATH = 'C:/path/to/ryzenai_corelib.dll'
+flm pull phi4-mini-it-aie4:4b
+flm run  phi4-mini-it-aie4:4b
+```
+
+### Why the context is 4096, and why the usable window is one less
+
+Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults to 32k. This backend gives you 4095. That is a real functional regression and it has two separate causes, which are worth keeping apart.
+
+**The 4096 ceiling is a correctness boundary, not a buffer size.** 4096 is exactly Phi-4-mini's `rope.scaling.original_context_length`. LongRoPE selects its factors by *sequence length*, not per position: at or below the original length the short factors apply, above it the long ones do. This implementation derives only the short branch, so 4096 is the point past which the rope tables would silently be wrong. It is enforced rather than assumed — loading fails with `invalid Phi-4 RoPE metadata` unless the GGUF reports `rope.scaling.original_context_length` of exactly 4096. Raising this ceiling means deriving the long factors, not enlarging an array.
+
+**The extra −1 is this frontend's own conservatism.** `kMaxDecodeWindow` is 4095, one below the attention window, so that any request the server admits is guaranteed to have room to finish rather than failing partway. It costs exactly one token and it is not imposed by corelib.
+
+### No fallback
+
+Backend selection is explicit: it comes from `execution_backend` in the model catalog and is never inferred from hardware, filename, or quantization level. If corelib is missing, unloadable, or the wrong version, this tag **fails to load with a diagnostic** — it will not quietly fall back to CPU or to the NPU2/Q4NX backend. A build without AIE4 support, and an AIE4 build with no DLL present, both still start and run every other model, including `phi4-mini-it:4b`.
+
 ---

src/CMakeLists.txt

@@ -48,6 +48,19 @@ set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
 # ———————————————————————————————————————————————
 option(FLM_USE_HRX "Use the HRX amdxdna NPU runtime instead of XRT (0=XRT default, 1=HRX)" OFF)
 option(FLM_PORTABLE_BUILD "Build portable distribution with bundled runtime libraries" OFF)
+option(FLM_ENABLE_CORELIB_AIE4
+       "Enable Phi-4 Q8_0 GGUF execution through ryzenai-corelib" OFF)
+
+if(FLM_ENABLE_CORELIB_AIE4)
+    if(NOT WIN32)
+        message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows")
+    endif()
+    find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED)
+    find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp
+        HINTS "$ENV{CONDA_PREFIX}/Library/include"
+              "$ENV{USERPROFILE}/anaconda3/Library/include"
+              "C:/dev/boost_1_88_0" REQUIRED)
+endif()
 
 if(FLM_USE_HRX)
     set(FLM_RUNTIME_NAME "hrx")
@@ -237,6 +250,7 @@ add_subdirectory(${CMAKE_SOURCE_DIR}/../third_party/tokenizers-cpp
 # ———————————————————————————————————————————————
 file(GLOB SOURCES "src/*.cpp" "runner/*.cpp" "common/*.cpp" "common/*/*.cpp" "server/*.cpp" "pull/*.cpp" )
 file(GLOB HEADERS "include/*.hpp" "runner/*.hpp" "common/*.hpp" "common/*/*.hpp" "server/*.hpp" "pull/*.hpp")
+list(FILTER SOURCES EXCLUDE REGEX ".*/common/corelib/.*\\.cpp$")
 
 # Exclude files that depend on missing libraries for Linux
 if(NOT WIN32)
@@ -269,6 +283,16 @@ endif()
 
 add_executable(flm ${SOURCES} ${HEADERS})
 
+if(FLM_ENABLE_CORELIB_AIE4)
+    include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake")
+    add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES})
+    target_include_directories(flm_corelib_aie4 PUBLIC
+        "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}"
+        "${XRT_INCLUDE_DIR}" "${FLM_CORELIB_BOOST_INCLUDE_DIR}")
+    target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1)
+    target_link_libraries(flm PRIVATE flm_corelib_aie4)
+endif()
+
 if(WIN32)
     if(VCPKG_TOOLCHAIN)
         # A vcpkg toolchain is active (e.g. the rocm-npu-staging dev.py build or

src/CMakePresets.json

@@ -53,6 +53,17 @@
         "CMAKE_BUILD_TYPE": "Release"
       }
     },
+    {
+      "name": "windows-aie4",
+      "displayName": "Windows AIE4",
+      "description": "Windows build with optional dynamically loaded ryzenai-corelib support",
+      "inherits": "windows-default",
+      "binaryDir": "${sourceDir}/build-aie4",
+      "cacheVariables": {
+        "FLM_ENABLE_CORELIB_AIE4": "ON",
+        "RYZENAI_CORELIB_INCLUDE_DIR": "$env{RYZENAI_CORELIB_INCLUDE_DIR}"
+      }
+    },
     {
       "name": "windows-vs18",
       "displayName": "Windows VS18",
@@ -85,6 +96,14 @@
       "configurePreset": "windows-default",
       "configuration": "Release",
       "jobs": 4
+    },
+    {
+      "name": "windows-aie4",
+      "displayName": "Windows AIE4 Build",
+      "description": "Build the optional dynamically loaded AIE4 runtime in Release",
+      "configurePreset": "windows-aie4",
+      "configuration": "Release",
+      "jobs": 4
     }
   ]
 }

src/common/AutoModel/automodel.cpp

@@ -8,6 +8,14 @@
 #include "AutoModel/automodel.hpp"
 
 
+ModelRequestError::ModelRequestError(
+    int http_code, bool session_cleared, std::string message)
+    : std::runtime_error(std::move(message)),
+      http_code_(http_code), session_cleared_(session_cleared) {}
+
+int ModelRequestError::http_code() const noexcept { return http_code_; }
+bool ModelRequestError::session_cleared() const noexcept { return session_cleared_; }
+
 AutoModel::AutoModel(flm_rt::device* npu_device_inst, std::string current_model) {
     this->npu_device_inst = npu_device_inst;
     this->current_model = current_model;
@@ -128,34 +136,49 @@ void AutoModel::_shared_load_model(std::string model_path, json model_info, int
         header_print("FLM", "Model already loaded: " << this->model_path);
         return;
     }
+    const int context_length = default_context_length != -1
+        ? default_context_length
+        : model_info["default_context_length"].get<int>();
+    this->_shared_initialize_model_state(
+        std::move(model_path), std::move(model_info), context_length);
+    this->_shared_initialize_legacy_npu(enable_preemption);
+}
 
-    this->model_path = model_path;
+void AutoModel::_shared_initialize_model_state(
+    std::string model_path, json, int context_length) {
+    this->model_path = std::move(model_path);
     header_print("FLM", "Loading model: " << this->model_path);
     this->lm_config = std::make_unique<LM_Config>();
     this->lm_config->from_pretrained(this->model_path);
-    if (this->npu_device_inst == nullptr) {
-        header_print("ERROR", "NPU device instance is nullptr");
-        exit(1);
-    }
-    this->npu = std::make_unique<npu_xclbin_manager>(npu_device::device_npu2, this->npu_device_inst, enable_preemption);
-    this->enable_preemption = enable_preemption;
-    // Set context length: use provided value if not -1, otherwise use model default
-    if (default_context_length != -1) {
-        this->MAX_L = default_context_length;
-    } else {
-        this->MAX_L = model_info["default_context_length"];
-    }
-
+    this->MAX_L = context_length;
     this->is_model_loaded = true;
-
     this->token_history.clear();
     this->token_history.reserve(this->MAX_L);
     this->tokenizer = std::make_unique<Tokenizer>(this->model_path);
-
     this->last_token = -1;
     this->total_tokens = 0;
 }
 
+void AutoModel::_shared_initialize_legacy_npu(bool enable_preemption) {
+    if (this->npu_device_inst == nullptr) {
+        header_print("ERROR", "NPU device instance is nullptr");
+        exit(1);
+    }
+    this->npu = std::make_unique<npu_xclbin_manager>(
+        npu_device::device_npu2, this->npu_device_inst, enable_preemption);
+    this->enable_preemption = enable_preemption;
+}
+
+std::string AutoModel::generate_with_prompt(
+    chat_meta_info_t& meta_info,
+    lm_uniform_input_t& input,
+    int length_limit,
+    std::ostream& os,
+    std::function<bool()> is_cancelled) {
+    if (!insert(meta_info, input, is_cancelled)) return {};
+    return generate(meta_info, length_limit, os, std::move(is_cancelled));
+}
+
 bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector<int>& tokens, std::function<bool()> is_cancelled, void* payload, int first_len_run) {
 
     // print token history
@@ -231,6 +254,14 @@ buffer<bf16> AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector
     }
     buffer<bf16> y;
     if (max_prefill_len < 512) {
+        if (is_cancelled()) {
+            meta_info.stop_reason = CANCEL_DETECTED;
+            buffer_.clear();
+            current_mode_ = StreamEventType::CONTENT;
+            tool_name_.clear();
+            is_in_tool_block_ = false;
+            return y;
+        }
         y = this->lm_engine->prefill(tokens, payload);
     }
     else{
@@ -242,19 +273,18 @@ buffer<bf16> AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector
         }
         int chunks = (tokens.size() + max_prefill_len - 1) / max_prefill_len;
         for (int i = 0; i < chunks; i++) {
+            int start = i * max_prefill_len;
+            int end = std::min(static_cast<int>(tokens.size()), (i + 1) * max_prefill_len);
+            std::vector<int> chunk_tokens(tokens.begin() + start, tokens.begin() + end);
+            header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens");
             if (is_cancelled()) {
                 meta_info.stop_reason = CANCEL_DETECTED;
-                // reset stream content 
                 buffer_.clear();
                 current_mode_ = StreamEventType::CONTENT;
                 tool_name_.clear();
                 is_in_tool_block_ = false;
                 break;
             }
-            int start = i * max_prefill_len;
-            int end = std::min(static_cast<int>(tokens.size()), (i + 1) * max_prefill_len);
-            std::vector<int> chunk_tokens(tokens.begin() + start, tokens.begin() + end);
-            header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens");
             buffer<bf16> chunk_y = this->lm_engine->prefill(chunk_tokens, (i == 0)? payload : nullptr);
             if (i == chunks - 1) {
                 y = chunk_y;