diff --git a/docs/docs/benchmarks/phi4_results.md b/docs/docs/benchmarks/phi4_results.md index 7d7de5ea..bd063990 100644 --- a/docs/docs/benchmarks/phi4_results.md +++ b/docs/docs/benchmarks/phi4_results.md @@ -41,3 +41,74 @@ AMD Ryzenβ„’ AI 7 350 (Kraken Point) with 32 GB DRAM; performance is comparable | **Model** | **HW** | **1k** | **2k** | **4k** | **8k** | **16k** | **32k** | |------------------|--------------------|--------:|--------:|--------:|--------:|---------:|---------:| | **Phi-4-mini-instruct** | NPU (FLM) | 643 | 787 | 857 | 809 | 644 | 447 | + +--- + +## πŸ§ͺ Phi-4-mini-instruct Q8_0 GGUF on AIE4 (`phi4-mini-it-aie4:4b`) + +These are **descriptive measurements from a single acceptance run**, not a benchmark sweep and not a pass threshold. They are not comparable to the tables above: the prompts here are 4–10 tokens, whereas those tables sweep 1k–32k, so the per-token rates are dominated by fixed overhead rather than by context length. + +### Provenance + +| | | +|---|---| +| Machine | `XCOMEDUSAD-43` | +| CPU | `AMD Eng Sample: 100-000001713-33_N` | +| NPU | `AMD XDNA(TM) NPU` | +| OS | Microsoft Windows 11 Enterprise 10.0.26100 build 26100 | +| Windows power scheme | Balanced (`381b4222-f694-41f0-9685-ff5bb260df2e`). The NPU power mode is separately set to `performance` by FLM at startup. | +| FastFlowLM commit | `87721089097396579ec4529f50616a6c0e1c7b74` | +| corelib commit / ABI | `3c35aebdefa3f0c2255668bab1be5648ece320f8` / `0.3.0` | +| corelib DLL SHA-256 | `f404da219a3cc84d3334c265e09ba7987f0c4bcc1b1cedeac7c3c45a7be2c9ae` | +| GGUF revision | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| Tokenizer/config revision | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| Run | 2026-09-12 00:34:19 β†’ 00:52:02, `passed: true`, 0 failures | + +### Measurements + +| Metric | Value | Conditions | +|---|---|---| +| Model load to serving | **5.1 / 5.3 s** | fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Was 44–49 s at the accepted commit; see below. | +| Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup | +| Warm TTFT | **65.0 ms** | subsequent prompts in the same process | +| Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens | +| Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process | + +### Startup: 45 s β†’ 5 s + +The acceptance run measured 44–49 s to serving. Profiling it with `FLM_AIE4_PROFILE_LOAD=1` found two independent costs, both since fixed: + +| Phase | Before | After | +|---|---|---| +| Startup integrity check β€” SHA-256 over the 4 GB GGUF | ~28 s (62%) | **0 s** β€” not run | +| Weight requantization β€” 161 objects from Q8_0 | 15.3 / 14.9 s | **2.5 / 3.0 s** | +| Shape plan | 0.05 s | 0.05 s | +| GGUF resolve, host prep, device tensors | < 0.2 s | < 0.2 s | +| **Process launch to serving** | **44.9 / 46.3 s** | **5.1 / 5.3 s** | + +The integrity check was re-hashing every pinned file on every launch β€” a pull-time concern on the startup path. `flm pull` and `flm check` still verify in full; only the run and serve paths were changed to ask for status alone. + +The packer was being given a threads hint of 0, which corelib treats as ONE deliberately. This requantizing path is compute-bound and scales with the hint, so 8 brings it to 2.5–3.0 s β€” within range of the 2.2 s that `python/phi4_driver.py` reports for the same 161 weights, and reached **without** the 8-concurrent-creates configuration whose failure mode is documented in corelib's header (2 of 10 loads producing all-zero output, attribution open). The creates remain serialized. + +Output was re-verified after the change: `2+2` β†’ `4`, `capital of France` β†’ `Paris`, `primary color` β†’ `Red.`, and a correct one-sentence description of AMD. No degeneration, no all-zero output. + +Separately, and **not** fixed: `calculate_file_sha256` uses a portable pure-C++ SHA-256 with no hardware acceleration, and takes ~28 s over 4 GB where `Get-FileHash` on the same machine takes **3.67 s**. That ~8Γ— gap is not specific to this model or backend β€” it is still paid by `flm pull` and `flm check` for every model. + +**Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed. + +The **5.4Γ—** spread between warm TTFT (65 ms) and cold TTFT (4.21 s), and the **1.7Γ—** spread between the REST and warm-CLI decode figures, are both unexplained by anything measured here. Treat single-run differences below roughly 2Γ— as noise. + +### Functional results + +All from the same run: + +- `flm pull` / `flm check` β€” four pinned files, all SHA-256 verified; the model directory contains exactly those four. +- CLI β€” 10/10 fresh-process load-and-generate cycles exited 0; `Backend: corelib_aie4_gguf` and the loaded DLL path reported in every one. +- REST β€” `/api/chat` and `/v1/chat/completions` both 200, streaming and non-streaming. +- Cancellation β€” an in-flight stream cancelled cleanly; the next request returned 200 on the same server. +- Capacity boundary β€” a request totalling 4096 tokens is rejected with **HTTP 400** before submission (`rendered prompt has 4 tokens and requested output has 4092 tokens`); a 4095-token request is admitted. +- No CPU or NPU2 fallback appears in the server log at any point. + +### Known issue + +One `/api/chat` reply to `What is 2+2?` came back as a truncated markdown image URL (`![](https://media.giphy.com/media/kZl76FZgu`, `done_reason: length`) instead of an answer. The identical prompt answered correctly on three other occasions in the same session, including the recovery request in the same run, so this looks like sampling nondeterminism rather than a routing fault β€” but it is a single-observation defect, it is not understood, and it is recorded rather than smoothed over. diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 9b2c59bf..ba05c407 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -22,4 +22,67 @@ parent: Models flm run phi4-mini-it:4b ``` +--- + +## πŸ§ͺ Model Card: Phi-4-mini-instruct on AIE4 (developer preview) + +- **Tag:** `phi4-mini-it-aie4:4b` +- **Backend:** `corelib_aie4_gguf` β€” runs on AIE4 through AMD's `ryzenai_corelib.dll` +- **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped. +- **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source. +- **Usable generation window:** 4095 tokens β€” the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why. +- **Availability:** Windows only, and this is a **developer build**. The AIE4 runtime is not packaged by the MSI or Inno installer; you supply the DLL yourself. + +This tag pulls from two pinned repositories, because the GGUF publisher does not ship the tokenizer files FastFlowLM's tokenizer frontend consumes: + +| File | Repository | Revision | +|---|---|---| +| `Phi-4-mini-instruct.Q8_0.gguf` | [`unsloth/Phi-4-mini-instruct-GGUF`](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF) | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| `tokenizer.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `tokenizer_config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | + +All four are SHA-256 verified before the download is promoted, and the pulled directory contains exactly these four files. + +### Building + +The AIE4 path is compiled only when you ask for it. With the option off, the binary contains no reference to corelib at all. + +From `FastFlowLM/src`, in a Visual Studio developer command prompt: + +```powershell +$env:RYZENAI_CORELIB_INCLUDE_DIR = 'C:/path/to/ryzenai-corelib/install/include' +cmake --preset windows-aie4 # sets FLM_ENABLE_CORELIB_AIE4=ON, builds into src/build-aie4 +cmake --build --preset windows-aie4 +``` + +The `windows-aie4` preset reads `RYZENAI_CORELIB_INCLUDE_DIR` from the environment, so set it before configuring. The configure step also locates a Boost include directory, and hard-errors if the option is enabled on a non-Windows host. Everything else β€” XRT, FFmpeg, curl, FFTW β€” is the ordinary FastFlowLM dependency set; the AIE4 option does not relax any of it. + +### Pointing FastFlowLM at the runtime + +`flm.exe` never links `ryzenai_corelib.lib`; the DLL is resolved and loaded at run time, by absolute path: + +1. `FLM_AIE4_CORELIB_PATH`, if set. It must be an **absolute path to a `.dll` file** β€” a relative path or a directory is rejected outright. +2. Otherwise `\aie4\ryzenai_corelib.dll`. + +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. The DLL's own dependency directory must be reachable on `PATH`. + +```powershell +$env:FLM_AIE4_CORELIB_PATH = 'C:/path/to/ryzenai_corelib.dll' +flm pull phi4-mini-it-aie4:4b +flm run phi4-mini-it-aie4:4b +``` + +### Why the context is 4096, and why the usable window is one less + +Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults to 32k. This backend gives you 4095. That is a real functional regression and it has two separate causes, which are worth keeping apart. + +**The 4096 ceiling is a correctness boundary, not a buffer size.** 4096 is exactly Phi-4-mini's `rope.scaling.original_context_length`. LongRoPE selects its factors by *sequence length*, not per position: at or below the original length the short factors apply, above it the long ones do. This implementation derives only the short branch, so 4096 is the point past which the rope tables would silently be wrong. It is enforced rather than assumed β€” loading fails with `invalid Phi-4 RoPE metadata` unless the GGUF reports `rope.scaling.original_context_length` of exactly 4096. Raising this ceiling means deriving the long factors, not enlarging an array. + +**The extra βˆ’1 is this frontend's own conservatism.** `kMaxDecodeWindow` is 4095, one below the attention window, so that any request the server admits is guaranteed to have room to finish rather than failing partway. It costs exactly one token and it is not imposed by corelib. + +### No fallback + +Backend selection is explicit: it comes from `execution_backend` in the model catalog and is never inferred from hardware, filename, or quantization level. If corelib is missing, unloadable, or the wrong version, this tag **fails to load with a diagnostic** β€” it will not quietly fall back to CPU or to the NPU2/Q4NX backend. A build without AIE4 support, and an AIE4 build with no DLL present, both still start and run every other model, including `phi4-mini-it:4b`. + --- \ No newline at end of file diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index dd4d33fe..b2b597b4 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -48,6 +48,19 @@ set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) # β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€” option(FLM_USE_HRX "Use the HRX amdxdna NPU runtime instead of XRT (0=XRT default, 1=HRX)" OFF) option(FLM_PORTABLE_BUILD "Build portable distribution with bundled runtime libraries" OFF) +option(FLM_ENABLE_CORELIB_AIE4 + "Enable Phi-4 Q8_0 GGUF execution through ryzenai-corelib" OFF) + +if(FLM_ENABLE_CORELIB_AIE4) + if(NOT WIN32) + message(FATAL_ERROR "FLM_ENABLE_CORELIB_AIE4 currently requires Windows") + endif() + find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) + find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" + "C:/dev/boost_1_88_0" REQUIRED) +endif() if(FLM_USE_HRX) set(FLM_RUNTIME_NAME "hrx") @@ -237,6 +250,12 @@ add_subdirectory(${CMAKE_SOURCE_DIR}/../third_party/tokenizers-cpp # β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€”β€” file(GLOB SOURCES "src/*.cpp" "runner/*.cpp" "common/*.cpp" "common/*/*.cpp" "server/*.cpp" "pull/*.cpp" ) file(GLOB HEADERS "include/*.hpp" "runner/*.hpp" "common/*.hpp" "common/*/*.hpp" "server/*.hpp" "pull/*.hpp") +list(FILTER SOURCES EXCLUDE REGEX ".*/common/corelib/.*\\.cpp$") + +# Model sources live two levels deeper than the globs above reach, so pull +# them in explicitly; the corelib half is built separately, below. +include("${CMAKE_SOURCE_DIR}/common/models/models_sources.cmake") +list(APPEND SOURCES ${FLM_MODELS_FLM_SOURCES}) # Exclude files that depend on missing libraries for Linux if(NOT WIN32) @@ -269,6 +288,16 @@ endif() add_executable(flm ${SOURCES} ${HEADERS}) +if(FLM_ENABLE_CORELIB_AIE4) + include("${CMAKE_SOURCE_DIR}/common/corelib/corelib_sources.cmake") + add_library(flm_corelib_aie4 STATIC ${FLM_CORELIB_AIE4_SOURCES}) + target_include_directories(flm_corelib_aie4 PUBLIC + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}" "${FLM_CORELIB_BOOST_INCLUDE_DIR}") + target_compile_definitions(flm_corelib_aie4 PUBLIC FLM_ENABLE_CORELIB_AIE4=1) + target_link_libraries(flm PRIVATE flm_corelib_aie4) +endif() + if(WIN32) if(VCPKG_TOOLCHAIN) # A vcpkg toolchain is active (e.g. the rocm-npu-staging dev.py build or diff --git a/src/CMakePresets.json b/src/CMakePresets.json index 39a07cdd..fbb1b1cb 100644 --- a/src/CMakePresets.json +++ b/src/CMakePresets.json @@ -53,6 +53,17 @@ "CMAKE_BUILD_TYPE": "Release" } }, + { + "name": "windows-aie4", + "displayName": "Windows AIE4", + "description": "Windows build with optional dynamically loaded ryzenai-corelib support", + "inherits": "windows-default", + "binaryDir": "${sourceDir}/build-aie4", + "cacheVariables": { + "FLM_ENABLE_CORELIB_AIE4": "ON", + "RYZENAI_CORELIB_INCLUDE_DIR": "$env{RYZENAI_CORELIB_INCLUDE_DIR}" + } + }, { "name": "windows-vs18", "displayName": "Windows VS18", @@ -85,6 +96,14 @@ "configurePreset": "windows-default", "configuration": "Release", "jobs": 4 + }, + { + "name": "windows-aie4", + "displayName": "Windows AIE4 Build", + "description": "Build the optional dynamically loaded AIE4 runtime in Release", + "configurePreset": "windows-aie4", + "configuration": "Release", + "jobs": 4 } ] } diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index 5df2f2d2..9bd61ed9 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -8,6 +8,14 @@ #include "AutoModel/automodel.hpp" +ModelRequestError::ModelRequestError( + int http_code, bool session_cleared, std::string message) + : std::runtime_error(std::move(message)), + http_code_(http_code), session_cleared_(session_cleared) {} + +int ModelRequestError::http_code() const noexcept { return http_code_; } +bool ModelRequestError::session_cleared() const noexcept { return session_cleared_; } + AutoModel::AutoModel(flm_rt::device* npu_device_inst, std::string current_model) { this->npu_device_inst = npu_device_inst; this->current_model = current_model; @@ -116,34 +124,49 @@ void AutoModel::_shared_load_model(std::string model_path, json model_info, int header_print("FLM", "Model already loaded: " << this->model_path); return; } + const int context_length = default_context_length != -1 + ? default_context_length + : model_info["default_context_length"].get(); + this->_shared_initialize_model_state( + std::move(model_path), std::move(model_info), context_length); + this->_shared_initialize_legacy_npu(enable_preemption); +} - this->model_path = model_path; +void AutoModel::_shared_initialize_model_state( + std::string model_path, json, int context_length) { + this->model_path = std::move(model_path); header_print("FLM", "Loading model: " << this->model_path); this->lm_config = std::make_unique(); this->lm_config->from_pretrained(this->model_path); - if (this->npu_device_inst == nullptr) { - header_print("ERROR", "NPU device instance is nullptr"); - exit(1); - } - this->npu = std::make_unique(npu_device::device_npu2, this->npu_device_inst, enable_preemption); - this->enable_preemption = enable_preemption; - // Set context length: use provided value if not -1, otherwise use model default - if (default_context_length != -1) { - this->MAX_L = default_context_length; - } else { - this->MAX_L = model_info["default_context_length"]; - } - + this->MAX_L = context_length; this->is_model_loaded = true; - this->token_history.clear(); this->token_history.reserve(this->MAX_L); this->tokenizer = std::make_unique(this->model_path); - this->last_token = -1; this->total_tokens = 0; } +void AutoModel::_shared_initialize_legacy_npu(bool enable_preemption) { + if (this->npu_device_inst == nullptr) { + header_print("ERROR", "NPU device instance is nullptr"); + exit(1); + } + this->npu = std::make_unique( + npu_device::device_npu2, this->npu_device_inst, enable_preemption); + this->enable_preemption = enable_preemption; +} + +std::string AutoModel::generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled) { + if (!insert(meta_info, input, is_cancelled)) return {}; + return generate(meta_info, length_limit, os, std::move(is_cancelled)); +} + bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled, void* payload, int first_len_run) { // print token history @@ -214,6 +237,14 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector max_prefill_len = 1 << static_cast(std::ceil(std::log2(max_prefill_len))); buffer y; if (max_prefill_len < 512) { + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + buffer_.clear(); + current_mode_ = StreamEventType::CONTENT; + tool_name_.clear(); + is_in_tool_block_ = false; + return y; + } y = this->lm_engine->prefill(tokens, payload); } else{ @@ -225,19 +256,18 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector } int chunks = (tokens.size() + max_prefill_len - 1) / max_prefill_len; for (int i = 0; i < chunks; i++) { + int start = i * max_prefill_len; + int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); + std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); + header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); if (is_cancelled()) { meta_info.stop_reason = CANCEL_DETECTED; - // reset stream content buffer_.clear(); current_mode_ = StreamEventType::CONTENT; tool_name_.clear(); is_in_tool_block_ = false; break; } - int start = i * max_prefill_len; - int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); - std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); - header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); buffer chunk_y = this->lm_engine->prefill(chunk_tokens, (i == 0)? payload : nullptr); if (i == chunks - 1) { y = chunk_y; diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 06c164f5..9ce7780a 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -1,125 +1,398 @@ -/// \file phi4.cpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class - +/// \file modeling_phi4.cpp +/// \brief Phi-4 frontend and backend routing #include "AutoModel/modeling_phi4.hpp" +#include "utils/file_access.hpp" -/************ Phi4 family **************/ -Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" +#endif + +#include +#include +#include +#include +#include +#include + +namespace { +constexpr std::string_view kAie4Backend = "corelib_aie4_gguf"; +constexpr std::string_view kAie4Gguf = "Phi-4-mini-instruct.Q8_0.gguf"; +constexpr int kAie4DecodeLimit = 4095; + +enum class Phi4Backend { LegacyNpu2, CorelibAie4Gguf }; + +Phi4Backend ResolveBackend(const json& model_info) { + const auto details = model_info.find("details"); + if (details == model_info.end() || !details->is_object() || + !details->contains("execution_backend")) { + return Phi4Backend::LegacyNpu2; + } + const auto& backend = details->at("execution_backend"); + if (!backend.is_string()) { + throw std::invalid_argument( + "Phi-4 details.execution_backend must be a string"); + } + const std::string value = backend.get(); + if (value == kAie4Backend) return Phi4Backend::CorelibAie4Gguf; + throw std::invalid_argument("Unknown Phi-4 execution backend '" + value + "'"); +} + +std::uint32_t ResolveContext(const json& model_info, int requested) { + const std::int64_t value = requested == -1 + ? model_info.at("default_context_length").get() + : requested; + if (value < 1 || value > 4096) { + throw std::out_of_range("Phi-4 AIE4 context length must be in 1..4096"); + } + return static_cast(value); +} -void Phi4::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == phi4 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - - this->lm_engine->clear_context(); - this->setup_tokenizer(model_path); - this->sampler.reset(); +nlohmann::json ReadJson(const std::filesystem::path& path) { + flm::file_access::ObserveOpen(path); + std::ifstream input(path, std::ios::binary); + if (!input) throw std::runtime_error("Cannot open " + path.string()); + try { + return nlohmann::json::parse(input); + } catch (const std::exception& error) { + throw std::runtime_error("Cannot parse " + path.string() + ": " + error.what()); + } +} +void ConfigureSampler(Phi4& model) { sampler_config config; config.top_k = 40; config.top_p = 0.9; config.min_p = 0.1; config.temperature = 0.8; + model.set_sampler(config); +} +} // namespace + +#if defined(FLM_CORELIB_TESTING) +Phi4::EngineFactoryForTesting Phi4::engine_factory_for_testing_; +std::function Phi4::engine_poisoned_for_testing_; +#endif - this->set_sampler(config); - for (size_t i = 0; i < PROFILER_TYPE_NUM; i++) { - this->profiler_list[i].reset(); +Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} + +void Phi4::load_model(std::string model_path, json model_info, + int default_context_length, bool enable_preemption) { + const Phi4Backend backend = ResolveBackend(model_info); + if (backend == Phi4Backend::LegacyNpu2) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + const bool switching_from_aie4 = uses_corelib_aie4_; + uses_corelib_aie4_ = false; + aie4_poisoned_ = false; + corelib_runtime_.reset(); + if (switching_from_aie4) is_model_loaded = false; +#endif + _shared_load_model(model_path, model_info, default_context_length, enable_preemption); + std::unique_ptr engine; +#if defined(FLM_CORELIB_TESTING) + if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); + engine = engine_factory_for_testing_(false, *lm_config, npu.get(), model_path, MAX_L); +#else + q4nx = std::make_unique(this->model_path); + engine = std::make_unique(*lm_config, npu.get(), MAX_L); + engine->load_weights(*q4nx); + q4nx.reset(); +#endif + engine->clear_context(); + setup_tokenizer(model_path); + lm_engine = std::move(engine); + sampler.reset(); + ConfigureSampler(*this); + } else { +#if !defined(FLM_ENABLE_CORELIB_AIE4) + throw std::runtime_error( + "This binary was built without Phi-4 AIE4 corelib support"); +#else + if (enable_preemption) { + throw std::invalid_argument("Phi-4 AIE4 does not support preemption"); + } + const std::uint32_t context_length = ResolveContext(model_info, default_context_length); + const std::filesystem::path root(model_path); + + // Read and validate every source of truth before runtime acquisition or + // engine/device creation. There is deliberately no alternate filename. + const auto config = ReadJson(root / "config.json"); + const auto tokenizer_json = ReadJson(root / "tokenizer.json"); + const auto tokenizer_config = ReadJson(root / "tokenizer_config.json"); + auto package = flm::phi4::Phi4GgufPackage::Open(root / kAie4Gguf); + package->ValidatePhi4Contract(config, tokenizer_json, tokenizer_config); + + uses_corelib_aie4_ = false; + aie4_poisoned_ = false; + try { + _shared_initialize_model_state(model_path, model_info, + static_cast(context_length)); + npu.reset(); + this->enable_preemption = false; + setup_tokenizer(model_path, &tokenizer_config); + sampler.reset(); + ConfigureSampler(*this); + + std::unique_ptr engine; +#if defined(FLM_CORELIB_TESTING) + if (!engine_factory_for_testing_) throw std::logic_error("test engine factory is not installed"); + engine = engine_factory_for_testing_(true, *lm_config, nullptr, + root, context_length); +#else + auto runtime = flm::corelib::CorelibRuntime::GetOrCreate( + utils::get_executable_directory()); + engine = std::make_unique( + *lm_config, package, runtime, context_length); + corelib_runtime_ = std::move(runtime); +#endif + engine->clear_context(); + lm_engine = std::move(engine); + uses_corelib_aie4_ = true; + } catch (...) { + lm_engine.reset(); + corelib_runtime_.reset(); + tokenizer.reset(); + sampler.reset(); + lm_config.reset(); + is_model_loaded = false; + uses_corelib_aie4_ = false; + throw; + } +#endif } + + for (auto& item : profiler_list) item.reset(); } -void Phi4::setup_tokenizer(std::string model_path) { - // load tokenizer configurations - #ifdef _WIN32 - std::string tokenizer_config_path = model_path + "\\tokenizer_config.json"; - #else - std::string tokenizer_config_path = model_path + "/tokenizer_config.json"; - #endif - std::ifstream fs_config(tokenizer_config_path, std::ios::in | std::ios::binary); - if (fs_config.fail()) { - std::cerr << "Cannot open " << tokenizer_config_path << std::endl; - exit(1); - } - std::string data_config; - fs_config.seekg(0, std::ios::end); - size_t size_config = static_cast(fs_config.tellg()); - fs_config.seekg(0, std::ios::beg); - data_config.resize(size_config); - fs_config.read(data_config.data(), size_config); - fs_config.close(); - auto tokenizer_config = nlohmann::json::parse(data_config); - this->has_bos_token = false; - // load chat template - this->chat_tmpl = std::make_unique( - tokenizer_config["chat_template"], - "", - "" - ); - - if (this->has_bos_token) { - this->bos_token_id = tokenizer_config["bos_token_id"].get(); - } - else { - this->bos_token_id = -1; - } - this->eos_token = ""; - for (auto& token : tokenizer_config["eos_token_id"]) { - this->eos_token_ids.push_back(token.get()); +void Phi4::setup_tokenizer(const std::string& model_path, + const nlohmann::json* verified_tokenizer_config) { + nlohmann::json config = verified_tokenizer_config + ? *verified_tokenizer_config + : ReadJson(std::filesystem::path(model_path) / "tokenizer_config.json"); + if (!config.contains("chat_template") || !config["chat_template"].is_string()) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string chat_template"); + + const bool aie4 = verified_tokenizer_config != nullptr; + // Preserve the legacy Phi-4 contract: minja receives no textual BOS/EOS. + // AIE4 also disables automatic BOS, with stop IDs supplied only after the + // cross-source package contract has been validated. + auto chat = std::make_unique( + config["chat_template"].get(), "", ""); + std::vector eos; + if (aie4) { + // ValidatePhi4Contract proved these exact independent sources. + eos = {200020, 199999}; + } else { + if (!config.contains("eos_token_id")) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires eos_token_id"); + const auto& ids = config["eos_token_id"]; + if (ids.is_number_integer()) eos.push_back(ids.get()); + else if (ids.is_array()) for (const auto& id : ids) eos.push_back(id.get()); + else throw std::invalid_argument("Phi-4 tokenizer_config.json eos_token_id must be integer or array"); } - this->user_system_prompt = ""; - this->extra_context["user_system_prompt"] = this->user_system_prompt; + has_bos_token = false; + bos_token_id = -1; + eos_token.clear(); + eos_token_ids = std::move(eos); + chat_tmpl = std::move(chat); + user_system_prompt.clear(); + extra_context["user_system_prompt"] = user_system_prompt; } -std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools) { +std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json) { minja::chat_template_inputs inputs; inputs.add_generation_prompt = true; inputs.messages = messages; - inputs.extra_context = this->extra_context; - return this->chat_tmpl->apply(inputs); + inputs.extra_context = extra_context; + return chat_tmpl->apply(inputs); } -bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled) { - // preprocess - this->profiler_list[TKOEN_ENCODE_TIME].start(); - std::string templated_text; - if (input.messages.empty() && input.prompt.empty()) { - header_print("WARNING", "No messages or prompt provided"); - return false; +#if defined(FLM_ENABLE_CORELIB_AIE4) +bool Phi4::engine_is_poisoned() const noexcept { +#if defined(FLM_CORELIB_TESTING) + return engine_poisoned_for_testing_ && lm_engine + ? engine_poisoned_for_testing_(lm_engine.get()) + : false; +#else + const auto* engine = dynamic_cast(lm_engine.get()); + return engine && engine->poisoned(); +#endif +} + +void Phi4::validate_aie4_capacity(std::size_t rendered_tokens, + std::optional requested) const { + const std::size_t cap = std::min(MAX_L, kAie4DecodeLimit); + const auto normalized = normalize_requested_max_new_tokens(requested); + if (rendered_tokens >= cap || + (normalized && static_cast(*normalized) > cap - rendered_tokens)) { + std::ostringstream message; + message << "Phi-4 AIE4 request exceeds the 4095-token decode limit: rendered prompt has " + << rendered_tokens << " tokens"; + if (normalized) message << " and requested output has " << *normalized << " tokens"; + throw ModelRequestError(400, false, message.str()); } - if (!input.messages.empty()) { // already a formated messages, usually from REST API - templated_text = this->apply_chat_template(input.messages); +} + +void Phi4::clear_after_inference_failure(bool poisoned) { + aie4_poisoned_ = poisoned; + total_tokens = 0; + last_token = -1; + token_history.clear(); + checkpoint_his.clear(); + if (!poisoned && lm_engine) { + try { lm_engine->clear_context(); } catch (...) {} } - else if (!input.prompt.empty()) { // a pure text, usually from the cli - nlohmann::ordered_json messages; + if (sampler) sampler->reset_penalties(); +} - messages.push_back({ {"role", "user"}, {"content", input.prompt} }); - templated_text = this->apply_chat_template(messages); +std::string Phi4::generate_aie4(chat_meta_info_t& meta_info, + std::ostream& os, + std::function is_cancelled) { + std::string result; + meta_info.stop_reason = EOT_DETECTED; + int generated = 0; + profiler_list[DECODING_TIME].reset(); + while (last_token != -1 && generated < aie4_generation_budget_) { + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + break; + } + const int token = last_token; + token_history.push_back(token); + ++total_tokens; + ++generated; + ++meta_info.generated_tokens; + if (is_normal_token(token)) { + const std::string text = tokenizer->run_time_decoder(token); + result += text; + os << text << std::flush; + } + if (is_eos(token)) { + last_token = -1; + break; + } + if (generated >= aie4_generation_budget_ || total_tokens >= std::min(MAX_L, kAie4DecodeLimit)) { + last_token = -1; + meta_info.stop_reason = MAX_LENGTH_REACHED; + break; + } + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + break; + } + profiler_list[DECODING_TIME].start(); + auto logits = lm_engine->forward(token); + profiler_list[DECODING_TIME].stop(1); + last_token = sampler->sample(logits); } + meta_info.decoding_duration = (uint64_t)(time_utils::cast_to_us( + profiler_list[DECODING_TIME].get_total_time()).first) * 1e3; + return result; +} +#endif - std::vector tokens = this->tokenizer->encode(templated_text); - this->profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); - // hardware +std::string Phi4::show_profile() { + std::string profile = AutoModel::show_profile(); +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + profile += " Backend: corelib_aie4_gguf\n"; + if (corelib_runtime_) + profile += " Corelib DLL: " + + corelib_runtime_->loaded_library_path().string() + "\n"; + } +#endif + return profile; +} - return this->_shared_insert(meta_info, tokens, is_cancelled); +void Phi4::clear_context() { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_ && aie4_poisoned_) { + total_tokens = 0; + last_token = -1; + token_history.clear(); + checkpoint_his.clear(); + if (sampler) sampler->reset_penalties(); + return; + } +#endif + AutoModel::clear_context(); } +bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_ && aie4_poisoned_) { + throw ModelRequestError(500, true, + "Phi-4 AIE4 model is poisoned; unload/reload is required"); + } +#endif + profiler_list[TKOEN_ENCODE_TIME].start(); + std::string rendered; + if (input.messages.empty() && input.prompt.empty()) return false; + if (!input.messages.empty()) rendered = apply_chat_template(input.messages); + else { + nlohmann::ordered_json messages = nlohmann::ordered_json::array(); + messages.push_back({{"role", "user"}, {"content", input.prompt}}); + rendered = apply_chat_template(messages); + } + std::vector tokens = tokenizer->encode(rendered); + profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); -std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled) { - return this->_shared_generate(meta_info, length_limit, os, is_cancelled); +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + validate_aie4_capacity(tokens.size(), input.requested_max_new_tokens); + const auto normalized = normalize_requested_max_new_tokens(input.requested_max_new_tokens); + const int remaining = static_cast( + std::min(MAX_L, kAie4DecodeLimit) - tokens.size()); + aie4_generation_budget_ = normalized ? *normalized : remaining; + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + return false; + } + try { + return _shared_insert(meta_info, tokens, std::move(is_cancelled)); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + const bool poisoned = engine_is_poisoned(); + clear_after_inference_failure(poisoned); + throw ModelRequestError(500, true, poisoned + ? "AIE4 inference failed; unload/reload is required because the model is poisoned" + : "AIE4 inference failed; the current conversation was cleared"); + } + } +#endif + return _shared_insert(meta_info, tokens, std::move(is_cancelled)); } -std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os) { - if (!this->insert(meta_info, input)) { - return ""; +std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled) { +#if defined(FLM_ENABLE_CORELIB_AIE4) + if (uses_corelib_aie4_) { + if (aie4_poisoned_) throw ModelRequestError(500, true, + "Phi-4 AIE4 model is poisoned; unload/reload is required"); + try { + return generate_aie4(meta_info, os, std::move(is_cancelled)); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + const bool poisoned = engine_is_poisoned(); + clear_after_inference_failure(poisoned); + throw ModelRequestError(500, true, poisoned + ? "AIE4 inference failed; unload/reload is required because the model is poisoned" + : "AIE4 inference failed; the current conversation was cleared"); + } } - return this->_shared_generate(meta_info, length_limit, os); -} \ No newline at end of file +#endif + return _shared_generate(meta_info, length_limit, os, std::move(is_cancelled)); +} + +std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os) { + if (!insert(meta_info, input)) return {}; + return generate(meta_info, length_limit, os); +} diff --git a/src/common/corelib/corelib_api.cpp b/src/common/corelib/corelib_api.cpp new file mode 100644 index 00000000..cb98762d --- /dev/null +++ b/src/common/corelib/corelib_api.cpp @@ -0,0 +1,172 @@ +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include +#include + +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#endif + +namespace flm::corelib { +namespace { +constexpr CorelibVersion kRequiredVersion{0, 3, 0}; + +std::string VersionText(CorelibVersion version) { + return std::to_string(version.major) + "." + std::to_string(version.minor) + + "." + std::to_string(version.patch); +} + +std::string ErrorText(std::string_view call, + std::string_view status, + std::string_view detail) { + std::string result(call); + result += " failed: "; + result += status; + if (!detail.empty()) { + result += ": "; + result += detail; + } + return result; +} + +bool HasDllExtension(const std::filesystem::path& path) { + std::string extension = path.extension().string(); + std::transform(extension.begin(), extension.end(), extension.begin(), + [](unsigned char value) { + return static_cast(std::tolower(value)); + }); + return extension == ".dll"; +} +} // namespace + +CorelibError::CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text) + : std::runtime_error(ErrorText(call, status_text, detail)), + status_(status), + call_(std::move(call)), + detail_(std::move(detail)) {} + +ryzenai_corelib_status CorelibError::status() const noexcept { return status_; } +const std::string& CorelibError::call() const noexcept { return call_; } +const std::string& CorelibError::detail() const noexcept { return detail_; } + +CorelibApi::CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path) + : resolver_(std::move(resolver)), + loaded_library_path_(std::move(loaded_library_path)) { + void* version_symbol = resolver_("ryzenai_corelib_get_version"); + if (!version_symbol) { + throw std::runtime_error("missing corelib symbol: ryzenai_corelib_get_version"); + } + functions_.get_version = + reinterpret_cast(version_symbol); + functions_.get_version(&runtime_version_.major, &runtime_version_.minor, + &runtime_version_.patch); + if (runtime_version_.major != kRequiredVersion.major || + runtime_version_.minor != kRequiredVersion.minor || + runtime_version_.patch != kRequiredVersion.patch) { + throw std::runtime_error("corelib ABI mismatch: runtime " + + VersionText(runtime_version_) + ", required " + + VersionText(kRequiredVersion)); + } + +#define FLM_RESOLVE_CORELIB_FUNCTION(member, symbol) \ + if constexpr (std::string_view(#symbol) != \ + std::string_view("ryzenai_corelib_get_version")) { \ + void* address = resolver_(#symbol); \ + if (!address) throw std::runtime_error("missing corelib symbol: " #symbol); \ + functions_.member = reinterpret_cast(address); \ + } + FLM_CORELIB_FUNCTIONS(FLM_RESOLVE_CORELIB_FUNCTION) +#undef FLM_RESOLVE_CORELIB_FUNCTION +} + +std::shared_ptr CorelibApi::ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path) { + if (!resolver) throw std::invalid_argument("corelib resolver is empty"); + return std::shared_ptr(new CorelibApi( + std::move(resolver), std::move(loaded_library_path))); +} + +std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { +#ifndef _WIN32 + (void)dll; + throw std::runtime_error("ryzenai-corelib loading currently requires Windows"); +#else + const std::filesystem::path absolute_dll = std::filesystem::absolute(dll); + HMODULE raw_module = LoadLibraryExW( + absolute_dll.c_str(), nullptr, + LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); + if (!raw_module) { + throw std::runtime_error("failed to load corelib DLL '" + + absolute_dll.string() + "' (Windows error " + + std::to_string(GetLastError()) + ")"); + } + auto module = std::shared_ptr(raw_module, [](void* handle) { + FreeLibrary(static_cast(handle)); + }); + Resolver resolver = [module](std::string_view name) -> void* { + const std::string terminated(name); + return reinterpret_cast( + GetProcAddress(static_cast(module.get()), terminated.c_str())); + }; + return ResolveForTest(std::move(resolver), absolute_dll); +#endif +} + +std::filesystem::path CorelibApi::ResolveLibraryPath( + const std::filesystem::path& executable_dir) { + const char* configured = std::getenv("FLM_AIE4_CORELIB_PATH"); + if (configured && *configured) { + const std::filesystem::path path(configured); + if (!path.is_absolute()) { + throw std::runtime_error( + "FLM_AIE4_CORELIB_PATH must be an absolute .dll path"); + } + if (!path.has_filename() || !HasDllExtension(path)) { + throw std::runtime_error( + "FLM_AIE4_CORELIB_PATH must name an absolute .dll file"); + } + return path; + } + return std::filesystem::absolute(executable_dir / "aie4" / + "ryzenai_corelib.dll"); +} + +const CorelibFunctions& CorelibApi::functions() const noexcept { return functions_; } +CorelibVersion CorelibApi::runtime_version() const noexcept { return runtime_version_; } +const std::filesystem::path& CorelibApi::loaded_library_path() const noexcept { + return loaded_library_path_; +} + +void CorelibApi::Check(ryzenai_corelib_status status, + std::string_view call) const { + if (status == ryzenai_corelib_status_success) return; + const char* detail_pointer = functions_.get_last_error_message(); + const std::string detail = detail_pointer ? detail_pointer : ""; + const char* status_pointer = functions_.status_to_string(status); + const std::string status_text = status_pointer ? status_pointer : "unknown"; + throw CorelibError(status, std::string(call), detail, status_text); +} + +void CorelibApi::RegisterObject() const noexcept { ++live_object_count_; } + +void CorelibApi::Release(void* object) const noexcept { + if (!object) return; + functions_.object_release(object); + --live_object_count_; +} + +std::size_t CorelibApi::live_object_count() const noexcept { + return live_object_count_.load(); +} + +} // namespace flm::corelib diff --git a/src/common/corelib/corelib_runtime.cpp b/src/common/corelib/corelib_runtime.cpp new file mode 100644 index 00000000..937051c9 --- /dev/null +++ b/src/common/corelib/corelib_runtime.cpp @@ -0,0 +1,102 @@ +#include "corelib/corelib_runtime.hpp" + +#include +#include + +namespace flm::corelib { +namespace { +std::mutex process_mutex; +std::shared_ptr process_runtime; +#if defined(FLM_CORELIB_TESTING) +std::function destruction_observer; +bool shutdown_execution_lock_held = false; +#endif +} + +CorelibRuntime::CorelibRuntime(std::shared_ptr api) + : api_(std::move(api)) {} + +CorelibRuntime::~CorelibRuntime() { +#if defined(FLM_CORELIB_TESTING) + if (destruction_observer) destruction_observer(shutdown_execution_lock_held); +#endif +} + +std::shared_ptr CorelibRuntime::CreateReady( + std::shared_ptr api) { + if (!api) throw std::invalid_argument("corelib API is null"); + api->Check(api->functions().selftest_dependencies(), + "ryzenai_corelib_selftest_dependencies"); + if (!api->functions().has_device_context()) { + throw std::runtime_error("corelib has no AIE4 device context"); + } + return std::shared_ptr(new CorelibRuntime(std::move(api))); +} + +std::shared_ptr CorelibRuntime::GetOrCreate( + const std::filesystem::path& executable_dir) { + std::lock_guard lock(process_mutex); + if (!process_runtime) { + auto api = CorelibApi::Load(CorelibApi::ResolveLibraryPath(executable_dir)); + process_runtime = CreateReady(std::move(api)); + } + return process_runtime; +} + +std::shared_ptr CorelibRuntime::CreateForTest( + std::shared_ptr api) { + auto runtime = CreateReady(std::move(api)); + std::lock_guard lock(process_mutex); + if (process_runtime) { + throw std::runtime_error("corelib runtime already exists"); + } + process_runtime = runtime; + return runtime; +} + +void CorelibRuntime::ShutdownProcess() { + std::lock_guard process_lock(process_mutex); + if (!process_runtime) return; + + auto runtime = process_runtime; + std::unique_lock execution_lock(runtime->execution_mutex_); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = true; +#endif + if (runtime->api_->live_object_count() != 0) { +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif + throw std::runtime_error("cannot shut down with live corelib objects"); + } + runtime->api_->functions().cleanup(); + runtime->api_.reset(); + process_runtime.reset(); + execution_lock.unlock(); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif + runtime.reset(); +} + +#if defined(FLM_CORELIB_TESTING) +void CorelibRuntime::SetDestructionObserverForTest( + std::function observer) { + destruction_observer = std::move(observer); +} +#endif + +std::unique_lock CorelibRuntime::AcquireExecution() { + return std::unique_lock(execution_mutex_); +} + +const std::shared_ptr& CorelibRuntime::api() const noexcept { + return api_; +} + +const std::filesystem::path& CorelibRuntime::loaded_library_path() const noexcept { + static const std::filesystem::path empty; + return api_ ? api_->loaded_library_path() : empty; +} + +} // namespace flm::corelib diff --git a/src/common/corelib/corelib_sources.cmake b/src/common/corelib/corelib_sources.cmake new file mode 100644 index 00000000..18c97811 --- /dev/null +++ b/src/common/corelib/corelib_sources.cmake @@ -0,0 +1,6 @@ +include("${CMAKE_CURRENT_LIST_DIR}/../models/models_sources.cmake") + +set(FLM_CORELIB_AIE4_SOURCES + "${CMAKE_CURRENT_LIST_DIR}/corelib_api.cpp" + "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp" + ${FLM_MODELS_CORELIB_SOURCES}) diff --git a/src/common/models/models_sources.cmake b/src/common/models/models_sources.cmake new file mode 100644 index 00000000..ae37997a --- /dev/null +++ b/src/common/models/models_sources.cmake @@ -0,0 +1,8 @@ +# Per-model sources, split by implementation. Each model lives in +# common/models// with one folder per implementation: +# /flm/ - the original flm implementation, built into the flm binary +# /corelib/ - the parallel corelib implementation, built into +# flm_corelib_aie4 when FLM_ENABLE_CORELIB_AIE4 is on +# Adding a model means adding the folders, not editing this file. +file(GLOB FLM_MODELS_FLM_SOURCES "${CMAKE_CURRENT_LIST_DIR}/*/flm/*.cpp") +file(GLOB FLM_MODELS_CORELIB_SOURCES "${CMAKE_CURRENT_LIST_DIR}/*/corelib/*.cpp") diff --git a/src/common/models/phi4/corelib/phi4_corelib_aie4.cpp b/src/common/models/phi4/corelib/phi4_corelib_aie4.cpp new file mode 100644 index 00000000..8605b89d --- /dev/null +++ b/src/common/models/phi4/corelib/phi4_corelib_aie4.cpp @@ -0,0 +1,243 @@ +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "corelib/corelib_object.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" +#include "models/phi4/phi4_corelib_host.hpp" +#include "models/phi4/phi4_corelib_shape_plan.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +using namespace flm::corelib; +std::string Name(std::size_t i, const char* suffix) { + return "blk." + std::to_string(i) + suffix; +} + +/// Load-time phase accounting. Model load on this backend is dominated by +/// requantizing every weight from Q8_0, and without a breakdown there is no way +/// to tell that from disk I/O or from shape planning. Set FLM_AIE4_PROFILE_LOAD +/// to print it; the timer itself always runs, it costs five clock reads. +struct LoadPhases { + std::chrono::steady_clock::time_point mark{std::chrono::steady_clock::now()}; + double shape_plan{}, tensor_resolve{}, host_prep{}, weight_create{}, device_tensors{}; + + double Lap() { + const auto now = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(now - mark).count(); + mark = now; + return seconds; + } + + void Report() const { + const char* enabled = std::getenv("FLM_AIE4_PROFILE_LOAD"); + if (!enabled || !*enabled || *enabled == '0') return; + const double total = shape_plan + tensor_resolve + host_prep + + weight_create + device_tensors; + std::ostringstream out; + out << std::fixed << std::setprecision(2) + << "[FLM] AIE4 load: " << total << " s total" + << " (shape plan " << shape_plan + << ", GGUF resolve " << tensor_resolve + << ", host prep " << host_prep + << ", weight requantize " << weight_create + << ", device tensors " << device_tensors << ")"; + std::cout << out.str() << std::endl; + } +}; +} + +struct phi4_corelib_aie4::Impl { + std::shared_ptr package; + std::shared_ptr runtime; + std::shared_ptr api; + // Declared before `plan` so it starts before the initializer list builds it. + LoadPhases phases; + Phi4ShapePlan plan; + std::uint32_t max_length; + int position{}; + std::optional saved; + bool poisoned{}; + UniqueStream stream; + std::array q_weights, k_weights, v_weights, o_weights; + std::array mlp_weights; + UniqueMatMulWeights lm_weights; + UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits, cosine, sine; + std::array k_cache, v_cache; + TensorView embedding; + FloatTensorView first_norm_scale; + + Impl(LM_Config, std::shared_ptr pkg, + std::shared_ptr rt, std::uint32_t maximum) + : package(std::move(pkg)), runtime(std::move(rt)), + api(runtime ? runtime->api() : nullptr), plan(Phi4ShapePlan::Build(api)), + max_length(maximum) { + if (!package) throw std::invalid_argument("Phi-4 GGUF package is null"); + if (!runtime || !api) throw std::invalid_argument("corelib runtime is null"); + if (!maximum || maximum > kMaxSequenceLength) + throw std::invalid_argument("Phi-4 maximum length must be in 1..4096"); + phases.shape_plan = phases.Lap(); + + // Validate and capture every mapped span before the first device create. + embedding = package->RequireQ8("token_embd.weight", std::array{kVocabularySize,kHiddenSize}); + auto final_norm = package->RequireF32("output_norm.weight", std::array{kHiddenSize}); + std::array an, fn; + std::array qkv, gu; + std::array ow, dw; + for (std::size_t i=0;iRequireF32(Name(i,".attn_norm.weight"),std::array{kHiddenSize}); + fn[i]=package->RequireF32(Name(i,".ffn_norm.weight"),std::array{kHiddenSize}); + qkv[i]=package->AttentionQkv(i); gu[i]=package->GateUp(i); + ow[i]=package->RequireQ8(Name(i,".attn_output.weight"),std::array{kHiddenSize,kHiddenSize}); + dw[i]=package->RequireQ8(Name(i,".ffn_down.weight"),std::array{kHiddenSize,kIntermediateSize}); + } + phases.tensor_resolve = phases.Lap(); + std::optional factors; + try { factors=package->RequireF32("rope_factors_short.weight",std::array{48}); } + catch (const std::runtime_error&) {} + auto rope=BuildShortRopeTables(package->Metadata(),factors); + auto final_bf=ConvertF32ToBf16(final_norm.values); + std::array,kLayerCount> an_bf,fn_bf; + for(std::size_t i=0;i epsf{kRmsEpsilon}; auto eps=ConvertF32ToBf16(epsf); + + first_norm_scale = an[0]; + phases.host_prep = phases.Lap(); + auto lease=runtime->AcquireExecution(); void* raw=nullptr; + api->Check(api->functions().create_stream(&raw),"ryzenai_corelib_create_stream"); stream=UniqueStream(api,raw); + auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ + ryzenai_corelib_matmul_bf16_weights_desc d{kk,nn,kRequantizedGroupSize,false}; + ryzenai_corelib_matmul_bf16_gguf_components c{tv.bytes.data(),ryzenai_corelib_gguf_quant_type_q8_0}; void* p=nullptr; + api->Check(api->functions().matmul_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&p),"ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized "+label); + return UniqueMatMulWeights(api,p); + }; + for(std::size_t i=0;iCheck(api->functions().ssmlp_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&raw),"ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized layer "+std::to_string(i)); + mlp_weights[i]=UniqueSsMlpWeights(api,raw); + } + lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); + phases.weight_create = phases.Lap(); + const auto& e=plan.maximum_extents(); + const auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows, + e.ssmlp_rows}); + const auto query_rows=std::max(e.query_rows,e.flat_mha_rows); + const auto key_rows=std::max(e.kv_rows,e.flat_mha_rows); + const auto attention_rows=std::max(e.flat_mha_rows,e.output_rows); + auto tensor=[&](ryzenai_corelib_data_type type,std::initializer_list dims,const char* label){ + std::vector shape(dims);void* p=nullptr; + api->Check(api->functions().create_device_tensor(type,shape.data(),shape.size(),&p),std::string("ryzenai_corelib_create_device_tensor ")+label); + return UniqueTensor(api,p); + }; + hidden=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"hidden"); + residual=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"residual"); + skip=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"skip"); + q=tensor(ryzenai_corelib_data_type_bf16,{query_rows,kQueryDimension},"query"); + k=tensor(ryzenai_corelib_data_type_bf16,{key_rows,kKvDimension},"key"); + attention=tensor(ryzenai_corelib_data_type_bf16,{attention_rows,kQueryDimension},"attention"); + lm_input=tensor(ryzenai_corelib_data_type_bf16,{1,kHiddenSize},"lm input"); + logits=tensor(ryzenai_corelib_data_type_bf16,{1,kVocabularySize},"logits"); + cosine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"cosine"); + sine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"sine"); + for(std::size_t i=0;iCheck(api->functions().tensor_write(cosine.get(),ryzenai_corelib_data_type_fp32,rope.cosine.data(),rope.cosine.size(),0),"ryzenai_corelib_tensor_write cosine"); + api->Check(api->functions().tensor_write(sine.get(),ryzenai_corelib_data_type_fp32,rope.sine.data(),rope.sine.size(),0),"ryzenai_corelib_tensor_write sine"); + phases.device_tensors = phases.Lap(); + phases.Report(); + } + + void usable() const {if(poisoned)throw std::runtime_error("Phi-4 corelib engine is poisoned");} + buffer run(std::span ids,bool prefill){ + usable(); if(ids.empty())throw std::invalid_argument("Phi-4 request contains no token IDs"); + if(prefill&&position)throw std::runtime_error("Phi-4 prefill must start at logical position zero"); + if(ids.size()>max_length||position+ids.size()>max_length||position+ids.size()>kMaxSequenceLength)throw std::out_of_range("Phi-4 request exceeds configured context capacity"); + if(!prefill&&position+ids.size()>kMaxDecodeWindow)throw std::out_of_range("Phi-4 decode window stops at position 4095"); + auto decoded=DecodeEmbeddingRowsQ8(embedding,ids);const auto&e=plan.ForRows(ids.size()); + auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows,e.ssmlp_rows}); + std::vector normalized(decoded.size()); + HostRmsNorm(decoded,first_norm_scale.values,ids.size(),kHiddenSize, + kRmsEpsilon,normalized); + std::vector input(static_cast(rows*kHiddenSize),0); + std::vector residual_input(static_cast(rows*kHiddenSize),0); + std::copy(normalized.begin(),normalized.end(),input.begin()); + std::copy(decoded.begin(),decoded.end(),residual_input.begin()); + + auto lease=runtime->AcquireExecution();bool submitted=false; + try{ + api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); + api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_fp32,residual_input.data(),residual_input.size(),0),"ryzenai_corelib_tensor_write residual embedding"); + void* res=residual.get();void* sk=skip.get(); + for(std::size_t i=0;ifunctions().matmul( + stream.get(),hidden.get(),ids.size(),q_weights[i].get(),q.get()); + submitted=submitted || query_status==ryzenai_corelib_status_success || + query_status==ryzenai_corelib_status_failure; + api->Check(query_status,"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),hidden.get(),ids.size(),k_weights[i].get(),k.get()),"ryzenai_corelib_matmul_bf16 key layer "+std::to_string(i)); + std::array shape{8,kMaxSequenceLength-position,128};void* p=nullptr; + api->Check(api->functions().create_tensor_window(v_cache[i].get(),shape.data(),shape.size(),static_cast(position)*128,&p),"ryzenai_corelib_create_tensor_window V");UniqueTensorWindow win(api,p); + api->Check(api->functions().matmul(stream.get(),hidden.get(),ids.size(),v_weights[i].get(),win.get()),"ryzenai_corelib_matmul_bf16 value layer "+std::to_string(i)); + api->Check(api->functions().flat_mha(stream.get(),&plan.attention_desc(),q.get(),k.get(),ids.size(),position,cosine.get(),sine.get(),k_cache[i].get(),v_cache[i].get(),attention.get()),"ryzenai_corelib_flat_mha_bf16 layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),attention.get(),ids.size(),o_weights[i].get(),hidden.get()),"ryzenai_corelib_matmul_bf16 output layer "+std::to_string(i)); + api->Check(api->functions().ssmlp(stream.get(),hidden.get(),res,ids.size(),mlp_weights[i].get(),sk,hidden.get()),"ryzenai_corelib_ssmlp_bf16 layer "+std::to_string(i));std::swap(res,sk); + } + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize hidden"); + std::vector row(kHiddenSize);api->Check(api->functions().tensor_read(hidden.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),(ids.size()-1)*kHiddenSize),"ryzenai_corelib_tensor_read final hidden row"); + api->Check(api->functions().tensor_write(lm_input.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),0),"ryzenai_corelib_tensor_write LM head input"); + api->Check(api->functions().matmul(stream.get(),lm_input.get(),1,lm_weights.get(),logits.get()),"ryzenai_corelib_matmul_bf16 LM head"); + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize logits"); + buffer out(kVocabularySize);api->Check(api->functions().tensor_read(logits.get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),0),"ryzenai_corelib_tensor_read logits");position+=static_cast(ids.size());return out; + }catch(...){if(submitted){(void)api->functions().stream_synchronize(stream.get());poisoned=true;position=0;saved.reset();}throw;} + } + buffer read_cache(bool is_k,int layer,int index){ + usable(); + if(layer<0||layer>=kLayerCount||index<0||index>=kMaxSequenceLength) + throw std::out_of_range("Phi-4 cache index is out of range"); + auto lease=runtime->AcquireExecution(); + api->Check(api->functions().stream_synchronize(stream.get()), + "ryzenai_corelib_stream_synchronize cache read"); + buffer out(kKvHeadCount*kHeadSize); + void* cache=is_k?k_cache[layer].get():v_cache[layer].get(); + for(std::size_t head=0;head(index))*kHeadSize; + api->Check(api->functions().tensor_read( + cache,ryzenai_corelib_data_type_bf16, + out.data()+head*kHeadSize,kHeadSize,offset), + "ryzenai_corelib_tensor_read cache head "+ + std::to_string(head)); + } + return out; + } +}; + +phi4_corelib_aie4::phi4_corelib_aie4(LM_Config c,std::shared_ptr p,std::shared_ptr r,std::uint32_t m):impl_(std::make_unique(std::move(c),std::move(p),std::move(r),m)){} +phi4_corelib_aie4::~phi4_corelib_aie4()=default; +buffer phi4_corelib_aie4::forward(int id){return impl_->run(std::span(&id,1),false);} +buffer phi4_corelib_aie4::prefill(std::vector&ids,void*){return impl_->run(ids,true);} +void phi4_corelib_aie4::set_context_length(int n){impl_->usable();if(n<0||static_cast(n)>impl_->max_length)throw std::out_of_range("Phi-4 context length is out of range");impl_->position=n;} +void phi4_corelib_aie4::load_weights(Q4NX&){impl_->usable();throw std::runtime_error("Phi-4 AIE4 weights are loaded only from GGUF");} +void phi4_corelib_aie4::update_max_length(std::uint32_t n){impl_->usable();if(!n||n>kMaxSequenceLength||n(impl_->position))throw std::out_of_range("Phi-4 maximum length is invalid");impl_->max_length=n;} +void phi4_corelib_aie4::clear_context(){impl_->usable();impl_->position=0;impl_->saved.reset();} +buffer phi4_corelib_aie4::get_k_cache(int l,int i){return impl_->read_cache(true,l,i);} +buffer phi4_corelib_aie4::get_v_cache(int l,int i){return impl_->read_cache(false,l,i);} +int phi4_corelib_aie4::get_current_context_length(){impl_->usable();return impl_->position;} +int phi4_corelib_aie4::checkpoint(){impl_->usable();impl_->saved=impl_->position;return impl_->position;} +int phi4_corelib_aie4::restore(){impl_->usable();if(!impl_->saved)return -1;return impl_->position=*impl_->saved;} +bool phi4_corelib_aie4::poisoned()const noexcept{return impl_&&impl_->poisoned;} +} // namespace flm::phi4 diff --git a/src/common/models/phi4/corelib/phi4_corelib_gguf.cpp b/src/common/models/phi4/corelib/phi4_corelib_gguf.cpp new file mode 100644 index 00000000..1b907d35 --- /dev/null +++ b/src/common/models/phi4/corelib/phi4_corelib_gguf.cpp @@ -0,0 +1,654 @@ +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" +#include "utils/file_access.hpp" + +#define NOMINMAX +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +constexpr std::uint32_t kMagic = 0x46554747; +constexpr std::uint32_t kVersion = 3; +constexpr std::uint32_t kTypeF32 = 0; +constexpr std::uint32_t kTypeQ8_0 = 8; + +[[noreturn]] void Fail(std::string_view field, std::string actual, + std::string expected) { + throw std::runtime_error(std::string(field) + ": actual " + actual + + ", expected " + expected); +} + +std::uint64_t CheckedAdd(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a > std::numeric_limits::max() - b) + throw std::runtime_error(std::string(field) + ": overflow in addition"); + return a + b; +} + +std::uint64_t CheckedMultiply(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a != 0 && b > std::numeric_limits::max() / a) + throw std::runtime_error(std::string(field) + ": overflow in product"); + return a * b; +} + +std::uint64_t AlignUp(std::uint64_t value, std::uint64_t alignment) { + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + return CheckedAdd(value, alignment - 1, "alignment") & ~(alignment - 1); +} + +std::span RequireRange(std::span file, + std::uint64_t offset, + std::uint64_t length, + std::string_view field) { + const auto end = CheckedAdd(offset, length, field); + if (end > file.size() || offset > std::numeric_limits::max() || + length > std::numeric_limits::max()) + Fail(field, "out-of-file range", "range within mapped file"); + return file.subspan(static_cast(offset), + static_cast(length)); +} + +class Cursor { +public: + Cursor(std::span file, std::uint64_t offset = 0) + : file_(file), offset_(offset) {} + + template + T Read(std::string_view field) { + const auto bytes = RequireRange(file_, offset_, sizeof(T), field); + T value; + std::memcpy(&value, bytes.data(), sizeof(T)); + offset_ = CheckedAdd(offset_, sizeof(T), field); + return value; + } + + std::string ReadString(std::string_view field) { + const auto length = Read(field); + const auto bytes = RequireRange(file_, offset_, length, field); + std::string value(reinterpret_cast(bytes.data()), bytes.size()); + offset_ = CheckedAdd(offset_, length, field); + return value; + } + + void Skip(std::uint64_t length, std::string_view field) { + RequireRange(file_, offset_, length, field); + offset_ = CheckedAdd(offset_, length, field); + } + + std::uint64_t offset() const noexcept { return offset_; } + +private: + std::span file_; + std::uint64_t offset_; +}; + +std::string MetadataTypeName(std::uint32_t type) { + static constexpr const char* names[] = {"UINT8", "INT8", "UINT16", "INT16", + "UINT32", "INT32", "FLOAT32", "BOOL", "STRING", "ARRAY", "UINT64", + "INT64", "FLOAT64"}; + return type < std::size(names) ? names[type] : "unknown(" + std::to_string(type) + ")"; +} + +std::uint64_t FixedMetadataSize(std::uint32_t type) { + switch (type) { + case 0: case 1: case 7: return 1; + case 2: case 3: return 2; + case 4: case 5: case 6: return 4; + case 10: case 11: case 12: return 8; + default: return 0; + } +} + +struct ArrayInfo { std::uint32_t type; std::uint64_t count; }; +using MetadataValue = std::variant; + +MetadataValue ReadMetadataValue(Cursor& cursor, std::uint32_t type, + std::string_view field, bool retain) { + switch (type) { + case 0: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 1: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 2: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 3: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 4: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 5: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 6: { auto v = cursor.Read(field); return retain ? MetadataValue(double(v)) : MetadataValue{}; } + case 7: { auto v = cursor.Read(field); if (v > 1) Fail(field, std::to_string(v), "GGUF boolean 0 or 1"); return retain ? MetadataValue(bool(v)) : MetadataValue{}; } + case 8: { auto v = cursor.ReadString(field); return retain ? MetadataValue(std::move(v)) : MetadataValue{}; } + case 9: { + const std::string array_field = std::string(field) + " array"; + const auto element_type = cursor.Read(array_field); + const auto count = cursor.Read(array_field); + if (element_type == 9 || element_type > 12) + Fail(array_field, MetadataTypeName(element_type), "a skippable GGUF array element type"); + const auto fixed = FixedMetadataSize(element_type); + if (fixed != 0) { + cursor.Skip(CheckedMultiply(count, fixed, array_field), array_field); + } else { + const auto minimum = CheckedMultiply(count, std::uint64_t{8}, array_field); + (void)minimum; + for (std::uint64_t i = 0; i < count; ++i) + (void)ReadMetadataValue(cursor, element_type, array_field, false); + } + return retain ? MetadataValue(ArrayInfo{element_type, count}) : MetadataValue{}; + } + case 10: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 11: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 12: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + default: + Fail(field, MetadataTypeName(type), "a supported metadata type"); + } +} + +bool IsRetainedKey(std::string_view key) { + static constexpr std::string_view keys[] = { + "general.architecture", "general.alignment", "phi3.block_count", + "phi3.context_length", "phi3.embedding_length", "phi3.feed_forward_length", + "phi3.attention.head_count", "phi3.attention.head_count_kv", + "phi3.attention.layer_norm_rms_epsilon", "phi3.rope.dimension_count", + "phi3.rope.freq_base", "phi3.rope.scaling.attn_factor", + "phi3.rope.scaling.original_context_length", "tokenizer.ggml.tokens", + "tokenizer.ggml.add_bos_token", "tokenizer.ggml.eos_token_id"}; + return std::find(std::begin(keys), std::end(keys), key) != std::end(keys); +} + +std::string ShapeText(std::span shape) { + std::ostringstream out; + out << '['; + for (std::size_t i = 0; i < shape.size(); ++i) { + if (i) out << ','; + out << shape[i]; + } + return out.str() + ']'; +} + +std::string GgmlTypeName(std::uint32_t type) { + if (type == kTypeF32) return "F32"; + if (type == kTypeQ8_0) return "Q8_0"; + return "GGML type " + std::to_string(type); +} + +std::uint64_t ElementCount(std::span shape, + std::string_view field) { + std::uint64_t result = 1; + for (const auto dimension : shape) { + if (dimension <= 0) Fail(field, std::to_string(dimension), "positive dimensions"); + result = CheckedMultiply(result, static_cast(dimension), field); + } + return result; +} + +std::uint64_t TensorByteLength(std::uint32_t type, + std::span shape, + std::string_view field) { + const auto elements = ElementCount(shape, field); + if (type == kTypeF32) return CheckedMultiply(elements, 4, field); + if (type == kTypeQ8_0) { + if (elements % 32 != 0) + Fail(field, std::to_string(elements) + " elements", "Q8_0 element count divisible by 32"); + return CheckedMultiply(elements / 32, 34, field); + } + Fail(field, GgmlTypeName(type), "F32 or Q8_0"); +} + +std::string JsonText(const nlohmann::json& value) { + return value.dump(); +} + +void RequireJsonString(const nlohmann::json& object, std::string_view key, + std::string_view expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_string()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::string(expected)); + const auto actual = it->get_ref(); + if (actual != expected) Fail(key, actual, std::string(expected)); +} + +void RequireJsonBoolean(const nlohmann::json& object, std::string_view key, + bool expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_boolean()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), expected ? "true" : "false"); + const auto actual = it->get(); + if (actual != expected) Fail(key, actual ? "true" : "false", expected ? "true" : "false"); +} + +void RequireJsonUnsigned(const nlohmann::json& object, std::string_view key, + std::uint64_t expected) { + const auto it = object.find(std::string(key)); + if (it == object.end()) Fail(key, "missing", std::to_string(expected)); + std::uint64_t actual; + if (it->is_number_unsigned()) { + actual = it->get(); + } else if (it->is_number_integer()) { + const auto signed_value = it->get(); + if (signed_value < 0) + Fail(key, JsonText(*it), "non-negative integer " + std::to_string(expected)); + actual = static_cast(signed_value); + } else { + Fail(key, JsonText(*it), "integer " + std::to_string(expected)); + } + if (actual != expected) Fail(key, std::to_string(actual), std::to_string(expected)); +} + +void RequireJsonDouble(const nlohmann::json& object, std::string_view key, + double expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_number()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::to_string(expected)); + const auto actual = it->get(); + if (!std::isfinite(actual) || actual != expected) + Fail(key, JsonText(*it), std::to_string(expected)); +} +} // namespace + +struct Phi4GgufPackage::Impl { + struct TensorRecord { + std::string name; + std::span bytes; + std::vector shape; + std::uint32_t type; + std::uint64_t absolute_offset; + }; + + HANDLE file = INVALID_HANDLE_VALUE; + HANDLE mapping = nullptr; + const std::byte* data = nullptr; + std::uint64_t size = 0; + std::map> tensors; + std::map> metadata; + std::map> metadata_types; + + ~Impl() { + if (data) UnmapViewOfFile(data); + if (mapping) CloseHandle(mapping); + if (file != INVALID_HANDLE_VALUE) CloseHandle(file); + } + + std::span bytes() const { + return {data, static_cast(size)}; + } + + const TensorRecord& Tensor(std::string_view name) const { + const auto it = tensors.find(name); + if (it == tensors.end()) Fail(name, "missing", "present tensor"); + return it->second; + } + + std::uint64_t Unsigned(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "unsigned integer metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "unsigned integer metadata"); + } + + double Number(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "floating-point metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "floating-point metadata"); + } + + bool Boolean(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "boolean metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "boolean metadata"); + } + + std::string String(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "string metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "string metadata"); + } + + std::uint64_t ArrayCount(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "array metadata"); + if (const auto* value = std::get_if(&it->second)) return value->count; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "array metadata"); + } +}; + +Phi4GgufPackage::Phi4GgufPackage(std::unique_ptr impl) + : impl_(std::move(impl)) {} +Phi4GgufPackage::~Phi4GgufPackage() = default; + +std::shared_ptr Phi4GgufPackage::Open( + const std::filesystem::path& gguf_path) { + auto impl = std::make_unique(); + flm::file_access::ObserveOpen(gguf_path); + impl->file = CreateFileW(gguf_path.c_str(), GENERIC_READ, FILE_SHARE_READ, + nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); + if (impl->file == INVALID_HANDLE_VALUE) + throw std::runtime_error("GGUF file: actual open failure " + + std::to_string(GetLastError()) + ", expected readable file"); + LARGE_INTEGER size; + if (!GetFileSizeEx(impl->file, &size) || size.QuadPart <= 0 || + static_cast(size.QuadPart) > std::numeric_limits::max()) + Fail("GGUF file size", std::to_string(size.QuadPart), "positive mappable size"); + impl->size = static_cast(size.QuadPart); + impl->mapping = CreateFileMappingW(impl->file, nullptr, PAGE_READONLY, 0, 0, nullptr); + if (!impl->mapping) + throw std::runtime_error("GGUF mapping: actual CreateFileMappingW failure " + + std::to_string(GetLastError()) + ", expected PAGE_READONLY mapping"); + impl->data = static_cast( + MapViewOfFile(impl->mapping, FILE_MAP_READ, 0, 0, 0)); + if (!impl->data) + throw std::runtime_error("GGUF mapping: actual MapViewOfFile failure " + + std::to_string(GetLastError()) + ", expected FILE_MAP_READ view"); + + const auto file = impl->bytes(); + Cursor cursor(file); + if (cursor.Read("GGUF header") != kMagic) + Fail("GGUF magic", "mismatch", "0x46554747"); + const auto version = cursor.Read("GGUF header"); + if (version != kVersion) Fail("GGUF version", std::to_string(version), "3"); + const auto tensor_count = cursor.Read("tensor count"); + const auto metadata_count = cursor.Read("metadata count"); + if (tensor_count > file.size() / 24) Fail("tensor count", std::to_string(tensor_count), "count fitting directory"); + if (metadata_count > file.size() / 12) Fail("metadata count", std::to_string(metadata_count), "count fitting metadata"); + + for (std::uint64_t i = 0; i < metadata_count; ++i) { + const auto key = cursor.ReadString("metadata key string"); + const auto type = cursor.Read(key); + const bool retain = IsRetainedKey(key); + auto value = ReadMetadataValue(cursor, type, key, retain); + if (retain) { + if (!impl->metadata.emplace(key, std::move(value)).second) + Fail(key, "duplicate metadata key", "unique metadata key"); + impl->metadata_types.emplace(key, type); + } + } + + constexpr std::uint64_t kDefaultAlignment = 32; + const auto alignment = impl->metadata.contains("general.alignment") + ? impl->Unsigned("general.alignment") + : kDefaultAlignment; + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + + struct DirectoryTensor { + std::string name; + std::vector shape; + std::uint32_t type; + std::uint64_t relative_offset; + std::uint64_t length; + }; + std::vector directory; + directory.reserve(static_cast(tensor_count)); + for (std::uint64_t i = 0; i < tensor_count; ++i) { + auto name = cursor.ReadString("tensor directory name"); + const auto dimension_count = cursor.Read("tensor directory dimensions"); + if (dimension_count == 0 || dimension_count > 4) + Fail(name, std::to_string(dimension_count), "1..4 tensor dimensions"); + std::vector shape; + shape.reserve(dimension_count); + for (std::uint32_t d = 0; d < dimension_count; ++d) { + const auto dimension = cursor.Read("tensor directory dimension"); + if (dimension > static_cast(std::numeric_limits::max())) + Fail(name, std::to_string(dimension), "dimension fitting int64"); + shape.push_back(static_cast(dimension)); + } + std::reverse(shape.begin(), shape.end()); + const auto type = cursor.Read("tensor directory type"); + const auto offset = cursor.Read("tensor directory offset"); + const auto length = TensorByteLength(type, shape, name); + directory.push_back({std::move(name), std::move(shape), type, offset, length}); + } + + const auto data_start = AlignUp(cursor.offset(), alignment); + struct Range { std::uint64_t begin, end; std::string name; }; + std::vector ranges; + ranges.reserve(directory.size()); + for (auto& tensor : directory) { + if (tensor.relative_offset % alignment != 0) + Fail(tensor.name, std::to_string(tensor.relative_offset), "offset aligned to " + std::to_string(alignment)); + const auto absolute = CheckedAdd(data_start, tensor.relative_offset, tensor.name); + const auto bytes = RequireRange(file, absolute, tensor.length, tensor.name + " range"); + const auto end = CheckedAdd(absolute, tensor.length, tensor.name); + ranges.push_back({absolute, end, tensor.name}); + auto [it, inserted] = impl->tensors.emplace(tensor.name, + Impl::TensorRecord{tensor.name, bytes, std::move(tensor.shape), tensor.type, absolute}); + if (!inserted) Fail(tensor.name, "duplicate tensor name", "unique tensor name"); + } + std::sort(ranges.begin(), ranges.end(), [](const Range& a, const Range& b) { + return a.begin < b.begin; + }); + for (std::size_t i = 1; i < ranges.size(); ++i) { + if (ranges[i].begin < ranges[i - 1].end) + Fail(ranges[i].name, "overlap with " + ranges[i - 1].name, "non-overlapping tensor range"); + } + return std::shared_ptr(new Phi4GgufPackage(std::move(impl))); +} + +TensorView Phi4GgufPackage::RequireQ8( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeQ8_0) + Fail(name, GgmlTypeName(tensor.type), "Q8_0"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeQ8_0, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + return {tensor.name, tensor.bytes, tensor.shape, tensor.type}; +} + +FloatTensorView Phi4GgufPackage::RequireF32( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeF32) + Fail(name, GgmlTypeName(tensor.type), "F32"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeF32, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + const auto address = reinterpret_cast(tensor.bytes.data()); + if (tensor.absolute_offset % alignof(float) != 0 || address % alignof(float) != 0) + Fail(name, "address/offset not aligned", "alignment 4"); + return {tensor.name, + {reinterpret_cast(tensor.bytes.data()), + tensor.bytes.size() / sizeof(float)}, + tensor.shape}; +} + +ProjectionViews Phi4GgufPackage::AttentionQkv(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("attention layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".attn_qkv.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{5120, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 3; + result.values[0] = {fused.name, fused.bytes.subspan(0, 3072 * row_bytes), {3072, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(3072 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + result.values[2] = {fused.name, fused.bytes.subspan(4096 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + return result; +} + +ProjectionViews Phi4GgufPackage::GateUp(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("MLP layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".ffn_up.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{16384, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 2; + result.values[0] = {fused.name, fused.bytes.subspan(0, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(8192 * row_bytes, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + return result; +} + +GgufPhi4Metadata Phi4GgufPackage::Metadata() const { + return {impl_->String("general.architecture"), + impl_->Unsigned("phi3.block_count"), + impl_->Unsigned("phi3.embedding_length"), + impl_->Unsigned("phi3.feed_forward_length"), + impl_->Unsigned("phi3.attention.head_count"), + impl_->Unsigned("phi3.attention.head_count_kv"), + impl_->Unsigned("phi3.context_length"), + impl_->Unsigned("phi3.rope.dimension_count"), + impl_->Number("phi3.rope.freq_base"), + impl_->Number("phi3.rope.scaling.attn_factor"), + impl_->Unsigned("phi3.rope.scaling.original_context_length"), + impl_->ArrayCount("tokenizer.ggml.tokens"), + impl_->Boolean("tokenizer.ggml.add_bos_token")}; +} + +void Phi4GgufPackage::ValidatePhi4Contract( + const nlohmann::json& config, const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const { + const auto metadata = Metadata(); + const auto require_unsigned = [](std::string_view field, std::uint64_t actual, + std::uint64_t expected) { + if (actual != expected) Fail(field, std::to_string(actual), std::to_string(expected)); + }; + if (metadata.architecture != "phi3") Fail("general.architecture", metadata.architecture, "phi3"); + require_unsigned("phi3.block_count", metadata.layer_count, kLayerCount); + require_unsigned("phi3.context_length", metadata.context_length, kModelContextLength); + require_unsigned("phi3.embedding_length", metadata.hidden_size, kHiddenSize); + require_unsigned("phi3.feed_forward_length", metadata.intermediate_size, kIntermediateSize); + require_unsigned("phi3.attention.head_count", metadata.attention_head_count, kQueryHeadCount); + require_unsigned("phi3.attention.head_count_kv", metadata.kv_head_count, kKvHeadCount); + require_unsigned("phi3.rope.dimension_count", metadata.rope_dimension_count, kRopeDimension); + require_unsigned("phi3.rope.scaling.original_context_length", metadata.rope_original_context_length, kMaxSequenceLength); + require_unsigned("tokenizer.ggml.tokens", metadata.tokenizer_vocabulary_size, kVocabularySize); + if (metadata.add_bos_token) Fail("tokenizer.ggml.add_bos_token", "true", "false"); + const auto rms = impl_->Number("phi3.attention.layer_norm_rms_epsilon"); + if (!std::isfinite(rms) || rms != static_cast(kRmsEpsilon)) + Fail("phi3.attention.layer_norm_rms_epsilon", std::to_string(rms), std::to_string(kRmsEpsilon)); + for (const auto [field, value] : std::array{ + std::pair{"phi3.rope.freq_base", metadata.rope_frequency_base}, + std::pair{"phi3.rope.scaling.attn_factor", metadata.rope_attention_factor}}) { + if (!std::isfinite(value) || value <= 0) Fail(field, std::to_string(value), "finite positive value"); + } + + RequireQ8("token_embd.weight", std::array{kVocabularySize, kHiddenSize}); + RequireF32("output_norm.weight", std::array{kHiddenSize}); + if (impl_->tensors.contains("output.weight")) Fail("output.weight", "present", "absent (tied token_embd.weight)"); + if (impl_->tensors.contains("rope_factors_long.weight")) + RequireF32("rope_factors_long.weight", std::array{48}); + for (std::size_t layer = 0; layer < static_cast(kLayerCount); ++layer) { + const auto prefix = "blk." + std::to_string(layer); + RequireF32(prefix + ".attn_norm.weight", std::array{kHiddenSize}); + RequireF32(prefix + ".ffn_norm.weight", std::array{kHiddenSize}); + RequireQ8(prefix + ".attn_qkv.weight", std::array{kQueryDimension + 2 * kKvDimension, kHiddenSize}); + RequireQ8(prefix + ".attn_output.weight", std::array{kHiddenSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_up.weight", std::array{2 * kIntermediateSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_down.weight", std::array{kHiddenSize, kIntermediateSize}); + } + if (impl_->tensors.contains("rope_factors_short.weight")) + RequireF32("rope_factors_short.weight", std::array{48}); + + RequireJsonString(config, "model_type", "phi3"); + RequireJsonUnsigned(config, "num_hidden_layers", kLayerCount); + RequireJsonUnsigned(config, "hidden_size", kHiddenSize); + RequireJsonUnsigned(config, "intermediate_size", kIntermediateSize); + RequireJsonUnsigned(config, "num_attention_heads", kQueryHeadCount); + RequireJsonUnsigned(config, "num_key_value_heads", kKvHeadCount); + if (config.contains("head_dim")) + RequireJsonUnsigned(config, "head_dim", kHeadSize); + RequireJsonUnsigned(config, "vocab_size", kVocabularySize); + RequireJsonDouble(config, "rms_norm_eps", 1.0e-5); + RequireJsonUnsigned(config, "original_max_position_embeddings", kMaxSequenceLength); + RequireJsonUnsigned(config, "eos_token_id", 199999); + + std::set vocabulary_ids; + std::map> token_ids; + const auto add_token = [&](const std::string& token, const nlohmann::json& encoded_id) { + const std::string field = "tokenizer.json token ID " + token; + std::uint64_t unsigned_id; + if (encoded_id.is_number_unsigned()) { + unsigned_id = encoded_id.get(); + } else if (encoded_id.is_number_integer()) { + const auto signed_id = encoded_id.get(); + if (signed_id < 0) + Fail(field, std::to_string(signed_id), "0..200063"); + unsigned_id = static_cast(signed_id); + } else { + Fail(field, JsonText(encoded_id), "integer in 0..200063"); + } + if (unsigned_id >= static_cast(kVocabularySize)) + Fail(field, std::to_string(unsigned_id), "0..200063"); + const auto id = static_cast(unsigned_id); + const auto [it, inserted] = token_ids.emplace(token, id); + if (!inserted && it->second != id) + Fail(token, std::to_string(id), std::to_string(it->second)); + vocabulary_ids.insert(id); + }; + try { + const auto& vocab = tokenizer.at("model").at("vocab"); + if (!vocab.is_object()) Fail("tokenizer.json model.vocab", JsonText(vocab), "object mapping tokens to IDs"); + for (auto it = vocab.begin(); it != vocab.end(); ++it) + add_token(it.key(), it.value()); + const auto added = tokenizer.find("added_tokens"); + if (added != tokenizer.end()) { + if (!added->is_array()) Fail("tokenizer.json added_tokens", JsonText(*added), "array"); + for (const auto& item : *added) { + const auto content = item.at("content").get(); + add_token(content, item.at("id")); + } + } + } catch (const nlohmann::json::exception& error) { + Fail("tokenizer.json vocabulary", error.what(), "valid token-to-ID mappings"); + } + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + const auto it = token_ids.find(token); + if (it == token_ids.end()) Fail(token, "missing", std::to_string(expected)); + if (it->second != expected) Fail(token, std::to_string(it->second), std::to_string(expected)); + } + constexpr std::int64_t kTokenizerMaximumAssignedId = 200028; + constexpr std::size_t kTokenizerDistinctAssignedIds = 200029; + const auto actual_count = vocabulary_ids.size(); + const auto actual_max = vocabulary_ids.empty() ? -1 : *vocabulary_ids.rbegin(); + if (actual_max != kTokenizerMaximumAssignedId) + Fail("tokenizer.json maximum vocabulary ID", std::to_string(actual_max), + std::to_string(kTokenizerMaximumAssignedId)); + if (actual_count != kTokenizerDistinctAssignedIds) + Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), + std::to_string(kTokenizerDistinctAssignedIds)); + const auto gguf_eos = impl_->Unsigned("tokenizer.ggml.eos_token_id"); + if (gguf_eos != 200020) Fail("tokenizer.ggml.eos_token_id", std::to_string(gguf_eos), "200020"); + + RequireJsonBoolean(tokenizer_config, "add_bos_token", false); + const auto template_it = tokenizer_config.find("chat_template"); + if (template_it == tokenizer_config.end() || !template_it->is_string()) + Fail("chat_template", template_it == tokenizer_config.end() ? "missing" : JsonText(*template_it), "string containing Phi-4 markers"); + const auto chat_template = template_it->get(); + const bool has_dynamic_role = + chat_template.find("'<|' + message['role'] + '|>'") != std::string::npos; + if (chat_template.find("<|user|>") == std::string::npos && !has_dynamic_role) + Fail("<|user|>", "missing from chat_template", "present in chat_template"); + for (const auto marker : {"<|end|>", "<|assistant|>"}) + if (chat_template.find(marker) == std::string::npos) + Fail(marker, "missing from chat_template", "present in chat_template"); +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/corelib/phi4_corelib_host.cpp b/src/common/models/phi4/corelib/phi4_corelib_host.cpp new file mode 100644 index 00000000..e817a601 --- /dev/null +++ b/src/common/models/phi4/corelib/phi4_corelib_host.cpp @@ -0,0 +1,177 @@ +#include "models/phi4/phi4_corelib_host.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" + +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +float HalfToFloat(std::uint16_t half) { + const std::uint32_t sign = static_cast(half & 0x8000) << 16; + const std::uint32_t exponent = (half >> 10) & 0x1f; + std::uint32_t fraction = half & 0x03ff; + std::uint32_t bits; + if (exponent == 0) { + if (fraction == 0) { + bits = sign; + } else { + int shift = 0; + while ((fraction & 0x0400) == 0) { + fraction <<= 1; + ++shift; + } + fraction &= 0x03ff; + bits = sign | (static_cast(127 - 14 - shift) << 23) | + (fraction << 13); + } + } else if (exponent == 0x1f) { + bits = sign | 0x7f800000 | (fraction << 13); + } else { + bits = sign | ((exponent + (127 - 15)) << 23) | (fraction << 13); + } + return std::bit_cast(bits); +} + +} // namespace + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, std::span token_ids) { + if (embedding.ggml_type != 8 || embedding.logical_shape.size() != 2 || + embedding.logical_shape[0] <= 0 || embedding.logical_shape[1] <= 0 || + embedding.logical_shape[1] % 32 != 0) { + throw std::runtime_error("embedding must be a two-dimensional Q8_0 tensor with block-aligned rows"); + } + const auto rows = static_cast(embedding.logical_shape[0]); + const auto width = static_cast(embedding.logical_shape[1]); + const auto blocks_per_row = width / 32; + const auto row_bytes = blocks_per_row * 34; + if (rows > std::numeric_limits::max() / row_bytes || + embedding.bytes.size() != rows * row_bytes) { + throw std::runtime_error("embedding Q8_0 byte length does not match its logical shape"); + } + + std::vector result; + result.reserve(token_ids.size() * width); + for (const int token_id : token_ids) { + if (token_id < 0 || static_cast(token_id) >= rows) { + throw std::out_of_range("embedding token id is outside the vocabulary"); + } + const std::byte* row = embedding.bytes.data() + + static_cast(token_id) * row_bytes; + for (std::size_t block = 0; block < blocks_per_row; ++block) { + const std::byte* encoded = row + block * 34; + std::uint16_t scale_bits; + std::memcpy(&scale_bits, encoded, sizeof(scale_bits)); + const float scale = HalfToFloat(scale_bits); + for (std::size_t element = 0; element < 32; ++element) { + const auto code = static_cast( + std::to_integer(encoded[2 + element])); + result.push_back(scale * static_cast(code)); + } + } + } + return result; +} + +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output) { + if (rows <= 0 || width <= 0) + throw std::invalid_argument("Phi-4 RMSNorm rows and width must be positive"); + const auto row_count = static_cast(rows); + const auto row_width = static_cast(width); + if (row_count > std::numeric_limits::max() / row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape overflow"); + const auto elements = row_count * row_width; + if (input.size() != elements || output.size() != elements || + scale.size() != row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape mismatch"); + if (!std::isfinite(epsilon) || epsilon < 0.0f) + throw std::invalid_argument("Phi-4 RMSNorm epsilon must be finite and nonnegative"); + + for (std::size_t row = 0; row < row_count; ++row) { + const auto base = row * row_width; + double sum_of_squares = 0.0; + for (std::size_t column = 0; column < row_width; ++column) { + const double value = input[base + column]; + sum_of_squares += value * value; + } + const float mean_square = static_cast( + sum_of_squares / static_cast(width)); + const float denominator = std::sqrt(mean_square + epsilon); + for (std::size_t column = 0; column < row_width; ++column) + output[base + column] = + (input[base + column] / denominator) * scale[column]; + } +} + +std::vector ConvertF32ToBf16(std::span values) { + std::vector result; + result.reserve(values.size()); + for (const float value : values) { + std::uint32_t bits = std::bit_cast(value); + if ((bits & 0x7fffffffU) > 0x7f800000U) { + bits |= 0x00400000U; + } else { + bits += 0x7fffU + ((bits >> 16) & 1U); + } + result.push_back(static_cast(bits >> 16)); + } + return result; +} + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors) { + if (metadata.context_length < static_cast(kMaxSequenceLength) || + metadata.rope_original_context_length != static_cast(kMaxSequenceLength) || + metadata.rope_dimension_count != static_cast(kRopeDimension) || + !std::isfinite(metadata.rope_frequency_base) || metadata.rope_frequency_base <= 0 || + !std::isfinite(metadata.rope_attention_factor)) { + throw std::runtime_error("invalid Phi-4 RoPE metadata"); + } + + std::array factors{}; + factors.fill(1.0); + if (short_factors) { + if (short_factors->logical_shape != std::vector{kRopeDimension / 2} || + short_factors->values.size() != factors.size()) { + throw std::runtime_error("rope_factors_short.weight must have shape [48]"); + } + for (std::size_t i = 0; i < factors.size(); ++i) { + factors[i] = short_factors->values[i]; + if (!std::isfinite(factors[i]) || factors[i] <= 0) + throw std::runtime_error("rope_factors_short.weight must contain finite positive values"); + } + } + + RopeTables tables; + tables.cosine.resize(kMaxSequenceLength * factors.size()); + tables.sine.resize(kMaxSequenceLength * factors.size()); + for (std::size_t i = 0; i < factors.size(); ++i) { + const double inv_freq = 1.0 / + (std::pow(metadata.rope_frequency_base, (2.0 * i) / 96.0) * factors[i]); + for (std::size_t position = 0; position < kMaxSequenceLength; ++position) { + const double angle = static_cast(position) * inv_freq; + const auto index = position * factors.size() + i; + tables.cosine[index] = static_cast( + std::cos(angle) * metadata.rope_attention_factor); + tables.sine[index] = static_cast( + std::sin(angle) * metadata.rope_attention_factor); + } + } + return tables; +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/corelib/phi4_corelib_shape_plan.cpp b/src/common/models/phi4/corelib/phi4_corelib_shape_plan.cpp new file mode 100644 index 00000000..4867b07b --- /dev/null +++ b/src/common/models/phi4/corelib/phi4_corelib_shape_plan.cpp @@ -0,0 +1,108 @@ +#include "models/phi4/phi4_corelib_shape_plan.hpp" + +#include "models/phi4/phi4_corelib_constants.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +std::int64_t MatmulRows(const std::shared_ptr& api, + std::int64_t rows, std::int64_t logical_k, + std::int64_t logical_n, const char* logical_name) { + auto m = rows; + auto k = logical_k; + auto n = logical_n; + const std::string call = std::string("ryzenai_corelib_matmul_bf16_pad_shape ") + + logical_name + " [" + std::to_string(rows) + "," + + std::to_string(logical_k) + "]x[" + std::to_string(logical_k) + "," + + std::to_string(logical_n) + "]"; + api->Check(api->functions().matmul_pad_shape( + &m, &k, &n, kRequantizedGroupSize), call); + if (k != logical_k || n != logical_n) { + throw std::runtime_error(call + ": helper changed padded K/N"); + } + return m; +} + +} // namespace + +Phi4ShapePlan Phi4ShapePlan::Build( + const std::shared_ptr& api) { + if (!api) throw std::invalid_argument("Phi4ShapePlan corelib API is null"); + + Phi4ShapePlan plan; + plan.attention_desc_ = {kQueryHeadCount, kKvHeadCount, kHeadSize, + kMaxSequenceLength, kRopeDimension}; + plan.lm_head_desc_ = {kHiddenSize, kVocabularySize, + kRequantizedGroupSize, false}; + plan.rows_.reserve(kMaxSequenceLength); + constexpr std::array execution_rows{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; + + for (const auto rows : execution_rows) { + Phi4RowExtents extents{}; + extents.query_rows = MatmulRows(api, rows, kHiddenSize, + kQueryDimension, "query"); + extents.kv_rows = MatmulRows(api, rows, kHiddenSize, + kKvDimension, "key/value"); + extents.output_rows = MatmulRows(api, rows, kHiddenSize, + kHiddenSize, "output"); + + extents.ssmlp_rows = rows; + const std::string ssmlp_call = + "ryzenai_corelib_ssmlp_bf16_pad_rows [" + std::to_string(rows) + + ",3072,8192]"; + api->Check(api->functions().ssmlp_pad_rows( + &extents.ssmlp_rows, kHiddenSize, kIntermediateSize, + kRequantizedGroupSize), ssmlp_call); + + extents.flat_mha_rows = rows; + const std::string mha_call = + "ryzenai_corelib_flat_mha_bf16_pad_rows [" + std::to_string(rows) + + ",24,8,128,4096,96]"; + api->Check(api->functions().flat_mha_pad_rows( + &extents.flat_mha_rows, &plan.attention_desc_), mha_call); + plan.maximum_extents_.query_rows = std::max( + plan.maximum_extents_.query_rows, extents.query_rows); + plan.maximum_extents_.kv_rows = std::max( + plan.maximum_extents_.kv_rows, extents.kv_rows); + plan.maximum_extents_.output_rows = std::max( + plan.maximum_extents_.output_rows, extents.output_rows); + plan.maximum_extents_.ssmlp_rows = std::max( + plan.maximum_extents_.ssmlp_rows, extents.ssmlp_rows); + plan.maximum_extents_.flat_mha_rows = std::max( + plan.maximum_extents_.flat_mha_rows, extents.flat_mha_rows); + while (plan.rows_.size() < static_cast(rows)) + plan.rows_.push_back(extents); + } + + (void)MatmulRows(api, 1, kHiddenSize, kVocabularySize, "lm_head"); + return plan; +} + +const Phi4RowExtents& Phi4ShapePlan::ForRows(std::size_t live_rows) const { + if (live_rows == 0 || live_rows > rows_.size()) { + throw std::out_of_range("Phi-4 live rows must be in 1..4096"); + } + return rows_[live_rows - 1]; +} + +const Phi4RowExtents& Phi4ShapePlan::maximum_extents() const noexcept { + return maximum_extents_; +} + +const ryzenai_corelib_flat_mha_bf16_desc& +Phi4ShapePlan::attention_desc() const noexcept { + return attention_desc_; +} + +const ryzenai_corelib_matmul_bf16_weights_desc& +Phi4ShapePlan::lm_head_desc() const noexcept { + return lm_head_desc_; +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/flm/.gitkeep b/src/common/models/phi4/flm/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/src/common/tokenizer/tokenizer.cpp b/src/common/tokenizer/tokenizer.cpp index 2b21981c..f048fc85 100644 --- a/src/common/tokenizer/tokenizer.cpp +++ b/src/common/tokenizer/tokenizer.cpp @@ -4,6 +4,7 @@ /// \date 2025-06-24 /// \version 0.9.10 #include "tokenizer/tokenizer.hpp" +#include "utils/file_access.hpp" #include #include #include @@ -15,6 +16,8 @@ /// \brief Constructor /// \param model_path the model path Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); #ifdef _WIN32 std::ifstream fs(model_path + "\\tokenizer.json", std::ios::in | std::ios::binary); #else diff --git a/src/include/AutoModel/automodel.hpp b/src/include/AutoModel/automodel.hpp index ed3a6726..e9088f9b 100644 --- a/src/include/AutoModel/automodel.hpp +++ b/src/include/AutoModel/automodel.hpp @@ -16,6 +16,8 @@ #include #include #include +#include +#include #include "typedef.hpp" #include "causal_lm.hpp" #include "lm_config.hpp" @@ -128,10 +130,26 @@ struct lm_uniform_input_t { std::vector audios; std::vector audio_payload_types; nlohmann::ordered_json tools; + std::optional requested_max_new_tokens; }; +inline std::optional normalize_requested_max_new_tokens( + std::optional requested) { + return requested.has_value() && *requested > 0 ? requested : std::nullopt; +} + using json = nlohmann::ordered_json; +class ModelRequestError final : public std::runtime_error { +public: + ModelRequestError(int http_code, bool session_cleared, std::string message); + int http_code() const noexcept; + bool session_cleared() const noexcept; +private: + int http_code_; + bool session_cleared_; +}; + class AutoModel { protected: std::string model_path = ""; @@ -188,6 +206,8 @@ class AutoModel { void _shared_load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false); + void _shared_initialize_model_state(std::string model_path, json model_info, int context_length); + void _shared_initialize_legacy_npu(bool enable_preemption); nlohmann::json _shared_setup_tokenizer(std::string model_path); /// \brief Insert tokens into the model @@ -220,6 +240,8 @@ class AutoModel { /// \return the current model std::string get_current_model(); + virtual bool uses_corelib_aie4() const noexcept { return false; } + /// \brief Get the current context length /// \return the current context length virtual int get_current_context_length(); @@ -361,6 +383,12 @@ class AutoModel { /// \brief Generate the tokens with prompt virtual std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) = 0; + std::string generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled); /// \brief Configure a parameter with type-erased value /// \param parameter_name the name of the parameter diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index 66937d6d..b416d56d 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -1,24 +1,69 @@ -/// \file phi4.hpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class +/// \file modeling_phi4.hpp +/// \brief Phi-4 frontend and backend routing #pragma once #include "AutoModel/automodel.hpp" -/************ phi4 family **************/ +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "corelib/corelib_runtime.hpp" +#endif + +#if defined(FLM_CORELIB_TESTING) +#include +#include +namespace flm::phi4::testing { class Phi4FrontendTestAccess; } +#endif + class Phi4 : public AutoModel { private: - void setup_tokenizer(std::string model_path); + void setup_tokenizer(const std::string& model_path, + const nlohmann::json* verified_tokenizer_config = nullptr); +#if defined(FLM_ENABLE_CORELIB_AIE4) + void validate_aie4_capacity(std::size_t rendered_tokens, + std::optional requested) const; + bool engine_is_poisoned() const noexcept; + void clear_after_inference_failure(bool poisoned); + std::string generate_aie4(chat_meta_info_t& meta_info, + std::ostream& os, + std::function is_cancelled); -public: - Phi4(flm_rt::device* npu_device_inst); + bool uses_corelib_aie4_ = false; + bool aie4_poisoned_ = false; + int aie4_generation_budget_ = 0; + std::shared_ptr corelib_runtime_; +#endif - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; - //void toggle_enable_think() override; - bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; - std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; - std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; - std::string apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; +#if defined(FLM_CORELIB_TESTING) + using EngineFactoryForTesting = std::function( + bool, const LM_Config&, npu_xclbin_manager*, + const std::filesystem::path&, std::uint32_t)>; + static EngineFactoryForTesting engine_factory_for_testing_; + static std::function engine_poisoned_for_testing_; + friend class flm::phi4::testing::Phi4FrontendTestAccess; +#endif + +public: + explicit Phi4(flm_rt::device* npu_device_inst); + void load_model(std::string model_path, json model_info, + int default_context_length = -1, + bool enable_preemption = false) override; + bool uses_corelib_aie4() const noexcept override { +#if defined(FLM_ENABLE_CORELIB_AIE4) + return uses_corelib_aie4_; +#else + return false; +#endif + } + std::string show_profile() override; + void clear_context() override; + bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled = [] { return false; }) override; + std::string generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled = [] { return false; }) override; + std::string generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os = std::cout) override; + std::string apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; }; diff --git a/src/include/corelib/corelib_api.hpp b/src/include/corelib/corelib_api.hpp new file mode 100644 index 00000000..d1e811fd --- /dev/null +++ b/src/include/corelib/corelib_api.hpp @@ -0,0 +1,104 @@ +#pragma once + +#include + +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 3 || \ + RYZENAI_CORELIB_VERSION_PATCH != 0 +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.3.0" +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define FLM_CORELIB_FUNCTIONS(X) \ + X(get_version, ryzenai_corelib_get_version) \ + X(status_to_string, ryzenai_corelib_status_to_string) \ + X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ + X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ + X(has_device_context, ryzenai_corelib_has_device_context) \ + X(object_release, ryzenai_corelib_object_release) \ + X(create_stream, ryzenai_corelib_create_stream) \ + X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ + X(create_device_tensor, ryzenai_corelib_create_device_tensor) \ + X(create_tensor_window, ryzenai_corelib_create_tensor_window) \ + X(tensor_write, ryzenai_corelib_tensor_write) \ + X(tensor_read, ryzenai_corelib_tensor_read) \ + X(tensor_get_byte_size, ryzenai_corelib_tensor_get_byte_size) \ + X(tensor_get_data_type, ryzenai_corelib_tensor_get_data_type) \ + X(matmul_pad_shape, ryzenai_corelib_matmul_bf16_pad_shape) \ + X(matmul_weights_create_gguf_requantized, \ + ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized) \ + X(matmul, ryzenai_corelib_matmul_bf16) \ + X(ssmlp_pad_rows, ryzenai_corelib_ssmlp_bf16_pad_rows) \ + X(ssmlp_weights_create_gguf_requantized, \ + ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) \ + X(ssmlp, ryzenai_corelib_ssmlp_bf16) \ + X(flat_mha_pad_rows, ryzenai_corelib_flat_mha_bf16_pad_rows) \ + X(flat_mha, ryzenai_corelib_flat_mha_bf16) \ + X(cleanup, ryzenai_corelib_cleanup) + +namespace flm::corelib { + +struct CorelibVersion { + std::uint32_t major; + std::uint32_t minor; + std::uint32_t patch; +}; + +class CorelibError final : public std::runtime_error { +public: + CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text); + ryzenai_corelib_status status() const noexcept; + const std::string& call() const noexcept; + const std::string& detail() const noexcept; + +private: + ryzenai_corelib_status status_; + std::string call_; + std::string detail_; +}; + +struct CorelibFunctions { +#define FLM_DECLARE_CORELIB_FUNCTION(member, symbol) decltype(&::symbol) member{}; + FLM_CORELIB_FUNCTIONS(FLM_DECLARE_CORELIB_FUNCTION) +#undef FLM_DECLARE_CORELIB_FUNCTION +}; + +class CorelibApi final { +public: + using Resolver = std::function; + static std::shared_ptr Load(const std::filesystem::path& dll); + static std::shared_ptr ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path = {}); + static std::filesystem::path ResolveLibraryPath( + const std::filesystem::path& executable_dir); + const CorelibFunctions& functions() const noexcept; + CorelibVersion runtime_version() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; + void Check(ryzenai_corelib_status status, std::string_view call) const; + void RegisterObject() const noexcept; + void Release(void* object) const noexcept; + std::size_t live_object_count() const noexcept; + +private: + explicit CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path = {}); + + Resolver resolver_; + CorelibFunctions functions_{}; + CorelibVersion runtime_version_{}; + std::filesystem::path loaded_library_path_; + mutable std::atomic live_object_count_{0}; +}; + +} // namespace flm::corelib diff --git a/src/include/corelib/corelib_object.hpp b/src/include/corelib/corelib_object.hpp new file mode 100644 index 00000000..faf44fd7 --- /dev/null +++ b/src/include/corelib/corelib_object.hpp @@ -0,0 +1,62 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include + +namespace flm::corelib { + +struct StreamTag {}; +struct TensorTag {}; +struct TensorWindowTag {}; +struct MatMulWeightsTag {}; +struct SsMlpWeightsTag {}; + +template +class UniqueObject final { +public: + UniqueObject() noexcept = default; + + UniqueObject(std::shared_ptr api, void* object) noexcept + : api_(std::move(api)), object_(object) { + if (object_) api_->RegisterObject(); + } + + ~UniqueObject() { reset(); } + + UniqueObject(const UniqueObject&) = delete; + UniqueObject& operator=(const UniqueObject&) = delete; + + UniqueObject(UniqueObject&& other) noexcept + : api_(std::move(other.api_)), object_(std::exchange(other.object_, nullptr)) {} + + UniqueObject& operator=(UniqueObject&& other) noexcept { + if (this != &other) { + reset(); + api_ = std::move(other.api_); + object_ = std::exchange(other.object_, nullptr); + } + return *this; + } + + void reset() noexcept { + if (object_) api_->Release(std::exchange(object_, nullptr)); + api_.reset(); + } + + void* get() const noexcept { return object_; } + explicit operator bool() const noexcept { return object_ != nullptr; } + +private: + std::shared_ptr api_; + void* object_{}; +}; + +using UniqueStream = UniqueObject; +using UniqueTensor = UniqueObject; +using UniqueTensorWindow = UniqueObject; +using UniqueMatMulWeights = UniqueObject; +using UniqueSsMlpWeights = UniqueObject; + +} // namespace flm::corelib diff --git a/src/include/corelib/corelib_runtime.hpp b/src/include/corelib/corelib_runtime.hpp new file mode 100644 index 00000000..d0ad61c2 --- /dev/null +++ b/src/include/corelib/corelib_runtime.hpp @@ -0,0 +1,38 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#if defined(FLM_CORELIB_TESTING) +#include +#endif + +namespace flm::corelib { + +class CorelibRuntime final { +public: + ~CorelibRuntime(); + static std::shared_ptr GetOrCreate( + const std::filesystem::path& executable_dir); + static std::shared_ptr CreateForTest( + std::shared_ptr api); + static void ShutdownProcess(); +#if defined(FLM_CORELIB_TESTING) + static void SetDestructionObserverForTest(std::function observer); +#endif + std::unique_lock AcquireExecution(); + const std::shared_ptr& api() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; + +private: + explicit CorelibRuntime(std::shared_ptr api); + static std::shared_ptr CreateReady( + std::shared_ptr api); + + std::shared_ptr api_; + std::mutex execution_mutex_; +}; + +} // namespace flm::corelib diff --git a/src/include/lm_config.hpp b/src/include/lm_config.hpp index 8eb4d0c0..201fd422 100644 --- a/src/include/lm_config.hpp +++ b/src/include/lm_config.hpp @@ -8,6 +8,7 @@ #include "typedef.hpp" #include "utils/utils.hpp" +#include "utils/file_access.hpp" #include "nlohmann/json.hpp" #include @@ -99,7 +100,9 @@ class LM_Config{ /// \brief read model_path/config.json into _json_config void _load_json(){ - std::ifstream file(this->model_path + "/config.json"); + const auto config_path = std::filesystem::path(this->model_path) / "config.json"; + flm::file_access::ObserveOpen(config_path); + std::ifstream file(config_path); if (!file.is_open()){ std::cerr << "Failed to open file: " << this->model_path << std::endl; exit(1); diff --git a/src/include/models/phi4/phi4_corelib_aie4.hpp b/src/include/models/phi4/phi4_corelib_aie4.hpp new file mode 100644 index 00000000..678f2e08 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_aie4.hpp @@ -0,0 +1,40 @@ +#pragma once + +#include "causal_lm.hpp" +#include "corelib/corelib_runtime.hpp" +#include "lm_config.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include +#include + +namespace flm::phi4 { + +class phi4_corelib_aie4 final : public causal_lm { +public: + phi4_corelib_aie4( + LM_Config config, + std::shared_ptr package, + std::shared_ptr runtime, + std::uint32_t max_length = 4096); + ~phi4_corelib_aie4() override; + + buffer forward(int id) override; + buffer prefill(std::vector& ids, void* payload = nullptr) override; + void set_context_length(int length) override; + void load_weights(Q4NX&) override; + void update_max_length(std::uint32_t max_length) override; + void clear_context() override; + buffer get_k_cache(int layer, int index) override; + buffer get_v_cache(int layer, int index) override; + int get_current_context_length() override; + int checkpoint() override; + int restore() override; + bool poisoned() const noexcept; + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_constants.hpp b/src/include/models/phi4/phi4_corelib_constants.hpp new file mode 100644 index 00000000..e34e87f6 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_constants.hpp @@ -0,0 +1,29 @@ +#pragma once + +#include + +namespace flm::phi4 { +inline constexpr std::int64_t kLayerCount = 32; +inline constexpr std::int64_t kHiddenSize = 3072; +inline constexpr std::int64_t kIntermediateSize = 8192; +inline constexpr std::int64_t kQueryHeadCount = 24; +inline constexpr std::int64_t kKvHeadCount = 8; +inline constexpr std::int64_t kHeadSize = 128; +inline constexpr std::int64_t kQueryDimension = 3072; +inline constexpr std::int64_t kKvDimension = 1024; +inline constexpr std::int64_t kVocabularySize = 200064; +inline constexpr std::int64_t kRopeDimension = 96; +inline constexpr std::int64_t kMaxSequenceLength = 4096; +inline constexpr std::int64_t kModelContextLength = 131072; +inline constexpr std::int64_t kMaxDecodeWindow = 4095; +inline constexpr std::uint32_t kRequantizedGroupSize = 64; +/// Intra-packer threads for the Q8_0 requantizing creates. corelib treats 0 as +/// ONE deliberately; this path is compute-bound and scales with the hint. +/// +/// This is the per-create hint, NOT concurrent creates. corelib documents that +/// loading a model with 8 CONCURRENT creates on this entry point failed 2 of 10 +/// with all-zero output, against 0 of 10 serialized, with attribution open. The +/// creates therefore stay serialized. +inline constexpr std::uint32_t kRequantizeThreads = 8; +inline constexpr float kRmsEpsilon = 1.0e-5f; +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_gguf.hpp b/src/include/models/phi4/phi4_corelib_gguf.hpp new file mode 100644 index 00000000..7af4e684 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_gguf.hpp @@ -0,0 +1,77 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace flm::phi4 { + +struct TensorView { + std::string_view name; + std::span bytes; + std::vector logical_shape; + std::uint32_t ggml_type; +}; + +struct FloatTensorView { + std::string_view name; + std::span values; + std::vector logical_shape; +}; + +struct ProjectionViews { + std::array values; + std::size_t count; +}; + +struct GgufPhi4Metadata { + std::string architecture; + std::uint64_t layer_count; + std::uint64_t hidden_size; + std::uint64_t intermediate_size; + std::uint64_t attention_head_count; + std::uint64_t kv_head_count; + std::uint64_t context_length; + std::uint64_t rope_dimension_count; + double rope_frequency_base; + double rope_attention_factor; + std::uint64_t rope_original_context_length; + std::uint64_t tokenizer_vocabulary_size; + bool add_bos_token; +}; + +class Phi4GgufPackage final { +public: + static std::shared_ptr Open( + const std::filesystem::path& gguf_path); + ~Phi4GgufPackage(); + + TensorView RequireQ8( + std::string_view name, + std::span expected_shape) const; + FloatTensorView RequireF32( + std::string_view name, + std::span expected_shape) const; + ProjectionViews AttentionQkv(std::size_t layer) const; + ProjectionViews GateUp(std::size_t layer) const; + GgufPhi4Metadata Metadata() const; + void ValidatePhi4Contract( + const nlohmann::json& config, + const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const; + +private: + struct Impl; + explicit Phi4GgufPackage(std::unique_ptr impl); + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_host.hpp b/src/include/models/phi4/phi4_corelib_host.hpp new file mode 100644 index 00000000..67fb1094 --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_host.hpp @@ -0,0 +1,35 @@ +#pragma once + +#include "models/phi4/phi4_corelib_gguf.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct RopeTables { + std::vector cosine; + std::vector sine; +}; + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, + std::span token_ids); + +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output); + +std::vector ConvertF32ToBf16(std::span values); + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors); + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/phi4_corelib_shape_plan.hpp b/src/include/models/phi4/phi4_corelib_shape_plan.hpp new file mode 100644 index 00000000..8c8010bb --- /dev/null +++ b/src/include/models/phi4/phi4_corelib_shape_plan.hpp @@ -0,0 +1,36 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct Phi4RowExtents { + std::int64_t query_rows; + std::int64_t kv_rows; + std::int64_t output_rows; + std::int64_t ssmlp_rows; + std::int64_t flat_mha_rows; +}; + +class Phi4ShapePlan final { +public: + static Phi4ShapePlan Build( + const std::shared_ptr& api); + const Phi4RowExtents& ForRows(std::size_t live_rows) const; + const Phi4RowExtents& maximum_extents() const noexcept; + const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; + const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; + +private: + std::vector rows_; + Phi4RowExtents maximum_extents_{}; + ryzenai_corelib_flat_mha_bf16_desc attention_desc_{}; + ryzenai_corelib_matmul_bf16_weights_desc lm_head_desc_{}; +}; + +} // namespace flm::phi4 diff --git a/src/include/utils/debug_utils.hpp b/src/include/utils/debug_utils.hpp index 9189b826..6106ae21 100644 --- a/src/include/utils/debug_utils.hpp +++ b/src/include/utils/debug_utils.hpp @@ -5,6 +5,7 @@ /// \version 0.9.24 /// \note This file contains the debug utilities for the FastFlowLM project. #pragma once +#include #include #include #include @@ -181,3 +182,16 @@ inline std::string size_t_to_string(size_t size){ return std::to_string(size / (1024 * 1024 * 1024)) + "G"; } } + +/// \brief Report how long loading a model took. +/// \param started the time point captured immediately before load_model +/// \note Model load is the one phase no profiler covers, and on backends that +/// repack weights at load it dominates the time to a first usable prompt. Call +/// this from every path that loads a model so the CLI and the server agree. +inline void report_load_time(std::chrono::steady_clock::time_point started) { + const double seconds = + std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + std::ostringstream message; + message << std::fixed << std::setprecision(2) << "Model loaded in " << seconds << " s"; + header_print("FLM", message.str()); +} diff --git a/src/include/utils/file_access.hpp b/src/include/utils/file_access.hpp new file mode 100644 index 00000000..21fc74c6 --- /dev/null +++ b/src/include/utils/file_access.hpp @@ -0,0 +1,27 @@ +#pragma once + +#include + +#if defined(FLM_CORELIB_TESTING) +#include +#include +#endif + +namespace flm::file_access { + +#if defined(FLM_CORELIB_TESTING) +using OpenObserver = std::function; +inline OpenObserver open_observer; + +inline void SetOpenObserver(OpenObserver observer) { + open_observer = std::move(observer); +} + +inline void ObserveOpen(const std::filesystem::path& path) { + if (open_observer) open_observer(path); +} +#else +inline void ObserveOpen(const std::filesystem::path&) {} +#endif + +} // namespace flm::file_access diff --git a/src/model_info.json b/src/model_info.json index cb62db94..4252a6de 100644 --- a/src/model_info.json +++ b/src/model_info.json @@ -3300,5 +3300,31 @@ "xetHash": "7c4d2da22b3de2ed3f3eae66c7034386df6a3c5d81039ad4a1c8067e7eaf0069", "path": "vision_weights.q4nx" } + ], + "phi4-mini-it-aie4:4b": [ + { + "type": "file", + "path": "Phi-4-mini-instruct.Q8_0.gguf", + "size": 4084611040, + "sha256": "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42" + }, + { + "type": "file", + "path": "tokenizer.json", + "size": 15524095, + "sha256": "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea" + }, + { + "type": "file", + "path": "tokenizer_config.json", + "size": 2932, + "sha256": "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db" + }, + { + "type": "file", + "path": "config.json", + "size": 2504, + "sha256": "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b" + } ] } \ No newline at end of file diff --git a/src/model_list.json b/src/model_list.json index d2c7b656..8600df7e 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -469,6 +469,47 @@ "footprint": 3.4 } }, + "phi4-mini-it-aie4": { + "4b": { + "name": "phi4-mini-it-aie4", + "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "size": 4100140571, + "default_context_length": 4096, + "max_prefill_len": 4096, + "details": { + "family": "phi4", + "think": false, + "think_toggleable": false, + "parameter_size": "4B", + "quantization_level": "Q8_0 -> AIE4 group-64", + "execution_backend": "corelib_aie4_gguf" + }, + "flm_min_version": "1.0.3", + "vlm": false, + "files": [ + "Phi-4-mini-instruct.Q8_0.gguf", + "tokenizer.json", + "tokenizer_config.json", + "config.json" + ], + "file_sources": { + "tokenizer.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "tokenizer_config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + } + }, + "footprint": 4.1 + } + }, "embed-gemma": { "300m": { "name": "Embedding-Gemma-300M-NPU2", diff --git a/src/pull/download_model.cpp b/src/pull/download_model.cpp index 38ccf44a..923b8b94 100644 --- a/src/pull/download_model.cpp +++ b/src/pull/download_model.cpp @@ -15,6 +15,9 @@ #include "nlohmann/json.hpp" #include "picosha2.h" #include "sha1.hpp" +#ifdef _WIN32 +#include +#endif namespace download_utils { @@ -122,6 +125,129 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota return 0; } +namespace { + +FILE* open_part_file(const std::filesystem::path& path, bool append) { +#ifdef _WIN32 + return _wfopen(path.c_str(), append ? L"ab" : L"wb"); +#else + return fopen(path.c_str(), append ? "ab" : "wb"); +#endif +} + +bool promote_atomically(const std::filesystem::path& part, + const std::filesystem::path& destination) { +#ifdef _WIN32 + return MoveFileExW(part.c_str(), destination.c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) != 0; +#else + std::error_code error; + std::filesystem::rename(part, destination, error); + return !error; +#endif +} + +bool request_hash_matches(const DownloadRequest& request, + const std::filesystem::path& path) { + const std::string actual = request.hash_algorithm == HashAlgorithm::Sha256 + ? calculate_file_sha256(path.string()) + : calculate_git_blob_oid(path.string()); + return actual == request.expected_hash; +} + +} // namespace + +bool download_file_atomic(const DownloadRequest& request, + std::function progress_cb) { + if (request.expected_hash.empty()) { + std::cerr << "Missing expected hash for: " << request.destination << std::endl; + return false; + } + + std::error_code error; + std::filesystem::create_directories(request.destination.parent_path(), error); + if (error) { + std::cerr << "Failed to create download directory: " << error.message() << std::endl; + return false; + } + + const std::filesystem::path part(request.destination.string() + ".part"); + std::uint64_t offset = 0; + if (std::filesystem::exists(part, error)) { + offset = std::filesystem::file_size(part, error); + if (error) { + return false; + } + if (offset > request.expected_size) { + std::filesystem::remove(part, error); + if (error) { + return false; + } + offset = 0; + } + } + + if (offset < request.expected_size) { + CURL* curl = curl_easy_init(); + if (!curl) { + std::cerr << "Failed to initialize CURL" << std::endl; + return false; + } + FILE* fp = open_part_file(part, offset != 0); + if (!fp) { + curl_easy_cleanup(curl); + std::cerr << "Failed to open partial file for writing: " << part << std::endl; + return false; + } + + g_progress_bar_shown = false; + hide_cursor(); + curl_easy_setopt(curl, CURLOPT_URL, request.url.c_str()); + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data_to_file); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp); + curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); + curl_easy_setopt(curl, CURLOPT_USERAGENT, "FastFlowLM/1.0"); + curl_easy_setopt(curl, CURLOPT_TIMEOUT, 3600L); + if (offset != 0) { + curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, + static_cast(offset)); + } + if (progress_cb) { + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); + curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, progress_callback); + } + + const CURLcode result = curl_easy_perform(curl); + fclose(fp); + curl_easy_cleanup(curl); + show_cursor(); + if (g_progress_bar_shown) { + std::cout << std::endl; + } + if (result != CURLE_OK) { + std::cerr << "CURL error: " << curl_easy_strerror(result) << std::endl; + return false; // Keep the partial file for the next resume attempt. + } + } + + const std::uint64_t completed_size = std::filesystem::file_size(part, error); + if (error || completed_size != request.expected_size || + !request_hash_matches(request, part)) { + std::filesystem::remove(part, error); + header_print("FLM", "Downloaded file size or hash did not match."); + return false; + } + + if (!promote_atomically(part, request.destination)) { + std::cerr << "Failed to atomically promote: " << request.destination << std::endl; + return false; + } + header_print("FLM", "Download completed: " << request.destination.string()); + return true; +} + /// \brief Download a file from URL to a local file /// \param url the URL to download from /// \param local_path the local path to save the file @@ -218,6 +344,22 @@ static bool download_with_retry(const std::string& url, const std::string& local return false; } +static bool download_with_retry(const DownloadRequest& request, + std::function progress_cb, + int max_retries = 3) { + for (int attempt = 0; attempt < max_retries; ++attempt) { + if (download_file_atomic(request, progress_cb)) { + return true; + } + header_print("FLM", "Download failed (attempt " << (attempt + 1) << "/" << max_retries << ")"); + if (attempt + 1 < max_retries) { + header_print("FLM", "Retrying..."); + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + } + return false; +} + /// \brief Download content from URL to a string /// \param url the URL to download from /// \return the downloaded string @@ -268,6 +410,14 @@ bool download_multiple_files(const nlohmann::json downloads, std::string filename = std::filesystem::path(url).filename().string(); std::string remote_oid = file["oid"]; bool is_lfs = file["is_lfs"]; + DownloadRequest request{ + url, + local_path, + file["expected_size"].get(), + file.value("hash_algorithm", std::string()) == "sha256" + ? HashAlgorithm::Sha256 + : HashAlgorithm::GitBlobSha1, + remote_oid}; // cut "?download=true" if (filename.find("?download=true") != std::string::npos) { @@ -282,7 +432,7 @@ bool download_multiple_files(const nlohmann::json downloads, } }; - if (!download_with_retry(url, local_path, is_lfs, remote_oid, file_progress)) { + if (!download_with_retry(request, file_progress)) { std::cerr << "Failed to download: " << url << std::endl; //show_cursor(); // Show cursor on error return false; diff --git a/src/pull/download_model.hpp b/src/pull/download_model.hpp index 12e9a1a1..d160bf9d 100644 --- a/src/pull/download_model.hpp +++ b/src/pull/download_model.hpp @@ -7,6 +7,8 @@ #pragma once #include +#include +#include #include #include #include @@ -15,6 +17,16 @@ namespace download_utils { +enum class HashAlgorithm { Sha256, GitBlobSha1 }; + +struct DownloadRequest { + std::string url; + std::filesystem::path destination; + std::uint64_t expected_size; + HashAlgorithm hash_algorithm; + std::string expected_hash; +}; + std::string calculate_file_sha256(const std::string& file_path); std::string calculate_git_blob_oid(const std::string& file_path); @@ -35,6 +47,11 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota bool download_file(const std::string& url, const std::string& local_path, bool is_lfs, std::string remote_oid, std::function progress_cb = nullptr); +// Download to a same-directory temporary file, verify it, then atomically promote it. +bool download_file_atomic( + const DownloadRequest& request, + std::function progress_cb = nullptr); + // Download content from URL to a string std::string download_string(const std::string& url); diff --git a/src/pull/model_downloader.cpp b/src/pull/model_downloader.cpp index 24fc98d4..9b545480 100644 --- a/src/pull/model_downloader.cpp +++ b/src/pull/model_downloader.cpp @@ -10,6 +10,140 @@ #include #include #include +#include +#include + +namespace { + +std::string percent_encode_filename(std::string_view filename) { + static constexpr char kHex[] = "0123456789ABCDEF"; + std::string encoded; + for (const unsigned char ch : filename) { + if ((ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') || + (ch >= '0' && ch <= '9') || ch == '-' || ch == '_' || ch == '.' || ch == '~') { + encoded.push_back(static_cast(ch)); + } else { + encoded.push_back('%'); + encoded.push_back(kHex[ch >> 4]); + encoded.push_back(kHex[ch & 0x0f]); + } + } + return encoded; +} + +bool is_hex_revision(const std::string& revision) { + return revision.size() == 40 && + std::all_of(revision.begin(), revision.end(), [](unsigned char ch) { + return std::isxdigit(ch) != 0; + }); +} + +nlohmann::json load_model_file_records(const std::string& model_tag) { + std::ifstream stream(utils::find_model_info()); + if (!stream.is_open()) { + throw std::runtime_error("model_info.json could not be opened"); + } + return nlohmann::json::parse(stream).at(model_tag); +} + +const nlohmann::json& find_file_record(const nlohmann::json& records, + const std::string& filename) { + const auto record = std::find_if(records.begin(), records.end(), [&](const auto& value) { + return value.at("path") == filename; + }); + if (record == records.end()) { + throw std::runtime_error("missing model_info record for " + filename); + } + return *record; +} + +struct ResolvedModelFile { + ModelFileSource source; + std::uint64_t size; + bool is_lfs; + download_utils::HashAlgorithm hash_algorithm; + std::string hash; +}; + +bool uses_pinned_aie4_integrity(const nlohmann::json& model_info) { + const auto details = model_info.find("details"); + return details != model_info.end() && details->is_object() && + details->value("execution_backend", std::string()) == "corelib_aie4_gguf"; +} + +} // namespace + +ModelFileSource resolve_file_source(const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope) { + if (model_info.contains("file_sources")) { + if (use_modelscope) { + throw std::runtime_error("pinned Hugging Face per-file sources are required; --modelscope is not supported"); + } + const auto& sources = model_info.at("file_sources"); + if (!sources.is_object()) { + throw std::runtime_error("file_sources must be an object"); + } + std::unordered_set files; + for (const auto& file : model_info.at("files")) { + files.insert(file.get()); + } + for (const auto& [key, value] : sources.items()) { + if (!files.contains(key)) { + throw std::runtime_error("unknown file_sources key: " + key); + } + if (!value.is_object() || value.size() != 2 || + !value.contains("url") || !value.at("url").is_string() || + value.at("url").get().empty()) { + throw std::runtime_error("file source requires exactly a non-empty string url and revision"); + } + if (!value.contains("revision") || !value.at("revision").is_string() || + !is_hex_revision(value.at("revision").get())) { + throw std::runtime_error("file source revision must be a 40-character hexadecimal string"); + } + } + const auto override = sources.find(std::string(filename)); + if (override != sources.end()) { + const std::string base = override->at("url"); + const std::string revision = override->at("revision"); + return {base + "/resolve/" + revision + "/" + + percent_encode_filename(filename) + "?download=true", + revision}; + } + } + + const std::string base_url = use_modelscope + ? model_info.at("ms_url").get() + : model_info.at("url").get(); + if (base_url.find("resolve") != std::string::npos) { + return {base_url + "/" + std::string(filename) + "?download=true", {}}; + } + return {base_url + "/resolve/main/" + std::string(filename) + "?download=true", {}}; +} + +namespace { + +ResolvedModelFile resolve_model_file(const nlohmann::json& model_info, + const nlohmann::json& records, + const std::string& filename, + bool use_modelscope) { + const auto& record = find_file_record(records, filename); + const bool is_lfs = record.contains("lfs"); + const bool has_explicit_sha256 = record.contains("sha256"); + return { + resolve_file_source(model_info, filename, use_modelscope), + record.at("size").get(), + is_lfs, + has_explicit_sha256 || is_lfs + ? download_utils::HashAlgorithm::Sha256 + : download_utils::HashAlgorithm::GitBlobSha1, + has_explicit_sha256 + ? record.at("sha256").get() + : (is_lfs ? record.at("lfs").at("oid").get() + : record.at("oid").get())}; +} + +} // namespace /// \brief Constructor /// \param models the model list @@ -22,23 +156,27 @@ ModelDownloader::ModelDownloader(model_list& models) /// \param model_tag the model tag /// \return true if the model is downloaded, false otherwise ModelDownloader::ModelStatus ModelDownloader::is_model_downloaded(const std::string& model_tag, bool sub_process_mode, bool fast_check) { - auto missing_files = get_missing_files(model_tag); + const auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const bool strict_integrity = uses_pinned_aie4_integrity(model_info); + auto missing_files = get_missing_files(new_model_tag); bool is_config_file_missing = std::find(missing_files.begin(), missing_files.end(), "config.json") != missing_files.end(); ModelStatus modelstatus = ModelStatus::Missing; if (!is_config_file_missing) { - modelstatus = check_model_compatibility(model_tag, sub_process_mode); + modelstatus = check_model_compatibility(new_model_tag, sub_process_mode); if (modelstatus == ModelStatus::Outdated) { if (!fast_check) { header_print("FLM", "Checking outdated files..."); - verify_and_clean_files(model_tag, sub_process_mode); + verify_and_clean_files(new_model_tag, false, sub_process_mode); } } - else if (modelstatus == ModelStatus::Ready && !missing_files.empty()) { - // config.json is present and the version check passed, but other - // files (e.g. weights) are still missing. - modelstatus = ModelStatus::Missing; + else if (modelstatus == ModelStatus::Ready) { + if (!missing_files.empty() || + (strict_integrity && !fast_check && + !verify_and_clean_files(new_model_tag, false, sub_process_mode))) { + modelstatus = ModelStatus::Missing; + } } } return modelstatus; @@ -51,8 +189,12 @@ ModelDownloader::ModelStatus ModelDownloader::check_model_compatibility(const st auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); LM_Config config; config.from_pretrained(this->supported_models.get_model_path(new_model_tag)); - std::string flm_version = config.flm_version; std::string flm_min_version = model_info["flm_min_version"]; + // The pinned Microsoft frontend config is upstream-native and intentionally + // has no FLM version. Its catalog contract supplies the compatibility floor. + std::string flm_version = uses_pinned_aie4_integrity(model_info) + ? flm_min_version + : config.flm_version; int l_l, m_l, r_l; //left, middle, right on local version int l_r, m_r, r_r; //left, middle, right on requried version int l_f, m_f, r_f; //left, middle, right on flm version @@ -89,6 +231,10 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::string model_name = model_info["name"]; std::string model_server = use_modelscope ? "ModelScope" : "HuggingFace"; + if (use_modelscope && model_info.contains("file_sources")) { + // Validate this before any ready-state early return. + resolve_file_source(model_info, model_info.at("files").at(0).get(), true); + } header_print("FLM", "Pulling model from " + model_server + "..."); header_print("FLM", "Model: " + new_model_tag); @@ -101,9 +247,13 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco header_print("FLM", "Model already downloaded. Use --force to re-download."); return true; } - verify_and_clean_files(new_model_tag, use_modelscope); break; case ModelStatus::Missing: + if (uses_pinned_aie4_integrity(model_info)) { + // Preserve valid finals, but remove corrupt pinned finals before + // deciding which files need to be downloaded. + verify_and_clean_files(new_model_tag, use_modelscope, true); + } break; case ModelStatus::Outdated: break; @@ -137,12 +287,13 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco } // Build download list - auto download_list = build_download_list(new_model_tag, use_modelscope); + auto download_list = build_download_list(new_model_tag, use_modelscope, force_redownload); auto downloads = download_list.first; float sum_fize_size = download_list.second; if (downloads.empty()) { header_print("FLM", "No files to download for model: " + new_model_tag); - return true; // Return true since all files are already present + return !uses_pinned_aie4_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope); } header_print("FLM", "Downloading " + std::to_string(downloads.size()) + " missing files..."); @@ -162,17 +313,17 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco if (success) { header_print("FLM", "Model downloaded successfully!"); - // Verify download + // Verify every final file using the same pinned metadata used to download it. auto final_missing = get_missing_files(new_model_tag); - if (final_missing.empty()) { + const bool verified = final_missing.empty() && + (!uses_pinned_aie4_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope)); + if (verified) { header_print("FLM", "All files verified successfully."); } else { - header_print("WARNING", "Some files may be missing after download:"); - for (const auto& file : final_missing) { - std::cout << " - " << file << std::endl; - } + header_print("WARNING", "Some files are missing or failed verification after download."); } - return true; + return verified; } else { header_print("ERROR", "Failed to download model files."); return false; @@ -285,82 +436,39 @@ std::string ModelDownloader::get_model_file_path(const std::string& model_path, /// \brief Build the download list /// \param model_tag the model tag /// \return the download list -std::pair ModelDownloader::build_download_list(const std::string& model_tag, bool modelscope) { - +std::pair ModelDownloader::build_download_list( + const std::string& model_tag, bool modelscope, bool force_redownload) { nlohmann::json downloads = nlohmann::json::array(); float sum_file_size = 0; - try { - auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); - std::string base_url = modelscope ? model_info["ms_url"] : model_info["url"]; - std::string model_name = model_info["name"]; - std::string file_url = model_info["file_url"]; - std::vector model_files = model_info["files"]; - - // Create model directory - std::string model_path = supported_models.get_model_path(new_model_tag); - std::filesystem::create_directories(model_path); - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } - - for (const auto& filename : model_files) { - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - - const auto& file = *it; - std::string local_path = get_model_file_path(model_path, filename); - - if (!file_exists(local_path)) { - std::string url; - if (std::string(base_url).find("resolve") != std::string::npos) { // resolve provided , may from a specific branch - url = base_url + "/" + filename + "?download=true"; - } - else { - url = base_url + "/resolve/main/" + filename + "?download=true"; - } - // header_print("URL", url); - bool is_lfs = file.contains("lfs"); - std::string oid = is_lfs ? file["lfs"]["oid"] : file["oid"]; - float file_size = static_cast(file["size"]) / 1024 / 1024; - sum_file_size += file_size; - - nlohmann::json entry = { - {"file", filename}, - {"size", file_size}, - {"url", url}, - {"localpath", local_path}, - {"oid", oid}, - {"is_lfs", is_lfs}, - }; - downloads.push_back(entry); - } - + auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const std::vector model_files = model_info.at("files"); + const std::string model_path = supported_models.get_model_path(new_model_tag); + std::filesystem::create_directories(model_path); + const nlohmann::json records = load_model_file_records(new_model_tag); + + for (const auto& filename : model_files) { + const std::string local_path = get_model_file_path(model_path, filename); + if (!force_redownload && file_exists(local_path)) { + continue; } - } - catch (const std::exception& e) { - header_print("ERROR", "Error building download list: " + std::string(e.what())); - } - return std::make_pair(downloads, sum_file_size); + const auto file = resolve_model_file(model_info, records, filename, modelscope); + const float file_size = static_cast(file.size) / 1024 / 1024; + sum_file_size += file_size; + downloads.push_back({ + {"file", filename}, + {"size", file_size}, + {"expected_size", file.size}, + {"url", file.source.url}, + {"localpath", local_path}, + {"oid", file.hash}, + {"is_lfs", file.is_lfs}, + {"hash_algorithm", file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? "sha256" : "git_blob_sha1"}, + }); + } + return {downloads, sum_file_size}; } /// \brief Remove a model and all its files @@ -421,18 +529,30 @@ bool ModelDownloader::remove_model(const std::string& model_tag, bool sub_proces /// \return true if all files are present and compatible, false otherwise bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelscope, bool sub_process_mode) { auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + if (use_modelscope && model_info.contains("file_sources")) { + try { + resolve_file_source( + model_info, model_info.at("files").at(0).get(), true); + } + catch (const std::exception& error) { + header_print("ERROR", error.what()); + return false; + } + } header_print("FLM", "Checking model: " + new_model_tag + "...\n"); - ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode); + // check_model owns the one full integrity pass below. Status discovery must + // remain presence/version-only so a pinned 4.1 GB model is not hashed twice. + ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode, true); switch (status) { case ModelStatus::Missing: header_print("FLM", "Model not found: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to download it."); - return true; + return false; case ModelStatus::Incompatible: header_print("FLM", "Model is incompatible with this version of FastFlowLM: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to re-download it."); - return true; + return false; case ModelStatus::Outdated: case ModelStatus::Ready: { bool ok = verify_and_clean_files(new_model_tag, use_modelscope, sub_process_mode); @@ -440,10 +560,10 @@ bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelsc header_print("FLM", "Model check completed with errors. Use `flm pull " + new_model_tag + "` to re-download corrupted files."); else header_print("FLM", "Model check completed successfully. All files are present and compatible."); - return true; + return ok; } } - return true; + return false; } /// \brief Verify each model file's hash against HuggingFace metadata and @@ -457,37 +577,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::vector model_files = model_info["files"]; std::string model_path = supported_models.get_model_path(new_model_tag); - std::string file_url = model_info["file_url"]; - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (use_modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } + const nlohmann::json records = load_model_file_records(new_model_tag); for (const auto& filename : model_files) { if (!sub_process_mode) { header_print("FLM", "Checking file: " + filename + "..."); } - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - const auto& file = *it; + const auto file = resolve_model_file( + model_info, records, filename, use_modelscope); std::string local_path = get_model_file_path(model_path, filename); // If the file isn't present locally, there's nothing to verify or @@ -498,11 +596,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool continue; } - bool is_lfs = file.contains("lfs"); - std::string oid_ref = is_lfs ? file["lfs"]["oid"] : file["oid"]; - std::string local_oid = is_lfs ? download_utils::calculate_file_sha256(local_path) : download_utils::calculate_git_blob_oid(local_path); + const std::string local_oid = + file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? download_utils::calculate_file_sha256(local_path) + : download_utils::calculate_git_blob_oid(local_path); + std::error_code size_error; + const auto local_size = std::filesystem::file_size(local_path, size_error); + const bool size_matches = !size_error && local_size == file.size; - if (local_oid == oid_ref) { + if (size_matches && local_oid == file.hash) { if (!sub_process_mode) { header_print("FLM", "Success!"); } diff --git a/src/pull/model_downloader.hpp b/src/pull/model_downloader.hpp index 52a90f81..4c1bd1fb 100644 --- a/src/pull/model_downloader.hpp +++ b/src/pull/model_downloader.hpp @@ -14,6 +14,17 @@ #include #include #include +#include + +struct ModelFileSource { + std::string url; + std::string revision; +}; + +ModelFileSource resolve_file_source( + const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope); class ModelDownloader { public: @@ -59,7 +70,8 @@ class ModelDownloader { std::string get_model_file_path(const std::string& model_path, const std::string& filename); // Build download URLs for model files - std::pair build_download_list(const std::string& model_tag, bool modelscope=0); + std::pair build_download_list( + const std::string& model_tag, bool modelscope=0, bool force_redownload=false); // bool check_model_compatibility(const std::string& model_tag); ModelStatus check_model_compatibility(const std::string& model_tag, bool sub_process_mode=0); diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index d38e5e9c..c81e52a9 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -58,7 +58,7 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra this->tag = auto_model.first; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -73,7 +73,9 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra // header_print("ASR", asr_supported); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); @@ -358,6 +360,8 @@ void Runner::run() { chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; uniformed_input.prompt = input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(this->generate_limit); this->auto_chat_engine->start_total_timer(); @@ -431,7 +435,7 @@ void Runner::cmd_load(std::vector& input_list) { if (model_name != this->tag) { this->tag = model_name; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -450,7 +454,9 @@ void Runner::cmd_load(std::vector& input_list) { auto [new_tag, model_info] = this->supported_models.get_model_info(this->tag); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index 95e90a3a..ec692ac8 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -20,6 +20,21 @@ #include #include "server.hpp" +namespace { +json ModelErrorResponse(const ModelRequestError& error) { + return {{"error", {{"message", error.what()}, + {"type", "model_error"}, + {"code", error.http_code()}, + {"session_cleared", error.session_cleared()}}}}; +} + +json ExceptionResponse(const std::exception& error) { + if (const auto* model_error = dynamic_cast(&error)) + return ModelErrorResponse(*model_error); + return {{"error", error.what()}}; +} +} + ///@brief Normalize messages by merging consecutive user messages (like Ollama does) ///@param messages the original messages ///@return normalized messages with consecutive user messages merged @@ -387,7 +402,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { std::pair> auto_model = get_auto_model(ensure_tag, this->supported_models, &this->npu_device_inst); auto_chat_engine = std::move(auto_model.second); ensure_tag = auto_model.first; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -400,7 +415,9 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { auto [new_ensure_tag, model_info] = supported_models.get_model_info(ensure_tag); auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { + const auto load_started = std::chrono::steady_clock::now(); auto_chat_engine->load_model(supported_models.get_model_path(new_ensure_tag), model_info, ctx_length, preemption); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); @@ -424,7 +441,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -456,7 +473,7 @@ void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { void RestHandler::ensure_embed_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (this->downloader.is_model_downloaded(ensure_tag)) { + switch (this->downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -655,6 +672,11 @@ void RestHandler::handle_generate(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); meta_info.max_prefill_len = this->prefill_chunk_len; meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; header_print("FLM", "Start generating..."); @@ -665,7 +687,7 @@ void RestHandler::handle_generate(const json& request, streaming_ostream ostream(model, send_streaming_response, false); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -673,15 +695,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -698,7 +720,7 @@ void RestHandler::handle_generate(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -706,15 +728,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -738,7 +760,7 @@ void RestHandler::handle_generate(const json& request, send_response(response); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -772,6 +794,11 @@ void RestHandler::handle_chat(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + options.contains("num_predict") + ? std::optional(options.at("num_predict").get()) + : std::nullopt); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; header_print("FLM", "Start generating..."); @@ -781,7 +808,7 @@ void RestHandler::handle_chat(const json& request, streaming_ostream ostream(model, send_streaming_response, true); // true for chat format uniformed_input.messages = messages; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -789,21 +816,16 @@ void RestHandler::handle_chat(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); - if (!success){ - json error_response = {{"error", "Max length reached"}}; - send_response(error_response); - this->auto_chat_engine->clear_context(); - return; - } + auto_chat_engine->generate(meta_info, length_limit, ostream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -823,9 +845,11 @@ void RestHandler::handle_chat(const json& request, //std::string response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, std::cout); std::string response_text; try { - response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, nstream); + response_text = auto_chat_engine->generate_with_prompt( + meta_info, uniformed_input, length_limit, nstream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -857,7 +881,7 @@ void RestHandler::handle_chat(const json& request, this->auto_chat_engine->clear_context(); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -915,7 +939,7 @@ void RestHandler::handle_embeddings(const json& request, send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -931,7 +955,7 @@ void RestHandler::handle_models(const json& request, json models = supported_models.get_all_models_ollama(); send_response(models); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1020,7 +1044,7 @@ void RestHandler::handle_ps(const json& request, // std::cout << "response: " << response.dump(4) << std::endl; send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1146,13 +1170,22 @@ void RestHandler::handle_openai_chat_completion(const json& request, lm_uniform_input_t uniformed_input; uniformed_input.messages = current_messages; uniformed_input.tools = tools; + const std::optional openai_chat_budget = request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : request.contains("max_completion_tokens") + ? std::optional(request.at("max_completion_tokens").get()) + : std::nullopt; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(openai_chat_budget); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; if (stream){ // Create a wrapper callback that passes the pre-formatted SSE string directly cancellation_token->reset(); auto_chat_engine->reset_parser(); - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; @@ -1184,7 +1217,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1193,8 +1226,18 @@ void RestHandler::handle_openai_chat_completion(const json& request, header_print("FLM", "Start generating..."); try { auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + if (error.session_cleared()) this->prompt_cache.reset(); + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1236,7 +1279,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1246,7 +1289,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, try { response_text = auto_chat_engine->generate(meta_info, length_limit, nstream, [&] { return cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1389,18 +1432,25 @@ void RestHandler::handle_openai_completion(const json& request, chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); header_print("FLM", "Start generating..."); if (stream) { // Create a wrapper callback that passes the pre-formatted SSE string directly - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; streaming_ostream_openai ostream(model, openai_stream_callback); // streaming in completion format uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1408,15 +1458,24 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -1431,7 +1490,7 @@ void RestHandler::handle_openai_completion(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1439,15 +1498,15 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; diff --git a/src/server/server.cpp b/src/server/server.cpp index bc612211..04f39df1 100644 --- a/src/server/server.cpp +++ b/src/server/server.cpp @@ -156,19 +156,6 @@ int NPUAccessManager::get_active_npu_requests() { return g_npu_active_requests.load(); } -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path) { - // NPU-intensive endpoints that should be restricted to one user at a time - if (method == "POST") { - return path == "/api/generate" || - path == "/api/chat" || - path == "/v1/chat/completions" || - path == "/v1/audio/transcriptions" || - path == "/v1/embeddings"; - } - return false; -} - ///@brief HttpSession class implementation ///@param socket the socket ///@param server the server @@ -591,37 +578,15 @@ void WebServer::do_accept() { ///@brief process_next_npu_request Handles one queued NPU task at a time void WebServer::process_next_npu_request() { - { - std::lock_guard lock(npu_queue_mutex_); - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; // Queue is empty, NPU is free - } - } - - // NPU cooldown before running the next queued task. - constexpr auto npu_cooldown = std::chrono::milliseconds(333); - std::this_thread::sleep_for(npu_cooldown); - - std::function task; - size_t remaining = 0; - { - std::lock_guard lock(npu_queue_mutex_); - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; - } - - task = npu_request_queue_.front(); - npu_request_queue_.pop(); - remaining = npu_request_queue_.size(); - } - - header_print("🟑 ", "Dequeuing NPU request (" + std::to_string(remaining) + " remaining)..."); - - // Post the task to be executed by the io_context - net::post(ioc, task); - + npu_request_coordinator_.complete_current( + [this](NPURequestCoordinator::Task task) { + const auto remaining = npu_request_coordinator_.size(); + header_print("🟑 ", "Dequeuing NPU request (" + + std::to_string(remaining) + " remaining)..."); + net::post(ioc, std::move(task)); + }, + [] { NPUAccessManager::release_npu_access(); }, + std::chrono::milliseconds(333)); } ///@brief handle request @@ -684,6 +649,9 @@ bool WebServer::handle_request(http::request& req, auto process_task = [this, it, req_ptr, res_ptr, session, needs_npu, key, is_json](bool is_deferred) { auto& req_ref = *req_ptr; auto& res_ref = *res_ptr; + NPURequestCompletionGuard completion([this, needs_npu] { + if (needs_npu) process_next_npu_request(); + }); // Parse JSON request body json request_json; @@ -704,10 +672,6 @@ bool WebServer::handle_request(http::request& req, // Only write from callback when deferred if (is_deferred && session) session->write_response_from_callback(); - - if (needs_npu) { - this->process_next_npu_request(); - } return; } @@ -737,10 +701,9 @@ bool WebServer::handle_request(http::request& req, if (code == 400) { status = http::status::bad_request; + } else if (code == 500) { + status = http::status::internal_server_error; } - //else if () { - - //} } response_ref.result(status); @@ -750,10 +713,6 @@ bool WebServer::handle_request(http::request& req, cancellation_token->complete(); unregister_active_request(request_id); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -769,10 +728,6 @@ bool WebServer::handle_request(http::request& req, } if (is_final) { unregister_active_request(request_id); - - if (needs_npu) { - this->process_next_npu_request(); - } } }; @@ -788,10 +743,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -805,10 +756,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -831,28 +778,25 @@ bool WebServer::handle_request(http::request& req, return false; } - //const int NPU_QUEUE_LIMIT = 10; - std::lock_guard lock(npu_queue_mutex_); - - if (npu_request_queue_.size() >= max_npu_queue_) { + if (!npu_request_coordinator_.try_enqueue([this, process_task]() { + process_task(true); + })) { res.result(http::status::service_unavailable); res.body() = json{ - {"error", "NPU is in use and request queue is full (limit: " + std::to_string(max_npu_queue_) + "). Please try again later."} + {"error", "NPU is in use and request queue is full (limit: " + + std::to_string(npu_request_coordinator_.capacity()) + + "). Please try again later."} }.dump(); res.set(http::field::content_type, "application/json"); res.prepare_payload(); header_print("🚫 ", "NPU busy and queue full, request denied: " + key); return false; } - else { - // Create a new lambda to bind process_task(true) - npu_request_queue_.push([this, process_task]() { - process_task(true); - }); - header_print("πŸ•’ ", "NPU busy, request queued (" + std::to_string(npu_request_queue_.size()) + "/" + std::to_string(max_npu_queue_) + "): " + key); - return true; - } + header_print("πŸ•’ ", "NPU busy, request queued (" + + std::to_string(npu_request_coordinator_.size()) + "/" + + std::to_string(npu_request_coordinator_.capacity()) + "): " + key); + return true; } ///@brief create lm server diff --git a/src/server/server.hpp b/src/server/server.hpp index 910c5884..de1f5198 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -47,8 +47,88 @@ extern std::mutex g_npu_access_mutex; extern std::atomic g_npu_in_use; extern std::atomic g_npu_active_requests; -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path); +// Helper function to check if an endpoint requires serialized accelerator access. +inline bool requires_npu_access(const std::string& method, const std::string& path) { + if (method != "POST") return false; + return path == "/api/generate" || path == "/api/chat" || + path == "/v1/chat/completions" || path == "/v1/completions" || + path == "/v1/audio/transcriptions" || path == "/v1/embeddings"; +} + +class NPURequestCoordinator final { +public: + using Task = std::function; + using Scheduler = std::function; + + explicit NPURequestCoordinator(std::size_t capacity = 10) + : capacity_(capacity) {} + void set_capacity(std::size_t capacity) { + std::lock_guard lock(mutex_); + capacity_ = capacity; + } + bool try_enqueue(Task task) { + std::lock_guard lock(mutex_); + if (tasks_.size() >= capacity_) return false; + tasks_.push(std::move(task)); + return true; + } + Task take_next() { + std::lock_guard lock(mutex_); + if (tasks_.empty()) return {}; + auto task = std::move(tasks_.front()); + tasks_.pop(); + return task; + } + void complete_current(const Scheduler& schedule, + const std::function& release, + std::chrono::milliseconds cooldown) { + auto task = take_next(); + if (!task) { + release(); + return; + } + if (cooldown.count() > 0) std::this_thread::sleep_for(cooldown); + schedule(std::move(task)); + } + bool empty() const { + std::lock_guard lock(mutex_); + return tasks_.empty(); + } + std::size_t size() const { + std::lock_guard lock(mutex_); + return tasks_.size(); + } + std::size_t capacity() const { + std::lock_guard lock(mutex_); + return capacity_; + } +private: + mutable std::mutex mutex_; + std::queue tasks_; + std::size_t capacity_; +}; + +class NPURequestCompletionGuard final { +public: + explicit NPURequestCompletionGuard(std::function completion) + : completion_(std::move(completion)) {} + NPURequestCompletionGuard(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard& operator=(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard(NPURequestCompletionGuard&& other) noexcept + : completion_(std::move(other.completion_)), active_(other.active_) { + other.active_ = false; + } + NPURequestCompletionGuard& operator=(NPURequestCompletionGuard&&) = delete; + ~NPURequestCompletionGuard() { complete(); } + void complete() noexcept { + if (!active_) return; + active_ = false; + try { if (completion_) completion_(); } catch (...) {} + } +private: + std::function completion_; + bool active_ = true; +}; ///@brief get current time string, format: hh:mm:ss mm:dd:yyyy ///@return the current time string @@ -116,7 +196,7 @@ class WebServer { void set_max_connections(size_t max_conns) { max_connections_ = max_conns; } void set_request_timeout(std::chrono::seconds timeout) { request_timeout_ = timeout; } void set_io_threads(size_t num_threads) { io_thread_count_ = num_threads; } - void set_npu_queue_length(size_t q_len) { max_npu_queue_ = q_len; } + void set_npu_queue_length(size_t q_len) { npu_request_coordinator_.set_capacity(q_len); } // Maximum accepted HTTP request body size (in bytes) void set_max_body_size_bytes(std::size_t bytes) { max_body_size_bytes_ = bytes; } std::size_t get_max_body_size_bytes() const { return max_body_size_bytes_; } @@ -161,7 +241,6 @@ class WebServer { std::chrono::seconds request_timeout_ = std::chrono::seconds(600); // 5 minutes size_t io_thread_count_ = 5; std::size_t max_body_size_bytes_ = 256ull * 1024 * 1024; // 256 MB default - size_t max_npu_queue_ = 10; // Request tracking mutable std::mutex active_requests_mutex_; @@ -170,8 +249,7 @@ class WebServer { // Connection tracking std::atomic active_connections_{0}; std::vector io_threads_; - std::queue> npu_request_queue_; - std::mutex npu_queue_mutex_; + NPURequestCoordinator npu_request_coordinator_; // Friend declaration for HttpSession to access private members friend class HttpSession; }; diff --git a/src/src/main.cpp b/src/src/main.cpp index 446b907b..983959d1 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -34,6 +34,9 @@ #include "utils/vm_args.hpp" #include #include "benchmarking.hpp" +#ifdef FLM_ENABLE_CORELIB_AIE4 +#include "corelib/corelib_runtime.hpp" +#endif #ifndef _WIN32 #include @@ -716,6 +719,9 @@ int main(int argc, char* argv[]) { return 1; } // Return 0 if the command is valid +#ifdef FLM_ENABLE_CORELIB_AIE4 + flm::corelib::CorelibRuntime::ShutdownProcess(); +#endif return 0; } catch (const std::exception& e) { // If an error occurs, this will be used to show the error diff --git a/src/test/phi4_corelib_aie4/CMakeLists.txt b/src/test/phi4_corelib_aie4/CMakeLists.txt new file mode 100644 index 00000000..f767c911 --- /dev/null +++ b/src/test/phi4_corelib_aie4/CMakeLists.txt @@ -0,0 +1,256 @@ +cmake_minimum_required(VERSION 3.22) +project(phi4_corelib_aie4_tests LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(NOT WIN32) + message(FATAL_ERROR "Phi-4 corelib AIE4 tests currently require Windows") +endif() + +find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) +find_path(BOOST_INCLUDE_DIR NAMES boost/program_options.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_path(XRT_INCLUDE_DIR NAMES xrt/xrt_bo.h + HINTS "$ENV{XRT_INCLUDE_DIR}" + "${CMAKE_CURRENT_LIST_DIR}/../../../../xrt_package/xrt/include" + "C:/dev/XRT/src/runtime_src/core/include" REQUIRED) +find_path(CURL_INCLUDE_DIR NAMES curl/curl.h + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_library(CURL_LIBRARY NAMES libcurl curl + HINTS "$ENV{CONDA_PREFIX}/Library/lib" + "$ENV{USERPROFILE}/anaconda3/Library/lib" REQUIRED) +find_package(CURL REQUIRED) +set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") +set(CORELIB_SOURCES + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/corelib_runtime.cpp") + +# A synthetic mismatched copy proves that the adapter rejects header drift at +# compile time. The caller-provided header remains untouched. +file(READ "${RYZENAI_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" CORELIB_HEADER_TEXT) +string(REGEX REPLACE + "#define RYZENAI_CORELIB_VERSION_PATCH[ \t]+0" + "#define RYZENAI_CORELIB_VERSION_PATCH 1" + WRONG_CORELIB_HEADER_TEXT "${CORELIB_HEADER_TEXT}") +set(WRONG_CORELIB_INCLUDE_DIR + "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/include") +file(MAKE_DIRECTORY "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai") +file(WRITE "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" + "${WRONG_CORELIB_HEADER_TEXT}") +file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + "#define RYZENAI_CORELIB_STATIC 1\n#include \"corelib/corelib_api.hpp\"\n") +try_compile(CORELIB_WRONG_VERSION_ACCEPTED + SOURCES "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + CMAKE_FLAGS + "-DCMAKE_CXX_STANDARD=20" + "-DCMAKE_CXX_FLAGS=/I${FLM_SOURCE_DIR}/include /I${WRONG_CORELIB_INCLUDE_DIR}" + OUTPUT_VARIABLE WRONG_CORELIB_COMPILE_OUTPUT) +if(CORELIB_WRONG_VERSION_ACCEPTED) + message(FATAL_ERROR "Corelib adapter accepted a non-0.3.0 header") +endif() + +add_executable(test_corelib_api + test_corelib_api.cpp fake_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_corelib_api PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_corelib_api PRIVATE + RYZENAI_CORELIB_STATIC=1 FLM_CORELIB_TESTING=1) + +add_executable(test_real_corelib test_real_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_real_corelib PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_gguf + test_phi4_gguf.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_gguf.cpp") +target_include_directories(test_phi4_gguf PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_gguf PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_host + test_phi4_host.cpp + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_host.cpp") +target_include_directories(test_phi4_host PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include") + +add_executable(test_phi4_shape_plan + test_phi4_shape_plan.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_shape_plan.cpp") +target_include_directories(test_phi4_shape_plan PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_shape_plan PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_engine + test_phi4_engine.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/corelib/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/corelib/corelib_runtime.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_gguf.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_host.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_shape_plan.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_aie4.cpp") +target_include_directories(test_phi4_engine PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_engine PRIVATE + RYZENAI_CORELIB_STATIC=1 USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_engine PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_engine PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_engine PRIVATE xrt_coreutil) + +set(PHI4_FRONTEND_SOURCES + "${FLM_SOURCE_DIR}/common/AutoModel/automodel.cpp" + "${FLM_SOURCE_DIR}/common/AutoModel/modeling_phi4.cpp") + +add_executable(test_phi4_frontend + test_phi4_frontend.cpp fake_corelib.cpp + ${PHI4_FRONTEND_SOURCES} + ${CORELIB_SOURCES} + "${FLM_SOURCE_DIR}/common/models/phi4/corelib/phi4_corelib_gguf.cpp") +target_include_directories(test_phi4_frontend PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend PRIVATE + FLM_ENABLE_CORELIB_AIE4=1 FLM_CORELIB_TESTING=1 + RYZENAI_CORELIB_STATIC=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend PRIVATE xrt_coreutil) + +add_executable(test_model_downloader + test_model_downloader.cpp + "${FLM_SOURCE_DIR}/pull/download_model.cpp" + "${FLM_SOURCE_DIR}/pull/model_downloader.cpp" + "${FLM_SOURCE_DIR}/common/utils.cpp") +target_include_directories(test_model_downloader PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/pull" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_model_downloader PRIVATE + FLM_SOURCE_DIR="${FLM_SOURCE_DIR}" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins" + __FLM_VERSION__="1.0.3" + __NPU_VERSION__="0.0.0.0" + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_model_downloader PRIVATE + $<$:/wd4005 /wd4244>) +target_link_libraries(test_model_downloader PRIVATE CURL::libcurl) + +add_executable(test_phi4_frontend_off + test_phi4_frontend.cpp + ${PHI4_FRONTEND_SOURCES}) +target_include_directories(test_phi4_frontend_off PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend_off PRIVATE + FLM_CORELIB_TESTING=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend_off PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend_off PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend_off PRIVATE xrt_coreutil) + +# Compile the actual production frontend translation units in both feature modes. +# Empty declaration-only FFmpeg headers isolate this compile check from an +# unrelated optional SDK that is absent on the standalone test host. +set(FRONTEND_STUB_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/frontend-compile-stubs") +foreach(STUB_HEADER IN ITEMS + libavcodec/avcodec.h + libavformat/avformat.h + libswscale/swscale.h + libavutil/imgutils.h + libavutil/frame.h + libavutil/pixfmt.h) + get_filename_component(STUB_PARENT + "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" DIRECTORY) + file(MAKE_DIRECTORY "${STUB_PARENT}") + file(WRITE "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" "#pragma once\n") +endforeach() + +set(FLM_PRODUCTION_FRONTEND_SOURCES + ${PHI4_FRONTEND_SOURCES} + "${FLM_SOURCE_DIR}/runner/runner.cpp" + "${FLM_SOURCE_DIR}/server/rest_handler.cpp" + "${FLM_SOURCE_DIR}/server/server.cpp") +function(add_frontend_compile_guard TARGET_NAME ENABLE_CORELIB) + add_library(${TARGET_NAME} OBJECT ${FLM_PRODUCTION_FRONTEND_SOURCES}) + target_include_directories(${TARGET_NAME} PRIVATE + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${FRONTEND_STUB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") + target_compile_definitions(${TARGET_NAME} PRIVATE + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX + __FLM_VERSION__="test" __NPU_VERSION__="0.0.0.0" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins") + target_compile_options(${TARGET_NAME} PRIVATE + $<$:/wd4005 /wd4244>) + if(ENABLE_CORELIB) + target_compile_definitions(${TARGET_NAME} PRIVATE + FLM_ENABLE_CORELIB_AIE4=1) + target_include_directories(${TARGET_NAME} PRIVATE + "${RYZENAI_CORELIB_INCLUDE_DIR}") + endif() +endfunction() + +# The OFF target deliberately has no RYZENAI_CORELIB_INCLUDE_DIR. +add_frontend_compile_guard(phi4_frontend_compile_off FALSE) +add_frontend_compile_guard(phi4_frontend_compile_on TRUE) + +include(CTest) +add_test(NAME test_corelib_api COMMAND test_corelib_api) +add_test(NAME test_real_corelib COMMAND test_real_corelib) +add_test(NAME test_phi4_gguf COMMAND test_phi4_gguf) +add_test(NAME test_phi4_host COMMAND test_phi4_host) +add_test(NAME test_phi4_shape_plan COMMAND test_phi4_shape_plan) +add_test(NAME test_phi4_engine COMMAND test_phi4_engine) +add_test(NAME test_phi4_frontend COMMAND test_phi4_frontend) +add_test(NAME test_phi4_frontend_off COMMAND test_phi4_frontend_off) +add_test(NAME test_model_downloader COMMAND test_model_downloader) +set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_corelib_aie4/fake_corelib.cpp b/src/test/phi4_corelib_aie4/fake_corelib.cpp new file mode 100644 index 00000000..2486bd59 --- /dev/null +++ b/src/test/phi4_corelib_aie4/fake_corelib.cpp @@ -0,0 +1,484 @@ +#include "fake_corelib.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +fake_corelib::State state; +std::recursive_mutex state_mutex; +thread_local std::string current_detail; + +struct FakeStorage { + std::size_t byte_size{}; + std::unique_ptr> bytes; +}; + +struct FakeObject { + std::string kind; + ryzenai_corelib_data_type data_type{ryzenai_corelib_data_type_bf16}; + std::vector shape; + std::size_t byte_size{}; + std::size_t window_offset{}; + std::shared_ptr storage; +}; + +void* NewObject(std::string kind = "generic") { + ++state.live_objects; + auto* object = new FakeObject; + object->kind = std::move(kind); + return object; +} + +ryzenai_corelib_status Status(std::string_view name) { + const auto configured = state.statuses.find(std::string(name)); + return configured == state.statuses.end() ? state.default_status + : configured->second; +} + +std::size_t Elements(const std::vector& shape) { + std::size_t result = 1; + for (const auto dimension : shape) result *= static_cast(dimension); + return result; +} + +std::size_t TypeBytes(ryzenai_corelib_data_type type) { + return RYZENAI_CORELIB_DATA_TYPE_BITS(type) / 8; +} + +std::int64_t PaddedRows(std::string_view helper, std::int64_t rows) { + const auto helpers = state.pad_row_overrides.find(std::string(helper)); + if (helpers != state.pad_row_overrides.end()) { + const auto found = helpers->second.find(rows); + if (found != helpers->second.end()) return found->second; + } + if (rows == 1 || state.pad_multiple <= 0) return rows; + return (rows + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; +} + +std::uint16_t Bf16(float value) { + std::uint32_t bits = std::bit_cast(value); + bits += 0x7fffU + ((bits >> 16) & 1U); + return static_cast(bits >> 16); +} + +float FloatFromBf16(std::uint16_t value) { + return std::bit_cast(static_cast(value) << 16); +} + +void EnsureStorage(FakeObject& object) { + if (!object.storage->bytes) + object.storage->bytes = std::make_unique>( + object.storage->byte_size, std::byte{0}); +} + +void ObserveCreateConcurrency() { + const int active = ++state.active_weight_creates; + int maximum = state.maximum_active_weight_creates.load(); + while (active > maximum && + !state.maximum_active_weight_creates.compare_exchange_weak(maximum, active)) {} +} + +#define FLM_DEFINE_FAKE_TAG(member, symbol) \ + struct member##_tag { \ + static constexpr std::string_view name = #symbol; \ + }; +FLM_CORELIB_FUNCTIONS(FLM_DEFINE_FAKE_TAG) +#undef FLM_DEFINE_FAKE_TAG + +template +inline constexpr bool kAlwaysFalse = false; + +template +struct TypedFake; + +template +struct TypedFake { + static Result Invoke(Args... args) { + std::unique_lock state_lock(state_mutex); + ++state.call_counts[std::string(Tag::name)]; + state.call_log.emplace_back(Tag::name); + auto arguments = std::forward_as_tuple(args...); + + if constexpr (std::is_same_v) { + if (std::get<0>(arguments)) *std::get<0>(arguments) = state.version.major; + if (std::get<1>(arguments)) *std::get<1>(arguments) = state.version.minor; + if (std::get<2>(arguments)) *std::get<2>(arguments) = state.version.patch; + return; + } else if constexpr (std::is_same_v) { + current_detail = "detail overwritten by status_to_string"; + return state.status_text.c_str(); + } else if constexpr (std::is_same_v) { + current_detail = state.detail; + return current_detail.c_str(); + } else if constexpr (std::is_same_v) { + return state.selftest_status; + } else if constexpr (std::is_same_v) { + return state.has_device_context; + } else if constexpr (std::is_same_v) { + void* object = std::get<0>(arguments); + if (object) { + delete static_cast(object); + --state.live_objects; + ++state.releases; + state.lifetime_events.emplace_back("release"); + } + return; + } else if constexpr (std::is_same_v) { + ++state.cleanup_calls; + state.lifetime_events.emplace_back("cleanup"); + return; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* out = std::get<0>(arguments); + if (out) *out = status == ryzenai_corelib_status_success ? NewObject("stream") : nullptr; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("tensor")); + object->data_type = type; + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(type); + object->storage = std::make_shared(); + object->storage->byte_size = object->byte_size; + *out = object; + state.tensor_creates.push_back({type, object->shape, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + void* parent = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + const auto offset = std::get<3>(arguments); + auto* out = std::get<4>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("window")); + if (parent) { + const auto* parent_object = static_cast(parent); + object->data_type = parent_object->data_type; + object->storage = parent_object->storage; + object->window_offset = parent_object->window_offset + offset; + } + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(object->data_type); + *out = object; + state.tensor_windows.push_back({parent, object->shape, offset, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->byte_size; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->data_type; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<1>(arguments); + const void* source = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + bool all_zero = true; + if (source) { + const auto* bytes = static_cast(source); + all_zero = std::all_of(bytes, bytes + count * TypeBytes(type), + [](unsigned char value) { return value == 0; }); + } + state.tensor_writes.push_back({std::get<0>(arguments), type, count, offset, all_zero}); + auto* object = static_cast(std::get<0>(arguments)); + if (status == ryzenai_corelib_status_success && object && source) { + const auto target_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + if (!all_zero || object->storage->bytes) EnsureStorage(*object); + if (object->storage->bytes) { + auto* target = object->storage->bytes->data() + target_offset; + if (object->data_type == type) { + std::memcpy(target, source, count * TypeBytes(type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + type == ryzenai_corelib_data_type_fp32) { + const auto* values = static_cast(source); + for (std::size_t i = 0; i < count; ++i) { + const auto converted = Bf16(values[i]); + std::memcpy(target + i * sizeof(converted), &converted, + sizeof(converted)); + } + } + } + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* object = static_cast(std::get<0>(arguments)); + const auto destination_type = std::get<1>(arguments); + void* destination = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + if (status == ryzenai_corelib_status_success && destination) { + std::memset(destination, 0, count * TypeBytes(destination_type)); + if (object && object->storage && object->storage->bytes) { + const auto source_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + const auto* source = object->storage->bytes->data() + source_offset; + if (object->data_type == destination_type) { + std::memcpy(destination, source, + count * TypeBytes(destination_type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + destination_type == ryzenai_corelib_data_type_fp32) { + auto* values = static_cast(destination); + for (std::size_t i = 0; i < count; ++i) { + std::uint16_t encoded; + std::memcpy(&encoded, source + i * sizeof(encoded), + sizeof(encoded)); + values[i] = FloatFromBf16(encoded); + } + } + } + } + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + auto* k = std::get<1>(arguments); + auto* n = std::get<2>(arguments); + const auto group = std::get<3>(arguments); + state.matmul_pad_calls.push_back({m ? *m : -1, k ? *k : -1, + n ? *n : -1, group}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success) { + if (m) *m = PaddedRows(n && *n == 1024 ? "matmul-1024" : "matmul-3072", *m); + if (k) *k += state.matmul_k_delta; + if (n) *n += state.matmul_n_delta; + } + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + state.rows_pad_calls.push_back({"ssmlp", m ? *m : -1, + std::get<1>(arguments), std::get<2>(arguments), std::get<3>(arguments)}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("ssmlp", *m); + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<0>(arguments); + auto* desc = std::get<1>(arguments); + state.mha_pad_calls.push_back({m ? *m : -1, desc ? *desc : ryzenai_corelib_flat_mha_bf16_desc{}}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("mha", *m); + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + if (desc && components) state.weight_creates.push_back({"matmul", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), {components->blocks}}); + if (status == ryzenai_corelib_status_success && out) *out = NewObject("matmul_weights"); + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + if (desc && components) { + fake_corelib::WeightCreateRecord record{"ssmlp", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), + {components->gate_blocks, components->up_blocks, components->down_blocks}}; + if (components->epsilon) record.epsilon = *static_cast(components->epsilon); + if (components->norm0) record.norm0.assign(static_cast(components->norm0), + static_cast(components->norm0) + desc->k); + if (components->norm1) record.norm1.assign(static_cast(components->norm1), + static_cast(components->norm1) + desc->k); + state.weight_creates.push_back(std::move(record)); + } + if (status == ryzenai_corelib_status_success && out) *out = NewObject("ssmlp_weights"); + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + state.work_in_flight = false; + return Status(Tag::name); + } else if constexpr (std::is_same_v || + std::is_same_v || + std::is_same_v) { + if (state.statuses.contains("test_observe_dispatch_concurrency")) { + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} + state_lock.unlock(); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + state_lock.lock(); + --state.active_leases; + } + const auto status = Status(Tag::name); + if (status != ryzenai_corelib_status_success) return status; + fake_corelib::DispatchRecord record{}; + record.thread_id = std::this_thread::get_id(); + record.kind = std::is_same_v ? "matmul" : + std::is_same_v ? "ssmlp" : "mha"; + record.stream = std::get<0>(arguments); + if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); record.rows = std::get<2>(arguments); + record.output = std::get<4>(arguments); + } else if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); record.rows = std::get<3>(arguments); + record.output = std::get<6>(arguments); + } else { + record.input = std::get<2>(arguments); record.rows = std::get<4>(arguments); + record.position = std::get<5>(arguments); record.output = std::get<10>(arguments); + } + if (record.output && static_cast(record.output)->kind == "window") + record.window_offset = static_cast(record.output)->window_offset; + state.dispatches.push_back(record); + if constexpr (std::is_same_v) { + auto* output = static_cast(record.output); + if (output && output->shape == std::vector({1, 200064})) { + EnsureStorage(*output); + const auto value = Bf16(1.0f); + std::memcpy(output->storage->bytes->data() + + output->window_offset * TypeBytes(output->data_type), + &value, sizeof(value)); + } + } + state.work_in_flight = true; + if (state.fail_after_submit == Tag::name) return ryzenai_corelib_status_failure; + return ryzenai_corelib_status_success; + } else if constexpr (std::is_same_v) { + return Status(Tag::name); + } else { + static_assert(kAlwaysFalse, "unhandled fake corelib ABI result"); + } + } +}; + +#define FLM_ASSERT_FAKE_ABI(member, symbol) \ + static_assert(std::is_same_v< \ + decltype(&TypedFake::Invoke), \ + decltype(&::symbol)>); +FLM_CORELIB_FUNCTIONS(FLM_ASSERT_FAKE_ABI) +#undef FLM_ASSERT_FAKE_ABI + +void* FunctionFor(std::string_view name) { +#define FLM_MAP_FAKE_FUNCTION(member, symbol) \ + if (name == #symbol) { \ + return reinterpret_cast( \ + &TypedFake::Invoke); \ + } + FLM_CORELIB_FUNCTIONS(FLM_MAP_FAKE_FUNCTION) +#undef FLM_MAP_FAKE_FUNCTION + return nullptr; +} + +template +void CallAndCollect(Result (*function)(Args...), + std::vector& statuses) { + if constexpr (std::is_same_v) { + statuses.push_back(function(Args{}...)); + } else { + function(Args{}...); + } +} +} // namespace + +namespace fake_corelib { + +State& GetState() { return state; } + +void Reset() { + std::lock_guard lock(state_mutex); + state.version = {0, 3, 0}; + state.selftest_status = ryzenai_corelib_status_success; + state.default_status = ryzenai_corelib_status_success; + state.has_device_context = true; + state.detail.clear(); + state.status_text = "success"; + state.missing_symbol.clear(); + state.resolution_order.clear(); + state.resolution_counts.clear(); + state.call_counts.clear(); + state.statuses.clear(); + state.lifetime_events.clear(); + state.live_objects = 0; + state.releases = 0; + state.cleanup_calls = 0; + state.active_leases = 0; + state.maximum_active_leases = 0; + state.matmul_pad_calls.clear(); + state.rows_pad_calls.clear(); + state.mha_pad_calls.clear(); + state.pad_multiple = 64; + state.matmul_k_delta = 0; + state.matmul_n_delta = 0; + state.pad_row_overrides.clear(); + state.tensor_creates.clear(); + state.tensor_windows.clear(); + state.weight_creates.clear(); + state.dispatches.clear(); + state.tensor_writes.clear(); + state.call_log.clear(); + state.active_weight_creates = 0; + state.maximum_active_weight_creates = 0; + state.work_in_flight = false; + state.fail_after_submit.clear(); +} + +flm::corelib::CorelibApi::Resolver Resolver() { + return [](std::string_view name) -> void* { + std::lock_guard lock(state_mutex); + state.resolution_order.emplace_back(name); + ++state.resolution_counts[std::string(name)]; + if (name == state.missing_symbol) return nullptr; + return FunctionFor(name); + }; +} + +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions) { + std::vector statuses; +#define FLM_CALL_FAKE_FUNCTION(member, symbol) CallAndCollect(functions.member, statuses); + FLM_CORELIB_FUNCTIONS(FLM_CALL_FAKE_FUNCTION) +#undef FLM_CALL_FAKE_FUNCTION + return statuses; +} + +void* MakeObject() { + std::lock_guard lock(state_mutex); + return NewObject(); +} + +void EnterLease() { + std::lock_guard lock(state_mutex); + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} +} + +void LeaveLease() { + std::lock_guard lock(state_mutex); + --state.active_leases; + state.lifetime_events.emplace_back("lease_leave"); +} + +} // namespace fake_corelib diff --git a/src/test/phi4_corelib_aie4/fake_corelib.hpp b/src/test/phi4_corelib_aie4/fake_corelib.hpp new file mode 100644 index 00000000..c46e2529 --- /dev/null +++ b/src/test/phi4_corelib_aie4/fake_corelib.hpp @@ -0,0 +1,127 @@ +#pragma once + +#include "corelib/corelib_api.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fake_corelib { + +struct MatmulPadCall { + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct RowsPadCall { + std::string helper; + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct MhaPadCall { + std::int64_t m; + ryzenai_corelib_flat_mha_bf16_desc desc; +}; + +struct TensorCreateRecord { + ryzenai_corelib_data_type data_type; + std::vector shape; + void* object; +}; + +struct TensorWindowRecord { + void* parent; + std::vector shape; + std::size_t offset; + void* object; +}; + +struct WeightCreateRecord { + std::string kind; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; + std::uint32_t threads; + std::vector pointers; + std::vector norm0; + std::vector norm1; + std::uint16_t epsilon{}; +}; + +struct DispatchRecord { + std::thread::id thread_id; + std::string kind; + void* stream; + void* input; + void* output; + std::int64_t rows; + std::int64_t position; + std::size_t window_offset; +}; + +struct TensorWriteRecord { + void* tensor; + ryzenai_corelib_data_type source_type; + std::size_t count; + std::size_t offset; + bool all_zero; +}; + +struct State { + flm::corelib::CorelibVersion version{0, 3, 0}; + ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; + ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; + bool has_device_context{true}; + std::string detail; + std::string status_text{"success"}; + std::string missing_symbol; + std::vector resolution_order; + std::unordered_map resolution_counts; + std::unordered_map call_counts; + std::unordered_map statuses; + std::vector lifetime_events; + std::atomic live_objects{0}; + std::atomic releases{0}; + std::atomic cleanup_calls{0}; + std::atomic active_leases{0}; + std::atomic maximum_active_leases{0}; + std::vector matmul_pad_calls; + std::vector rows_pad_calls; + std::vector mha_pad_calls; + std::int64_t pad_multiple{64}; + std::int64_t matmul_k_delta{0}; + std::int64_t matmul_n_delta{0}; + std::unordered_map> pad_row_overrides; + std::vector tensor_creates; + std::vector tensor_windows; + std::vector weight_creates; + std::vector dispatches; + std::vector tensor_writes; + std::vector call_log; + std::atomic active_weight_creates{0}; + std::atomic maximum_active_weight_creates{0}; + bool work_in_flight{false}; + std::string fail_after_submit; +}; + +State& GetState(); +void Reset(); +flm::corelib::CorelibApi::Resolver Resolver(); +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions); +void* MakeObject(); +void EnterLease(); +void LeaveLease(); + +} // namespace fake_corelib diff --git a/src/test/phi4_corelib_aie4/gguf_fixture.hpp b/src/test/phi4_corelib_aie4/gguf_fixture.hpp new file mode 100644 index 00000000..ac70bed1 --- /dev/null +++ b/src/test/phi4_corelib_aie4/gguf_fixture.hpp @@ -0,0 +1,415 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace gguf_fixture { + +inline constexpr std::uint32_t kF32 = 0; +inline constexpr std::uint32_t kQ8_0 = 8; + +enum class Mutation { + None, + TruncatedString, + TruncatedDirectory, + CountOverflow, + ProductOverflow, + OffsetOverflow, + ZeroAlignment, + NonPowerOfTwoAlignment, + DuplicateName, + OutOfFileRange, + OverlappingRanges, + UnsupportedMetadataType, + DtypeMismatch, + ShapeMismatch, + PayloadLengthMismatch, + MisalignedF32, +}; + +struct ArrayValue { + std::uint32_t element_type; + std::uint64_t count; + std::vector encoded_elements; +}; +using MetadataValue = std::variant; + +struct Tensor { + std::string name; + std::vector logical_shape; + std::uint32_t type; + std::uint64_t offset = 0; + bool explicit_offset = false; +}; + +struct TempFile { + std::filesystem::path path; + TempFile() = default; + explicit TempFile(std::filesystem::path value) : path(std::move(value)) {} + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; + TempFile(TempFile&& other) noexcept : path(std::move(other.path)) { + other.path.clear(); + } + TempFile& operator=(TempFile&& other) noexcept { + if (this != &other) { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + path = std::move(other.path); + other.path.clear(); + } + return *this; + } + ~TempFile() { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + } +}; + +template +void Append(std::vector& out, T value) { + static_assert(std::is_trivially_copyable_v); + const auto bytes = std::bit_cast>(value); + out.insert(out.end(), bytes.begin(), bytes.end()); +} + +inline void AppendString(std::vector& out, const std::string& value) { + Append(out, static_cast(value.size())); + for (const char c : value) out.push_back(static_cast(c)); +} + +inline std::uint64_t TensorBytes(const Tensor& tensor) { + std::uint64_t elements = 1; + for (const auto dimension : tensor.logical_shape) { + if (dimension != 0 && elements > std::numeric_limits::max() / dimension) + throw std::overflow_error("fixture tensor product"); + elements *= dimension; + } + if (tensor.type == kF32) return elements * 4; + if (tensor.type == kQ8_0) { + if (elements % 32 != 0) throw std::runtime_error("fixture Q8_0 divisibility"); + return elements / 32 * 34; + } + return elements; +} + +class Builder { +public: + Builder() { AddContractMetadata(); } + + Builder& Alignment(std::uint32_t alignment) { + alignment_ = alignment; + SetMetadata("general.alignment", alignment); + return *this; + } + + Builder& AddMetadata(std::string key, MetadataValue value) { + metadata_.emplace_back(std::move(key), std::move(value)); + return *this; + } + + Builder& SetMetadata(std::string key, MetadataValue value) { + for (auto& entry : metadata_) { + if (entry.first == key) { + entry.second = std::move(value); + return *this; + } + } + return AddMetadata(std::move(key), std::move(value)); + } + + Builder& RemoveMetadata(const std::string& key) { + std::erase_if(metadata_, [&](const auto& entry) { return entry.first == key; }); + return *this; + } + + Builder& AddTensor(std::string name, std::vector logical_shape, + std::uint32_t type) { + tensors_.push_back({std::move(name), std::move(logical_shape), type}); + return *this; + } + + Builder& AddExactFixtureTensors() { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_norm.weight", {3072}, kF32); + AddTensor("blk.0.ffn_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor("blk.0.attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor("blk.0.ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor("blk.0.ffn_down.weight", {3072, 8192}, kQ8_0); + AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& AddFullContractTensors(bool short_rope = true) { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + AddTensor(prefix + ".attn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".ffn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor(prefix + ".attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_down.weight", {3072, 8192}, kQ8_0); + } + if (short_rope) AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& MutateTensor(const std::string& name, std::uint32_t type, + std::vector shape) { + auto* tensor = FindTensor(name); + if (!tensor) throw std::runtime_error("fixture tensor not found: " + name); + tensor->type = type; + tensor->logical_shape = std::move(shape); + return *this; + } + + Builder& RemoveTensor(const std::string& name) { + std::erase_if(tensors_, [&](const Tensor& tensor) { return tensor.name == name; }); + return *this; + } + + Builder& TruncateTensorPayload(std::string name) { + truncated_tensor_ = std::move(name); + return *this; + } + + Builder& AddEverySkippableMetadataType() { + AddMetadata("skip.u8", std::uint8_t{1}); + AddMetadata("skip.i8", std::int8_t{-1}); + AddMetadata("skip.u16", std::uint16_t{2}); + AddMetadata("skip.i16", std::int16_t{-2}); + AddMetadata("skip.u32", std::uint32_t{3}); + AddMetadata("skip.i32", std::int32_t{-3}); + AddMetadata("skip.f32", 1.25f); + AddMetadata("skip.bool", true); + AddMetadata("skip.string", std::string("value")); + std::vector strings; + AppendString(strings, "a"); AppendString(strings, "bc"); + AddMetadata("skip.array", ArrayValue{8, 2, std::move(strings)}); + AddMetadata("skip.u64", std::uint64_t{4}); + AddMetadata("skip.i64", std::int64_t{-4}); + AddMetadata("skip.f64", 2.5); + return *this; + } + + Builder& Apply(Mutation mutation) { mutation_ = mutation; return *this; } + + TempFile Write(std::string_view label = "fixture") const { + static std::uint64_t serial = 0; + auto path = std::filesystem::temp_directory_path() / + ("flm_phi4_" + std::string(label) + "_" + + std::to_string(++serial) + ".gguf"); + auto bytes = Encode(); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + if (!stream) throw std::runtime_error("cannot create fixture"); + stream.write(reinterpret_cast(bytes.prefix.data()), + static_cast(bytes.prefix.size())); + if (bytes.file_size > bytes.prefix.size()) { + stream.seekp(static_cast(bytes.file_size - 1)); + const char zero = 0; + stream.write(&zero, 1); + } + stream.close(); + return TempFile(path); + } + +private: + struct Encoded { std::vector prefix; std::uint64_t file_size; }; + + void AddContractMetadata() { + AddMetadata("general.architecture", std::string("phi3")); + AddMetadata("general.alignment", std::uint32_t{32}); + AddMetadata("phi3.block_count", std::uint32_t{32}); + AddMetadata("phi3.context_length", std::uint32_t{131072}); + AddMetadata("phi3.embedding_length", std::uint32_t{3072}); + AddMetadata("phi3.feed_forward_length", std::uint32_t{8192}); + AddMetadata("phi3.attention.head_count", std::uint32_t{24}); + AddMetadata("phi3.attention.head_count_kv", std::uint32_t{8}); + AddMetadata("phi3.attention.layer_norm_rms_epsilon", 1.0e-5f); + AddMetadata("phi3.rope.dimension_count", std::uint32_t{96}); + AddMetadata("phi3.rope.freq_base", 10000.0f); + AddMetadata("phi3.rope.scaling.attn_factor", 1.0f); + AddMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{4096}); + AddMetadata("tokenizer.ggml.tokens", + ArrayValue{0, 200064, std::vector(200064)}); + AddMetadata("tokenizer.ggml.add_bos_token", false); + AddMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{200020}); + } + + Tensor* FindTensor(const std::string& name) { + const auto it = std::find_if(tensors_.begin(), tensors_.end(), + [&](const Tensor& tensor) { return tensor.name == name; }); + return it == tensors_.end() ? nullptr : &*it; + } + + static std::uint32_t TypeOf(const MetadataValue& value) { + return static_cast(value.index()); + } + + static void EncodeValue(std::vector& out, const MetadataValue& value) { + std::visit([&](const auto& item) { + using T = std::decay_t; + if constexpr (std::is_same_v) AppendString(out, item); + else if constexpr (std::is_same_v) { + Append(out, item.element_type); Append(out, item.count); + out.insert(out.end(), item.encoded_elements.begin(), item.encoded_elements.end()); + } else if constexpr (std::is_same_v) Append(out, std::uint8_t(item)); + else Append(out, item); + }, value); + } + + Encoded Encode() const { + auto metadata = metadata_; + auto tensors = tensors_; + std::uint32_t alignment = alignment_; + if (mutation_ == Mutation::ZeroAlignment || + mutation_ == Mutation::NonPowerOfTwoAlignment) { + alignment = mutation_ == Mutation::ZeroAlignment ? 0 : 24; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = alignment; + } + if (mutation_ == Mutation::DuplicateName && !tensors.empty()) tensors.push_back(tensors.front()); + if (mutation_ == Mutation::DtypeMismatch && !tensors.empty()) tensors.front().type = kF32; + if (mutation_ == Mutation::ShapeMismatch && !tensors.empty()) tensors.front().logical_shape[0]--; + if (!truncated_tensor_.empty()) { + const auto it = std::find_if(tensors.begin(), tensors.end(), [&](const Tensor& tensor) { + return tensor.name == truncated_tensor_; + }); + if (it == tensors.end()) throw std::runtime_error("fixture tensor not found: " + truncated_tensor_); + Tensor target = std::move(*it); + tensors.erase(it); + tensors.push_back(std::move(target)); + } + if (mutation_ == Mutation::MisalignedF32) { + alignment = 1; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = std::uint32_t{1}; + } + + std::vector out; + Append(out, std::uint32_t{0x46554747}); Append(out, std::uint32_t{3}); + Append(out, mutation_ == Mutation::CountOverflow ? std::numeric_limits::max() + : static_cast(tensors.size())); + Append(out, static_cast(metadata.size())); + for (const auto& [key, value] : metadata) { + AppendString(out, key); + if (mutation_ == Mutation::UnsupportedMetadataType && key == metadata.front().first) { + Append(out, std::uint32_t{99}); + } else { + Append(out, TypeOf(value)); EncodeValue(out, value); + } + } + std::uint64_t running = 0; + std::vector encoded_offset_positions; + for (std::size_t index = 0; index < tensors.size(); ++index) { + auto& tensor = tensors[index]; + if (mutation_ == Mutation::ProductOverflow && index == 0) + tensor.logical_shape = { + static_cast(std::numeric_limits::max()), 3}; + if (tensor.explicit_offset) running = tensor.offset; + if (alignment != 0 && (alignment & (alignment - 1)) == 0) + running = (running + alignment - 1) & ~(std::uint64_t(alignment) - 1); + tensor.offset = running; + const auto size = mutation_ == Mutation::ProductOverflow && index == 0 + ? 0 : TensorBytes(tensor); + if (mutation_ == Mutation::OverlappingRanges && index == 1) { + tensor.offset = 0; + running += size; + } else if (mutation_ == Mutation::OutOfFileRange && index == 0) + tensor.offset = std::uint64_t{1} << 40; + else if (mutation_ == Mutation::OffsetOverflow && index == 0) + tensor.offset = std::numeric_limits::max() - 31; + else running += size; + AppendString(out, tensor.name); + Append(out, static_cast(tensor.logical_shape.size())); + for (auto it = tensor.logical_shape.rbegin(); it != tensor.logical_shape.rend(); ++it) + Append(out, *it); + Append(out, tensor.type); + encoded_offset_positions.push_back(out.size()); + Append(out, tensor.offset); + } + if (mutation_ == Mutation::TruncatedDirectory && !out.empty()) { + out.pop_back(); return {std::move(out), static_cast(out.size())}; + } + const auto data_start = alignment == 0 ? static_cast(out.size()) + : (static_cast(out.size()) + alignment - 1) & ~(std::uint64_t(alignment) - 1); + if (mutation_ == Mutation::MisalignedF32 && !tensors.empty()) { + const std::uint64_t offset = (1 + alignof(float) - data_start % alignof(float)) % alignof(float); + const auto encoded = std::bit_cast>(offset); + std::copy(encoded.begin(), encoded.end(), out.begin() + encoded_offset_positions.front()); + running = std::max(running, offset + TensorBytes(tensors.front())); + } + out.resize(static_cast(data_start), std::byte{0}); + std::uint64_t file_size = data_start + running; + if ((mutation_ == Mutation::PayloadLengthMismatch || !truncated_tensor_.empty()) && + file_size > data_start) --file_size; + if (mutation_ == Mutation::TruncatedString) { + const auto impossible = std::bit_cast>( + std::numeric_limits::max()); + std::copy(impossible.begin(), impossible.end(), out.begin() + 24); + } + return {std::move(out), file_size}; + } + + std::uint32_t alignment_ = 32; + std::vector> metadata_; + std::vector tensors_; + Mutation mutation_ = Mutation::None; + std::string truncated_tensor_; +}; + +inline nlohmann::json ValidConfig() { + return {{"model_type", "phi3"}, {"num_hidden_layers", 32}, + {"hidden_size", 3072}, {"intermediate_size", 8192}, + {"num_attention_heads", 24}, {"num_key_value_heads", 8}, + {"head_dim", 128}, {"vocab_size", 200064}, + {"rms_norm_eps", 1.0e-5}, {"original_max_position_embeddings", 4096}, + {"eos_token_id", 199999}}; +} + +inline nlohmann::json ValidTokenizer() { + nlohmann::json vocab = nlohmann::json::object(); + for (int id = 0; id < 200019; ++id) vocab["t" + std::to_string(id)] = id; + vocab["<|endoftext|>"] = 199999; + vocab["<|end|>"] = 200020; + return {{"model", {{"vocab", std::move(vocab)}}}, + {"added_tokens", nlohmann::json::array({ + {{"id", 200019}, {"content", "<|assistant|>"}}, + {{"id", 200020}, {"content", "<|end|>"}}, + {{"id", 200021}, {"content", "<|user|>"}}, + {{"id", 200022}, {"content", "<|system|>"}}, + {{"id", 200023}, {"content", "<|tool|>"}}, + {{"id", 200024}, {"content", "<|/tool|>"}}, + {{"id", 200025}, {"content", "<|tool_call|>"}}, + {{"id", 200026}, {"content", "<|/tool_call|>"}}, + {{"id", 200027}, {"content", "<|tool_response|>"}}, + {{"id", 200028}, {"content", "<|tag|>"}}, + {{"id", 200018}, {"content", "<|endofprompt|>"}}, + {{"id", 199999}, {"content", "<|endoftext|>"}}})}}; +} + +inline nlohmann::json ValidTokenizerConfig() { + return {{"add_bos_token", false}, + {"chat_template", "<|user|>{{ message }}<|end|><|assistant|>"}}; +} + +} // namespace gguf_fixture diff --git a/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 new file mode 100644 index 00000000..5cc4c4da --- /dev/null +++ b/src/test/phi4_corelib_aie4/run_real_aie4_acceptance.ps1 @@ -0,0 +1,217 @@ +param( + [string]$FlmExe = 'src/build-aie4/Release/flm.exe', + [string]$Model = 'phi4-mini-it-aie4:4b', + [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', + [string]$Output = 'src/build-aie4/phi4-gguf-aie4-acceptance.json', + [int]$Port = 52625, + [string]$Python = 'python', + # Diagnostics only: skips the 16-minute CLI matrix so the REST phase can be + # iterated on quickly. A record produced this way can never report success. + [switch]$SkipCli +) +$ErrorActionPreference='Stop' +$root=(Resolve-Path (Join-Path $PSScriptRoot '../../..')).Path +$exe=(Resolve-Path (Join-Path $root $FlmExe)).Path +$core=(Resolve-Path $CorelibDll).Path +$outPath=[IO.Path]::GetFullPath((Join-Path $root $Output)) +$outDir=Split-Path $outPath +$modelDir=Join-Path $env:USERPROFILE '.flm/models/phi4-mini-it-aie4' +$env:FLM_AIE4_CORELIB_PATH=$core +$env:FLM_CONFIG_PATH=Join-Path $root 'src/model_list.json' +$env:FLM_XCLBIN_PATH=Join-Path $root 'src' +$runtime=Split-Path $core +$env:PATH="$(Join-Path $root 'src/lib/xrt');$(Join-Path $root 'src/lib');$runtime;C:/Users/chiz/.conda/envs/hybrid-llm/Library/bin;C:/Users/chiz/work/hybrid-llm/install/xrt_package/xrt;$env:PATH" +New-Item -ItemType Directory -Force $outDir | Out-Null +$record=[ordered]@{started=(Get-Date).ToString('o');passed=$false;commands=@();host=[ordered]@{};provenance=[ordered]@{};files=@();cli=[ordered]@{};rest=[ordered]@{};performance=[ordered]@{};failures=@()} +# Progress markers go to stdout so a run that stalls can be located from the +# transcript alone; a silent 30-minute stall is indistinguishable from work. +function Mark([string]$m){Write-Host ("[mark] "+(Get-Date).ToString('HH:mm:ss.fff')+" "+$m)} +# ConvertTo-Json cannot be used on the record as a whole. Some of the values it +# holds are live .NET objects whose property graphs loop back on themselves, and +# ConvertTo-Json expands such a graph until -Depth runs out, which allocates tens +# of gigabytes and never returns. This was confirmed on both Windows PowerShell +# 5.1 and PowerShell 7.0.0 (3.4 GB and still climbing when killed) β€” moving to a +# newer engine does not avoid it, so do not remove this. This emitter walks +# the record itself: it refuses to descend past $script:JsonMaxDepth, and it +# refuses to re-enter an object that is already an ancestor of the current node. +# ConvertTo-Json is still used, but only ever on a single scalar string. +$script:JsonMaxDepth=10 +function JsonScalar($s){return (ConvertTo-Json -InputObject ([string]$s))} +function EmitJson($v,[string]$label,[int]$level,$ancestors){ + if($null -eq $v){return 'null'} + if($v -is [string]){return (JsonScalar $v)} + if($v -is [bool]){if($v){return 'true'}else{return 'false'}} + if($v -is [datetime]){return (JsonScalar $v.ToString('o'))} + if($v -is [double] -or $v -is [single]){if([double]::IsNaN($v)-or[double]::IsInfinity($v)){return 'null'};return (([double]$v).ToString('R',[Globalization.CultureInfo]::InvariantCulture))} + if($v -is [ValueType] -and $v -isnot [char] -and $v -isnot [Enum]){return (([string]$v))} + if($level -ge $script:JsonMaxDepth){return (JsonScalar $v)} + # The ancestor test exists for live .NET objects, whose property graphs loop. + # It deliberately does not apply to a PSCustomObject: ConvertFrom-Json only + # ever builds trees, and every object it produces shares one singleton base + # instance, so testing those would report every nested JSON object as a loop. + $bo=$null;try{$bo=$v.PSObject.BaseObject}catch{} + $next=$ancestors + if($null -ne $bo -and $bo -isnot [System.Management.Automation.PSCustomObject]){ + foreach($a in $ancestors){if([object]::ReferenceEquals($a,$bo)){return (JsonScalar '')}} + # The ancestor list must be built with Add, not with "+". Adding an array + # with "+" splices its elements in, which would put every element of an + # array on the ancestor list and make each of them look like a loop. + $next=New-Object Collections.ArrayList + if($null -ne $ancestors){[void]$next.AddRange($ancestors)} + [void]$next.Add($bo) + } + $parts=New-Object Collections.ArrayList + if($v -is [System.Collections.IDictionary]){ + foreach($k in @($v.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + [void]$parts.Add((JsonScalar $k)+':'+(EmitJson $v[$k] "$label.$k" ($level+1) $next)) + if($level -lt 2){Mark ("json {0}.{1} in {2:N1}s" -f $label,$k,$sw.Elapsed.TotalSeconds)} + } + return '{'+($parts -join ',')+'}' + } + if($v -is [System.Collections.IEnumerable]){ + foreach($e in $v){[void]$parts.Add((EmitJson $e "$label[]" ($level+1) $next))} + return '['+($parts -join ',')+']' + } + $props=@($v.PSObject.Properties) + if($props.Count -gt 0){ + foreach($p in $props){ + $pv=$null;try{$pv=$p.Value}catch{$pv=""} + [void]$parts.Add((JsonScalar $p.Name)+':'+(EmitJson $pv "$label.$($p.Name)" ($level+1) $next)) + } + return '{'+($parts -join ',')+'}' + } + return (JsonScalar $v) +} +function WriteRecord($rec,[string]$path){ + [IO.File]::WriteAllText($path,"{`r`n",[Text.Encoding]::UTF8) + $first=$true + foreach($k in @($rec.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + try{$t=EmitJson $rec[$k] $k 1 (New-Object Collections.ArrayList)}catch{$t=JsonScalar ("")} + if(-not$first){[IO.File]::AppendAllText($path,",`r`n",[Text.Encoding]::UTF8)} + $first=$false + [IO.File]::AppendAllText($path,(' "{0}": {1}' -f $k,$t),[Text.Encoding]::UTF8) + Mark ("json section {0} written in {1:N1}s" -f $k,$sw.Elapsed.TotalSeconds) + } + [IO.File]::AppendAllText($path,"`r`n}`r`n",[Text.Encoding]::UTF8) +} +# Piping an ErrorRecord to Out-String yields nothing but a newline under some +# host configurations, which would record a failure with no reason attached. +function ErrText($e){ + $parts=@("$($e.Exception.GetType().FullName): $($e.Exception.Message)") + $rendered=($e|Out-String);if(-not [string]::IsNullOrWhiteSpace($rendered)){$parts+=$rendered.Trim()} + if($e.InvocationInfo -and $e.InvocationInfo.PositionMessage){$parts+=$e.InvocationInfo.PositionMessage.Trim()} + if($e.ScriptStackTrace){$parts+=$e.ScriptStackTrace.Trim()} + return ($parts -join "`n") +} +function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$t=ErrText $_;$record.failures+=$t;Mark ("FAILURE in ${line}: "+$t);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} +# A non-2xx reply is an error in both engines, but the two expose it +# differently: Windows PowerShell hands back a WebResponse to read a stream +# from, PowerShell 7 hands back an HttpResponseMessage and puts the body in +# ErrorDetails. An expected 400 must not depend on which engine is running. +function Post([string]$path,$body,[int]$TimeoutSec=900){ + try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -TimeoutSec $TimeoutSec -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}} + catch{ + $resp=$_.Exception.Response + if($null -eq $resp){throw} + $text=$null + if($_.ErrorDetails -and $_.ErrorDetails.Message){$text=$_.ErrorDetails.Message} + elseif($resp.PSObject.Methods['GetResponseStream']){$text=(New-Object IO.StreamReader($resp.GetResponseStream())).ReadToEnd()} + elseif($resp.Content){$text=$resp.Content.ReadAsStringAsync().GetAwaiter().GetResult()} + $json=$null;if(-not [string]::IsNullOrWhiteSpace($text)){try{$json=$text|ConvertFrom-Json}catch{}} + return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=$json} + } +} +# Two things must never reach curl as inline arguments. A JSON body loses its +# double quotes to PowerShell's native-argument quoting and the server sees a +# malformed object, so every body goes to a file and is read back with "@file". +# A header value containing a space is split into two arguments, so the second +# half is taken as another URL ("Could not resolve host: application"); the +# colon form without a space carries the same meaning and cannot split. +$ContentTypeArg='Content-Type:application/json' +function BodyFile([string]$name,$body){$p=Join-Path $outDir $name;Set-Content -Path $p -Value ($body|ConvertTo-Json -Depth 8 -Compress) -Encoding ASCII -NoNewline;return $p} +function CurlStream([string]$name,[string]$path,$body){$f=BodyFile $name $body;$out=(&curl.exe -sS -N -H $ContentTypeArg -d "@$f" "http://127.0.0.1:$Port$path" 2>&1|Out-String);if($LASTEXITCODE-ne 0){throw "curl failed ($LASTEXITCODE) for ${path}: $out"};if($out-match 'Could not resolve host'){throw "curl argument splitting for ${path}: $out"};return $out} +function CurlBackground([string]$name,[string]$path,$body,[string]$outFile){$f=BodyFile $name $body;return (Start-Process curl.exe -ArgumentList @('-sS','-N','-H',$ContentTypeArg,'-d',"@$f","http://127.0.0.1:$Port$path") -RedirectStandardOutput $outFile -PassThru)} +try{ + $record.host.computer=$env:COMPUTERNAME;$record.host.cpu=(Get-CimInstance Win32_Processor).Name;$record.host.npu=(Get-CimInstance Win32_PnPEntity|Where-Object Name -match 'NPU|Neural').Name;$os=Get-CimInstance Win32_OperatingSystem;$record.host.windows="$($os.Caption) $($os.Version) build $($os.BuildNumber)";$record.host.power=(powercfg /getactivescheme|Out-String).Trim() + $record.provenance.fastflow=(git -C $root rev-parse HEAD).Trim();$coreRoot=(Resolve-Path (Join-Path $runtime '..')).Path;$record.provenance.corelib=(git -C $coreRoot rev-parse HEAD).Trim();$record.provenance.corelib_abi='0.3.0';$record.provenance.gguf_revision='78eb92a46fc37e6b524df991ed9aca9bc6aa7b80';$record.provenance.tokenizer_revision='cfbefacb99257ffa30c83adab238a50856ac3083';$record.provenance.corelib_sha256=(Get-FileHash $core -Algorithm SHA256).Hash.ToLower() + Cmd "$exe check $Model" {&$exe check $Model|Out-Host;if($LASTEXITCODE-ne 0){throw 'check failed'}} + $names=@('Phi-4-mini-instruct.Q8_0.gguf','tokenizer.json','tokenizer_config.json','config.json');$actual=@(Get-ChildItem $modelDir -File|% Name);if((Compare-Object ($names|Sort-Object) ($actual|Sort-Object))){throw 'model directory is not exactly four files'};foreach($n in $names){$f=Get-Item (Join-Path $modelDir $n);$record.files+=@([ordered]@{name=$n;bytes=$f.Length;sha256=(Get-FileHash $f.FullName -Algorithm SHA256).Hash.ToLower()})} + $py=@' +from winpty import PtyProcess +import os,sys,time,threading,json,re +exe,model,out=sys.argv[1:4] +def run(cmds,timeout=900): + p=PtyProcess.spawn(f'{exe} run {model}',env=os.environ.copy(),dimensions=(50,200));chunks=[] + def rd(): + while p.isalive(): + try: chunks.append(p.read(8192)) + except: break + threading.Thread(target=rd,daemon=True).start(); end=time.time()+timeout + while time.time() +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibError; +using flm::corelib::CorelibRuntime; +using flm::corelib::UniqueMatMulWeights; +using flm::corelib::UniqueSsMlpWeights; +using flm::corelib::UniqueStream; +using flm::corelib::UniqueTensor; +using flm::corelib::UniqueTensorWindow; + +void SetCorelibPath(const char* value) { +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", value ? value : ""); +#else + if (value) setenv("FLM_AIE4_CORELIB_PATH", value, 1); + else unsetenv("FLM_AIE4_CORELIB_PATH"); +#endif +} + +std::shared_ptr ValidApi() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +void TestVersionIsResolvedBeforeEveryOtherSymbol() { + fake_corelib::Reset(); + ValidApi(); + const auto& order = fake_corelib::GetState().resolution_order; + TEST_REQUIRE(order.size() == 23); + TEST_REQUIRE(order.front() == "ryzenai_corelib_get_version"); +} + +void TestExactlyVersion030IsAccepted() { + fake_corelib::Reset(); + const auto api = ValidApi(); + const auto version = api->runtime_version(); + TEST_REQUIRE(version.major == 0); + TEST_REQUIRE(version.minor == 3); + TEST_REQUIRE(version.patch == 0); +} + +void TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions() { + for (const auto version : {flm::corelib::CorelibVersion{1, 3, 0}, + flm::corelib::CorelibVersion{0, 4, 0}, + flm::corelib::CorelibVersion{0, 3, 1}}) { + fake_corelib::Reset(); + fake_corelib::GetState().version = version; + const std::string error = RequireThrows([&] { ValidApi(); }); + RequireContains(error, "0.3.0"); + RequireContains(error, std::to_string(version.major) + "." + + std::to_string(version.minor) + "." + + std::to_string(version.patch)); + TEST_REQUIRE(fake_corelib::GetState().resolution_order.size() == 1); + } +} + +void TestEveryRequiredSymbolIsResolvedExactlyOnce() { + fake_corelib::Reset(); + ValidApi(); + TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 23); + for (const auto& [name, count] : fake_corelib::GetState().resolution_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } +} + +void TestEveryResolvedFakeFunctionUsesItsExactAbi() { + fake_corelib::Reset(); + const auto api = ValidApi(); + fake_corelib::GetState().call_counts.clear(); + fake_corelib::GetState().default_status = ryzenai_corelib_status_bad_argument; + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_bad_argument; + const auto statuses = fake_corelib::CallEveryResolvedFunction(api->functions()); + TEST_REQUIRE(statuses.size() == 17); + TEST_REQUIRE(std::all_of(statuses.begin(), statuses.end(), [](auto status) { + return status == ryzenai_corelib_status_bad_argument; + })); + TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 23); + for (const auto& [name, count] : fake_corelib::GetState().call_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } + + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = + ryzenai_corelib_status_unsupported; + TEST_REQUIRE(api->functions().tensor_write( + nullptr, ryzenai_corelib_data_type_bf16, nullptr, 0, 0) == + ryzenai_corelib_status_unsupported); + TEST_REQUIRE(fake_corelib::GetState() + .call_counts["ryzenai_corelib_tensor_write"] == 2); +} + +void TestStandaloneRmsNormSymbolsAreNotRequired() { + for (const auto* symbol : { + "ryzenai_corelib_rmsnorm_bf16_weights_create_scale", + "ryzenai_corelib_rmsnorm_bf16_pad_rows", + "ryzenai_corelib_rmsnorm_bf16"}) { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = symbol; + (void)ValidApi(); + TEST_REQUIRE(!fake_corelib::GetState().resolution_counts.contains(symbol)); + } +} + +void TestMissingSymbolNamesTheSymbolAndUnloadsTheDll() { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = "ryzenai_corelib_create_stream"; + std::weak_ptr unloaded; + std::string error; + { + auto module_lifetime = std::make_shared(1); + unloaded = module_lifetime; + auto base = fake_corelib::Resolver(); + CorelibApi::Resolver resolver = + [module_lifetime, base](std::string_view name) { return base(name); }; + module_lifetime.reset(); + error = RequireThrows([&] { CorelibApi::ResolveForTest(std::move(resolver)); }); + } + RequireContains(error, "ryzenai_corelib_create_stream"); + TEST_REQUIRE(unloaded.expired()); +} + +void TestCorelibErrorCopiesStatusCallAndThreadLocalDetail() { + fake_corelib::Reset(); + fake_corelib::GetState().detail = "invalid tensor row count"; + fake_corelib::GetState().status_text = "bad argument"; + const auto api = ValidApi(); + try { + api->Check(ryzenai_corelib_status_bad_argument, "tensor_write"); + TEST_REQUIRE(false); + } catch (const CorelibError& error) { + TEST_REQUIRE(error.status() == ryzenai_corelib_status_bad_argument); + TEST_REQUIRE(error.call() == "tensor_write"); + TEST_REQUIRE(error.detail() == "invalid tensor row count"); + RequireContains(error.what(), "bad argument"); + } +} + +void TestEnvironmentPathMustBeAnAbsoluteDllPath() { + SetCorelibPath("relative/ryzenai_corelib.dll"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + "absolute"); + SetCorelibPath("C:/apps/flm/aie4"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + ".dll"); + SetCorelibPath(nullptr); +} + +void TestEnvironmentPathWinsOverExecutableRelativePath() { + SetCorelibPath("C:/corelib/custom.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == + std::filesystem::path("C:/corelib/custom.dll")); + SetCorelibPath(nullptr); +} + +void TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory() { + SetCorelibPath(nullptr); + const auto expected = std::filesystem::absolute( + std::filesystem::path("C:/apps/flm") / "aie4" / "ryzenai_corelib.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == expected); +} + +void TestEveryUniqueObjectReleasesExactlyOnceAfterMoves() { + fake_corelib::Reset(); + const auto api = ValidApi(); + { + UniqueTensor first(api, fake_corelib::MakeObject()); + UniqueTensor moved(std::move(first)); + UniqueTensor assigned; + assigned = std::move(moved); + UniqueStream stream(api, fake_corelib::MakeObject()); + UniqueTensorWindow window(api, fake_corelib::MakeObject()); + UniqueMatMulWeights matmul(api, fake_corelib::MakeObject()); + UniqueSsMlpWeights ssmlp(api, fake_corelib::MakeObject()); + TEST_REQUIRE(!first && !moved && assigned); + TEST_REQUIRE(api->live_object_count() == 5); + TEST_REQUIRE(fake_corelib::GetState().releases == 0); + } + TEST_REQUIRE(fake_corelib::GetState().releases == 5); + TEST_REQUIRE(api->live_object_count() == 0); +} + +void TestRuntimeRunsDependencySelftestAndRequiresDeviceContext() { + fake_corelib::Reset(); + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "selftest_dependencies"); + + fake_corelib::Reset(); + fake_corelib::GetState().has_device_context = false; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "device context"); + + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + TEST_REQUIRE(runtime->api() != nullptr); + CorelibRuntime::ShutdownProcess(); +} + +void TestExecutionLeaseSerializesTwoThreads() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + std::atomic ready{0}; + auto worker = [&] { + ++ready; + while (ready.load() != 2) std::this_thread::yield(); + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }; + std::thread first(worker); + std::thread second(worker); + first.join(); + second.join(); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + CorelibRuntime::ShutdownProcess(); +} + +void TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner() { + fake_corelib::Reset(); + bool destroyed = false; + bool destroyed_while_locked = false; + CorelibRuntime::SetDestructionObserverForTest([&](bool execution_lock_held) { + destroyed = true; + destroyed_while_locked = execution_lock_held; + }); + auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + CorelibRuntime::SetDestructionObserverForTest({}); + TEST_REQUIRE(destroyed); + TEST_REQUIRE(!destroyed_while_locked); +} + +void TestCleanupRunsAfterTheLastObjectAndOnlyOnce() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + auto object = std::make_unique(runtime->api(), + fake_corelib::MakeObject()); + RequireContains(RequireThrows([] { CorelibRuntime::ShutdownProcess(); }), + "live corelib object"); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 0); + object.reset(); + + std::atomic lease_acquired{false}; + std::thread holder([&] { + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + lease_acquired = true; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }); + while (!lease_acquired.load()) std::this_thread::yield(); + CorelibRuntime::ShutdownProcess(); + holder.join(); + CorelibRuntime::ShutdownProcess(); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 1); + TEST_REQUIRE(fake_corelib::GetState().releases == 1); + TEST_REQUIRE(fake_corelib::GetState().lifetime_events == + std::vector({"release", "lease_leave", "cleanup"})); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestVersionIsResolvedBeforeEveryOtherSymbol); + RUN_TEST(TestExactlyVersion030IsAccepted); + RUN_TEST(TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions); + RUN_TEST(TestEveryRequiredSymbolIsResolvedExactlyOnce); + RUN_TEST(TestEveryResolvedFakeFunctionUsesItsExactAbi); + RUN_TEST(TestStandaloneRmsNormSymbolsAreNotRequired); + RUN_TEST(TestMissingSymbolNamesTheSymbolAndUnloadsTheDll); + RUN_TEST(TestCorelibErrorCopiesStatusCallAndThreadLocalDetail); + RUN_TEST(TestEnvironmentPathMustBeAnAbsoluteDllPath); + RUN_TEST(TestEnvironmentPathWinsOverExecutableRelativePath); + RUN_TEST(TestFallbackIsExeDirectoryAie4DllNotCurrentDirectory); + RUN_TEST(TestEveryUniqueObjectReleasesExactlyOnceAfterMoves); + RUN_TEST(TestRuntimeRunsDependencySelftestAndRequiresDeviceContext); + RUN_TEST(TestExecutionLeaseSerializesTwoThreads); + RUN_TEST(TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner); + RUN_TEST(TestCleanupRunsAfterTheLastObjectAndOnlyOnce); +#undef RUN_TEST + return 0; +} diff --git a/src/test/phi4_corelib_aie4/test_model_downloader.cpp b/src/test/phi4_corelib_aie4/test_model_downloader.cpp new file mode 100644 index 00000000..804bd8b7 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_model_downloader.cpp @@ -0,0 +1,389 @@ +#include "download_model.hpp" +#include "model_downloader.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include + +namespace { +namespace fs = std::filesystem; + +constexpr const char* kAie4Tag = "phi4-mini-it-aie4:4b"; +constexpr const char* kUnslothRevision = "78eb92a46fc37e6b524df991ed9aca9bc6aa7b80"; +constexpr const char* kMicrosoftRevision = "cfbefacb99257ffa30c83adab238a50856ac3083"; + +nlohmann::json ReadJson(const fs::path& path) { + std::ifstream stream(path); + TEST_REQUIRE(stream.is_open()); + return nlohmann::json::parse(stream); +} + +void Write(const fs::path& path, std::string_view bytes) { + fs::create_directories(path.parent_path()); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + stream.write(bytes.data(), static_cast(bytes.size())); + TEST_REQUIRE(stream.good()); +} + +std::string Read(const fs::path& path) { + std::ifstream stream(path, std::ios::binary); + return {std::istreambuf_iterator(stream), std::istreambuf_iterator()}; +} + +std::size_t CountOccurrences(std::string_view text, std::string_view needle) { + std::size_t count = 0; + for (std::size_t position = text.find(needle); position != std::string_view::npos; + position = text.find(needle, position + needle.size())) { + ++count; + } + return count; +} + +fs::path TempDirectory(std::string_view name) { + const auto path = fs::temp_directory_path() / ("flm-task5-" + std::string(name)); + std::error_code ignored; + fs::remove_all(path, ignored); + fs::create_directories(path); + return path; +} + +std::string FileUrl(const fs::path& path) { + std::string value = fs::absolute(path).generic_string(); +#ifdef _WIN32 + return "file:///" + value; +#else + return "file://" + value; +#endif +} + +void TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + const std::vector expected = { + "Phi-4-mini-instruct.Q8_0.gguf", "tokenizer.json", + "tokenizer_config.json", "config.json"}; + TEST_REQUIRE(model.at("name") == "phi4-mini-it-aie4"); + TEST_REQUIRE(model.at("files").get>() == expected); + TEST_REQUIRE(model.at("size").get() == 4100140571ULL); +} + +void TestGgufUrlContainsUnslothRevisionAndFilename() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + const auto source = resolve_file_source(model, "Phi-4-mini-instruct.Q8_0.gguf", false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/") + + kUnslothRevision + "/Phi-4-mini-instruct.Q8_0.gguf?download=true"); +} + +void TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it-aie4").at("4b"); + for (const std::string filename : {"tokenizer.json", "tokenizer_config.json", "config.json"}) { + const auto source = resolve_file_source(model, filename, false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/") + + kMicrosoftRevision + "/" + filename + "?download=true"); + } +} + +void TestExistingSingleSourceEntryKeepsItsCurrentUrl() { + const auto catalog = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto& model = catalog.at("models").at("phi4-mini-it").at("4b"); + const auto source = resolve_file_source(model, "config.json", false); + TEST_REQUIRE(source.url == + "https://huggingface.co/FastFlowLM/Phi4-mini-Instruct-NPU2/resolve/main/config.json?download=true"); +} + +void TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail() { + nlohmann::json model = { + {"url", "https://example.invalid/base"}, + {"files", {"config.json"}}, + {"file_sources", {{"unknown.json", {{"url", "https://example.invalid/source"}, + {"revision", std::string(40, 'a')}}}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), + "unknown file_sources key"); + + model["file_sources"] = {{"config.json", {{"revision", std::string(40, 'a')}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "url"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}, + {"revision", "NOT-A-COMMIT"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); +} + +void TestActualAie4CatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible() { + const auto root = TempDirectory("actual-catalog-version"); + const auto committed = ReadJson(FLM_SOURCE_DIR "/model_list.json"); + const auto model = committed.at("models").at("phi4-mini-it-aie4").at("4b"); + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"phi4-mini-it-aie4", {{"4b", model}}}}}}; + const auto catalog_path = root / "model_list.json"; + Write(catalog_path, catalog.dump()); + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / "phi4-mini-it-aie4"; + for (const auto& filename : model.at("files")) { + Write(model_path / filename.get(), "placeholder"); + } + Write(model_path / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded(kAie4Tag, true, true) == + ModelDownloader::ModelStatus::Ready); +} + +void TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile() { + const auto all_info = ReadJson(FLM_SOURCE_DIR "/model_info.json"); + const auto& records = all_info.at(kAie4Tag); + const std::vector> expected = { + {"Phi-4-mini-instruct.Q8_0.gguf", 4084611040ULL, "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42"}, + {"tokenizer.json", 15524095ULL, "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea"}, + {"tokenizer_config.json", 2932ULL, "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db"}, + {"config.json", 2504ULL, "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b"}}; + TEST_REQUIRE(records.size() == expected.size()); + std::uint64_t total = 0; + for (const auto& [path, size, sha256] : expected) { + const auto match = std::find_if(records.begin(), records.end(), [&](const auto& record) { + return record.at("path") == path; + }); + TEST_REQUIRE(match != records.end()); + TEST_REQUIRE(match->at("size").get() == size); + TEST_REQUIRE(match->at("sha256") == sha256); + total += size; + } + TEST_REQUIRE(total == 4100140571ULL); +} + +struct DownloaderFixture { + fs::path root = TempDirectory("ready"); + fs::path catalog_path = root / "model_list.json"; + fs::path info_path = root / "model_info.json"; + std::string catalog_string; + std::string root_string; + model_list models; + + DownloaderFixture() + : catalog_string(catalog_path.string()), root_string(root.string()), models() { + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"test-model", {{"1b", { + {"name", "test-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"details", {{"execution_backend", "corelib_aie4_gguf"}}}, + {"file_sources", nlohmann::json::object()}, + {"files", {"config.json", "a.bin", "b.bin", "c.bin"}} + }}}}}}}; + const nlohmann::json info = {{"test-model:1b", { + {{"path", "config.json"}, {"size", 57}, {"sha256", "b8bfba5e42c4cb0b8660ea39fec6fefafddc42fb6a0b17d472177fb7683b2290"}}, + {{"path", "a.bin"}, {"size", 5}, {"sha256", "8ed3f6ad685b959ead7022518e1af76cd816f8e8ec7ccdda1ed4018e8f2223f8"}}, + {{"path", "b.bin"}, {"size", 4}, {"sha256", "f44e64e75f3948e9f73f8dfa94721c4ce8cbb4f265c4790c702b2d41cfbf2753"}}, + {{"path", "c.bin"}, {"size", 5}, {"sha256", "be9d587defa1f0c09ef49eb17e206983a5f8f8289e4281860bd0ee5a19592c67"}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + models = model_list(catalog_string, root_string); + } + + fs::path model_path() const { return root / "models" / "test-model"; } + void WriteValidFiles() const { + Write(model_path() / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); + Write(model_path() / "a.bin", "alpha"); + Write(model_path() / "b.bin", "beta"); + Write(model_path() / "c.bin", "gamma"); + } +}; + +void TestModelIsReadyOnlyWhenAllFourFinalFilesValidate() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Ready); + Write(fixture.model_path() / "b.bin", "BETA"); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +void TestPartFileNeverMakesModelReady() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + fs::rename(fixture.model_path() / "c.bin", fixture.model_path() / "c.bin.part"); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +void TestLegacyReadyCheckDoesNotHashOrDeleteWeights() { + const auto root = TempDirectory("legacy-ready"); + const auto catalog_path = root / "model_list.json"; + const auto info_path = root / "model_info.json"; + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"legacy-model", {{"1b", { + {"name", "legacy-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"files", {"config.json", "model.bin"}} + }}}}}}}; + const nlohmann::json info = {{"legacy-model:1b", { + {{"path", "config.json"}, {"size", 23}, {"oid", std::string(40, '0')}}, + {{"path", "model.bin"}, {"size", 8}, {"oid", std::string(40, '0')}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / "legacy-model"; + Write(model_path / "config.json", R"({"flm_version":"1.0.3"})"); + Write(model_path / "model.bin", "bad-data"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b") == + ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(Read(model_path / "model.bin") == "bad-data"); + + Write(model_path / "config.json", R"({"model_type":"legacy"})"); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b", true, true) == + ModelDownloader::ModelStatus::Outdated); +} + +void TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks() { + DownloaderFixture fixture; + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool pull_ok = downloader.pull_model("test-model:1b", true); + const bool check_ok = downloader.check_model("test-model:1b", true, true); + std::cout.rdbuf(previous); + + TEST_REQUIRE(!pull_ok); + TEST_REQUIRE(!check_ok); + TEST_REQUIRE(CountOccurrences(output.str(), + "pinned Hugging Face per-file sources are required") == 2); +} + +void TestCheckHashesPinnedFilesExactlyOnce() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(output.str(), "Checking file:") == 4); +} + +void TestStartupStatusDoesNotRehashButCheckStillDoes() { + // Re-hashing the 4 GB GGUF on every launch cost ~28 s, 62% of startup, and + // buys nothing a pull-time verification has not already established. The + // run/serve paths ask for status only; `flm check` remains the full check. + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + + std::ostringstream fast; + auto* previous = std::cout.rdbuf(fast.rdbuf()); + const auto fast_status = downloader.is_model_downloaded("test-model:1b", false, true); + std::cout.rdbuf(previous); + TEST_REQUIRE(fast_status == ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(CountOccurrences(fast.str(), "Checking file:") == 0); + + std::ostringstream full; + previous = std::cout.rdbuf(full.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(full.str(), "Checking file:") == 4); +} + +download_utils::DownloadRequest Request(const fs::path& source, const fs::path& destination, + std::uint64_t size, std::string hash) { + return {FileUrl(source), destination, size, download_utils::HashAlgorithm::Sha256, std::move(hash)}; +} + +void TestResumeAppendsToPartThenAtomicallyPromotes() { + const auto root = TempDirectory("resume"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "abcdefgh"); + Write(destination.string() + ".part", "abcd"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestWrongSizeOrHashNeverReplacesAValidFinalFile() { + const auto root = TempDirectory("wrong"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(!download_utils::download_file_atomic( + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestInterruptedTransferKeepsPartForNextResume() { + const auto root = TempDirectory("interrupted"); + const auto destination = root / "destination.bin"; + Write(destination.string() + ".part", "abcd"); + auto request = Request(root / "missing.bin", destination, 8, + "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"); + TEST_REQUIRE(!download_utils::download_file_atomic(request)); + TEST_REQUIRE(Read(destination.string() + ".part") == "abcd"); + TEST_REQUIRE(!fs::exists(destination)); +} + +void TestSuccessfulForceDownloadAtomicallyReplacesFinalFile() { + const auto root = TempDirectory("replace"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9ac2197d9258257b1ae8463e4214e4cd0a578bc1517f2415928b91be4283fc48"))); + TEST_REQUIRE(Read(destination) == "ABCDEFGH"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} +} // namespace + +int main() { + RunTest(TestAie4CatalogHasExactlyFourFilesAndExpectedDirectoryName, "AIE4 catalog"); + RunTest(TestGgufUrlContainsUnslothRevisionAndFilename, "GGUF URL"); + RunTest(TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename, "frontend URLs"); + RunTest(TestExistingSingleSourceEntryKeepsItsCurrentUrl, "legacy URL"); + RunTest(TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail, "source validation"); + RunTest(TestActualAie4CatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible, + "AIE4 pinned config compatibility"); + RunTest(TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile, "model metadata"); + RunTest(TestModelIsReadyOnlyWhenAllFourFinalFilesValidate, "ready integrity"); + RunTest(TestPartFileNeverMakesModelReady, "part is not ready"); + RunTest(TestLegacyReadyCheckDoesNotHashOrDeleteWeights, "legacy ready behavior"); + RunTest(TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks, + "modelscope rejection"); + RunTest(TestCheckHashesPinnedFilesExactlyOnce, "single check verification"); + RunTest(TestStartupStatusDoesNotRehashButCheckStillDoes, "startup status skips rehash"); + RunTest(TestResumeAppendsToPartThenAtomicallyPromotes, "resume and promote"); + RunTest(TestWrongSizeOrHashNeverReplacesAValidFinalFile, "invalid transfer isolation"); + RunTest(TestInterruptedTransferKeepsPartForNextResume, "interrupted transfer"); + RunTest(TestSuccessfulForceDownloadAtomicallyReplacesFinalFile, "atomic replacement"); +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_engine.cpp b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp new file mode 100644 index 00000000..48ae7d65 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_engine.cpp @@ -0,0 +1,546 @@ +#include "models/phi4/phi4_corelib_aie4.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" +#include "models/phi4/phi4_corelib_host.hpp" +#include "fake_corelib.hpp" +#include "gguf_fixture.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibRuntime; +using flm::phi4::Phi4GgufPackage; +using flm::phi4::phi4_corelib_aie4; + +const std::filesystem::path& FullPackagePath() { + static auto file = gguf_fixture::Builder().AddFullContractTensors(false).Write("engine"); + return file.path; +} + +struct Harness { + std::shared_ptr runtime; + std::shared_ptr package; + std::unique_ptr engine; + + explicit Harness(std::function configure = {}) { + fake_corelib::Reset(); + if (configure) configure(fake_corelib::GetState()); + runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + package = Phi4GgufPackage::Open(FullPackagePath()); + engine = std::make_unique(LM_Config{}, package, runtime); + } + ~Harness() { + engine.reset(); + package.reset(); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } +}; + +void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { + Harness h; + const auto& state = fake_corelib::GetState(); + TEST_REQUIRE(state.call_counts.at("ryzenai_corelib_create_stream") == 1); + TEST_REQUIRE(state.tensor_creates.size() == 74); + TEST_REQUIRE(state.tensor_creates[0].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[3].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[4].shape == std::vector({4096, 1024})); + TEST_REQUIRE(state.tensor_creates[6].shape == std::vector({1, 3072})); + TEST_REQUIRE(state.tensor_creates[7].shape == std::vector({1, 200064})); +} + +void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { + Harness h([](auto& state) { + state.pad_row_overrides["matmul-3072"][2048] = 5000; + state.pad_row_overrides["matmul-1024"][2048] = 6000; + state.pad_row_overrides["ssmlp"][2048] = 8000; + state.pad_row_overrides["mha"][2048] = 9000; + }); + const auto& tensors = fake_corelib::GetState().tensor_creates; + TEST_REQUIRE(tensors[0].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[1].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[2].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[3].shape == std::vector({9000, 3072})); + TEST_REQUIRE(tensors[4].shape == std::vector({9000, 1024})); + TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); +} + +void TestEngineCreatesExactly129MatmulAnd32SsmlpWeights() { + Harness h; + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.size() == 161); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "matmul"; }) == 129); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "ssmlp"; }) == 32); + TEST_REQUIRE(std::none_of(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; })); +} + +void TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint() { + // corelib treats threads 0 as ONE deliberately, and this requantizing path + // is compute-bound and scales with the hint. The hint is per-create; the + // creates themselves stay serialized, which + // TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate pins -- + // corelib records 8 CONCURRENT creates on this entry point failing 2 of 10 + // with all-zero output, against 0 of 10 serialized. + Harness h; + for (const auto& record : fake_corelib::GetState().weight_creates) { + TEST_REQUIRE(record.group_size == 64); + TEST_REQUIRE(record.threads == flm::phi4::kRequantizeThreads); + } + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_matmul_bf16_weights_create_gguf"] == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_ssmlp_bf16_weights_create_gguf"] == 0); +} + +void TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate() { + Harness h; + TEST_REQUIRE(fake_corelib::GetState().maximum_active_weight_creates == 1); + const auto& records = fake_corelib::GetState().weight_creates; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto base = layer * 5; + TEST_REQUIRE(records[base + 0].kind == "matmul"); + TEST_REQUIRE(records[base + 1].kind == "matmul"); + TEST_REQUIRE(records[base + 2].kind == "matmul"); + TEST_REQUIRE(records[base + 3].kind == "matmul"); + TEST_REQUIRE(records[base + 4].kind == "ssmlp"); + } + TEST_REQUIRE(records.back().kind == "matmul"); +} + +void TestQkvAndGateUpPointersMatchExactMappedSubranges() { + Harness h; + const auto qkv = h.package->AttentionQkv(0); + const auto gate_up = h.package->GateUp(0); + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records[0].pointers[0] == qkv.values[0].bytes.data()); + TEST_REQUIRE(records[1].pointers[0] == qkv.values[1].bytes.data()); + TEST_REQUIRE(records[2].pointers[0] == qkv.values[2].bytes.data()); + TEST_REQUIRE(records[4].pointers[0] == gate_up.values[0].bytes.data()); + TEST_REQUIRE(records[4].pointers[1] == gate_up.values[1].bytes.data()); +} + +void TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates() { + Harness h; + const auto& records = fake_corelib::GetState().weight_creates; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto base = layer * 5; + const auto qkv = h.package->AttentionQkv(layer); + const auto gate_up = h.package->GateUp(layer); + TEST_REQUIRE(records[base + 0].pointers == + std::vector{qkv.values[0].bytes.data()}); + TEST_REQUIRE(records[base + 1].pointers == + std::vector{qkv.values[1].bytes.data()}); + TEST_REQUIRE(records[base + 2].pointers == + std::vector{qkv.values[2].bytes.data()}); + const std::vector output_pointer{ + h.package->RequireQ8("blk." + std::to_string(layer) + + ".attn_output.weight", std::array{3072, 3072}) + .bytes.data()}; + TEST_REQUIRE(records[base + 3].pointers == output_pointer); + const std::vector mlp_pointers{ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk." + std::to_string(layer) + + ".ffn_down.weight", std::array{3072, 8192}) + .bytes.data()}; + TEST_REQUIRE(records[base + 4].pointers == mlp_pointers); + } + const std::vector embedding_pointer{ + h.package->RequireQ8("token_embd.weight", + std::array{200064, 3072}).bytes.data()}; + TEST_REQUIRE(records.back().pointers == embedding_pointer); +} + +void TestFusedNormsAndEpsilonReachCorelibAsBf16() { + Harness h; + const auto expected = flm::phi4::ConvertF32ToBf16(std::array{1.0e-5f})[0]; + const auto& records = fake_corelib::GetState().weight_creates; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto& record = records[layer * 5 + 4]; + TEST_REQUIRE(record.epsilon == expected); + TEST_REQUIRE(record.norm0.size() == 3072); + TEST_REQUIRE(record.norm1.size() == 3072); + } +} + +void TestEmbeddingMappingOutlivesAllLazyRowReads() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + std::weak_ptr lifetime = package; + auto engine = std::make_unique(LM_Config{}, package, runtime); + package.reset(); + TEST_REQUIRE(!lifetime.expired()); + (void)engine->forward(0); + engine.reset(); + TEST_REQUIRE(lifetime.expired()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); +} + +void TestNoDeviceObjectExistsWhenPackageValidationFails() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto file = gguf_fixture::Builder().AddExactFixtureTensors().Write("invalid-engine"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { + phi4_corelib_aie4 engine(LM_Config{}, package, runtime); + }), "blk.1"); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_create_stream"] == 0); + package.reset(); runtime.reset(); CorelibRuntime::ShutdownProcess(); +} + +void TestPrefillDecodesEmbeddingRowsAndAdvancesPosition() { + Harness h; + std::vector ids{2, 1, 2}; + const auto logits = h.engine->prefill(ids); + TEST_REQUIRE(logits.size() == 200064); + TEST_REQUIRE(h.engine->get_current_context_length() == 3); + TEST_REQUIRE(fake_corelib::GetState().tensor_writes[2].source_type == ryzenai_corelib_data_type_fp32); +} + +void TestDecodeUsesOneRowAndAdvancesPosition() { + Harness h; + (void)h.engine->forward(4); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().dispatches.front().rows == 1); +} + +void TestVProjectionWritesWindowAtPositionTimes128() { + Harness h; + h.engine->set_context_length(7); + (void)h.engine->forward(1); + const auto& windows = fake_corelib::GetState().tensor_windows; + TEST_REQUIRE(windows.size() == 32); + TEST_REQUIRE(windows.front().shape == std::vector({8, 4089, 128})); + TEST_REQUIRE(windows.front().offset == 7 * 128); + TEST_REQUIRE(fake_corelib::GetState().dispatches[2].window_offset == 7 * 128); +} + +void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { + Harness h; + (void)h.engine->forward(1); + const auto& calls = fake_corelib::GetState().dispatches; + TEST_REQUIRE(calls.size() == 193); + const void* stream = calls.front().stream; + TEST_REQUIRE(calls.front().kind == "matmul"); + for (std::size_t layer = 0; layer < 32; ++layer) { + const std::size_t base = layer * 6; + TEST_REQUIRE(calls[base + 0].kind == "matmul"); + TEST_REQUIRE(calls[base + 1].kind == "matmul"); + TEST_REQUIRE(calls[base + 2].kind == "matmul"); + TEST_REQUIRE(calls[base + 3].kind == "mha"); + TEST_REQUIRE(calls[base + 4].kind == "matmul"); + TEST_REQUIRE(calls[base + 5].kind == "ssmlp"); + } + TEST_REQUIRE(std::all_of(calls.begin(), calls.end(), [&](const auto& c) { return c.stream == stream; })); +} + +void TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual() { + Harness h; + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes.size() >= 2); + TEST_REQUIRE(writes[0].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[1].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[0].count == writes[1].count); +} + +void TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket() { + Harness h([](auto& state) { + state.pad_row_overrides["matmul-1024"][64] = 96; + }); + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes[0].count == 96 * 3072); + TEST_REQUIRE(writes[1].count == 96 * 3072); + TEST_REQUIRE(writes[1].all_zero); +} + +void TestForwardSynchronizesBeforeHostReadAndLmHeadRead() { + Harness h; + fake_corelib::GetState().call_log.clear(); + (void)h.engine->forward(1); + const auto& log = fake_corelib::GetState().call_log; + const auto first_sync = std::find(log.begin(), log.end(), "ryzenai_corelib_stream_synchronize"); + const auto first_read = std::find(log.begin(), log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(first_sync < first_read); + const auto lm_submit = std::find(first_read, log.end(), "ryzenai_corelib_matmul_bf16"); + const auto second_sync = std::find(lm_submit, log.end(), "ryzenai_corelib_stream_synchronize"); + const auto logits_read = std::find(second_sync, log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(lm_submit < second_sync && second_sync < logits_read); +} + +void TestKVCachesRemainFixedAt8By4096By128() { + Harness h; + const auto& creates = fake_corelib::GetState().tensor_creates; + const auto count = std::count_if(creates.begin(), creates.end(), [](const auto& record) { + return record.shape == std::vector({8, 4096, 128}); + }); + TEST_REQUIRE(count == 64); +} + +void TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken() { + Harness h; + std::vector ids(4096, 0); + (void)h.engine->prefill(ids); + TEST_REQUIRE(h.engine->get_current_context_length() == 4096); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "capacity"); +} + +void TestTotalDecodeWindowStopsAt4095() { + Harness h; + h.engine->set_context_length(4094); + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->get_current_context_length() == 4095); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "4095"); +} + +void TestClearContextResetsLogicalPositionWithoutRecreatingWeights() { + Harness h; + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + h.engine->clear_context(); + TEST_REQUIRE(h.engine->get_current_context_length() == 0); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestCheckpointRestoreChangesOnlyLogicalPosition() { + Harness h; + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->checkpoint() == 1); + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + TEST_REQUIRE(h.engine->restore() == 1); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestPreSubmitFailureIsRecoverable() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = ryzenai_corelib_status_bad_argument; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "tensor_write"); + TEST_REQUIRE(!h.engine->poisoned()); + fake_corelib::GetState().statuses.erase("ryzenai_corelib_tensor_write"); + (void)h.engine->forward(0); +} + +void TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState() { + Harness h; + h.engine->set_context_length(3); + h.engine->checkpoint(); + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "matmul"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestSynchronizeFailurePoisonsAndClearsState() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_stream_synchronize"] = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "synchronize"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestPoisonedInstanceRejectsEveryLaterEntryPoint() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + RequireContains(RequireThrows([&] { h.engine->clear_context(); }), "poisoned"); + RequireContains(RequireThrows([&] { (void)h.engine->get_current_context_length(); }), "poisoned"); + std::vector ids{0}; + RequireContains(RequireThrows([&] { (void)h.engine->prefill(ids); }), "poisoned"); +} + +void TestFakeTensorWindowRetainsAndPropagatesParentStorage() { + fake_corelib::Reset(); + auto api = CorelibApi::ResolveForTest(fake_corelib::Resolver()); + const std::array parent_shape{16}; + void* parent = nullptr; + api->Check(api->functions().create_device_tensor( + ryzenai_corelib_data_type_bf16, parent_shape.data(), parent_shape.size(), &parent), + "create parent"); + const std::array original{11, 22, 33, 44}; + api->Check(api->functions().tensor_write(parent, ryzenai_corelib_data_type_bf16, + original.data(), original.size(), 4), + "write parent"); + const std::array window_shape{4}; + void* window = nullptr; + api->Check(api->functions().create_tensor_window( + parent, window_shape.data(), window_shape.size(), 4, &window), "create window"); + std::array read{}; + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + read.data(), read.size(), 0), "read window"); + TEST_REQUIRE(read == original); + const std::array replacement{77, 88}; + api->Check(api->functions().tensor_write(window, ryzenai_corelib_data_type_bf16, + replacement.data(), replacement.size(), 1), + "write window"); + std::array reread{}; + api->Check(api->functions().tensor_read(parent, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 4), "read parent"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(parent); + reread.fill(0); + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 0), "reread retained window"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(window); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); +} + +void WriteCacheRow(Harness& h, std::size_t tensor_index, int position, + std::uint16_t base) { + auto& record = fake_corelib::GetState().tensor_creates[tensor_index]; + for (std::size_t head = 0; head < 8; ++head) { + std::array values{}; + values.fill(static_cast(base + head)); + h.runtime->api()->Check(h.runtime->api()->functions().tensor_write( + record.object, ryzenai_corelib_data_type_bf16, values.data(), values.size(), + (head * 4096 + position) * 128), "seed cache row"); + } +} + +void TestGetKCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 10, 7, 100); + const auto result = h.engine->get_k_cache(0, 7); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 100 + head); +} + +void TestGetVCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 11, 9, 200); + const auto result = h.engine->get_v_cache(0, 9); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 200 + head); +} + +void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_ssmlp_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestTwoConcurrentAie4RequestsNeverOverlapDispatch() { + Harness h; + auto second_engine = std::make_unique( + LM_Config{}, h.package, h.runtime); + fake_corelib::GetState().maximum_active_leases = 0; + fake_corelib::GetState().statuses["test_observe_dispatch_concurrency"] = + ryzenai_corelib_status_success; + fake_corelib::GetState().dispatches.clear(); + std::barrier start(3); + std::thread first([&] { start.arrive_and_wait(); (void)h.engine->forward(1); }); + std::thread second([&] { start.arrive_and_wait(); (void)second_engine->forward(2); }); + start.arrive_and_wait(); + first.join(); + second.join(); + + const auto& dispatches = fake_corelib::GetState().dispatches; + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + TEST_REQUIRE(dispatches.size() == 386); + const auto first_request = dispatches.front().thread_id; + TEST_REQUIRE(first_request != dispatches.back().thread_id); + TEST_REQUIRE(std::all_of(dispatches.begin(), dispatches.begin() + 193, + [&](const auto& call) { + return call.thread_id == first_request; + })); + TEST_REQUIRE(std::all_of(dispatches.begin() + 193, dispatches.end(), + [&](const auto& call) { + return call.thread_id != first_request; + })); + + // Prove the fake itself does not serialize or race when the runtime lease is + // intentionally bypassed: the overlap detector must report both calls. + fake_corelib::GetState().dispatches.clear(); + fake_corelib::GetState().maximum_active_leases = 0; + std::barrier unsafe_start(3); + std::atomic unsafe_calls_succeeded{true}; + const auto invoke_without_lease = [&] { + unsafe_start.arrive_and_wait(); + if (h.runtime->api()->functions().matmul( + nullptr, nullptr, 1, nullptr, nullptr) != + ryzenai_corelib_status_success) + unsafe_calls_succeeded = false; + }; + std::thread unsafe_first(invoke_without_lease); + std::thread unsafe_second(invoke_without_lease); + unsafe_start.arrive_and_wait(); + unsafe_first.join(); + unsafe_second.join(); + TEST_REQUIRE(unsafe_calls_succeeded); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 2); + TEST_REQUIRE(fake_corelib::GetState().dispatches.size() == 2); +} + +void TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits() { + for (int cycle = 0; cycle < 10; ++cycle) { + { + Harness h; + const auto logits = h.engine->forward(cycle); + const auto* bits = reinterpret_cast(logits.data()); + TEST_REQUIRE(std::any_of(bits, bits + logits.size(), + [](std::uint16_t value) { return value != 0; })); + } + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + } +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); + RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); + RUN_TEST(TestEngineCreatesExactly129MatmulAnd32SsmlpWeights); + RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint); + RUN_TEST(TestWeightCreationIsSerialAndNeverExceedsOneInFlightCreate); + RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); + RUN_TEST(TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates); + RUN_TEST(TestFusedNormsAndEpsilonReachCorelibAsBf16); + RUN_TEST(TestEmbeddingMappingOutlivesAllLazyRowReads); + RUN_TEST(TestNoDeviceObjectExistsWhenPackageValidationFails); + RUN_TEST(TestPrefillDecodesEmbeddingRowsAndAdvancesPosition); + RUN_TEST(TestDecodeUsesOneRowAndAdvancesPosition); + RUN_TEST(TestVProjectionWritesWindowAtPositionTimes128); + RUN_TEST(TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream); + RUN_TEST(TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual); + RUN_TEST(TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket); + RUN_TEST(TestForwardSynchronizesBeforeHostReadAndLmHeadRead); + RUN_TEST(TestKVCachesRemainFixedAt8By4096By128); + RUN_TEST(TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken); + RUN_TEST(TestTotalDecodeWindowStopsAt4095); + RUN_TEST(TestClearContextResetsLogicalPositionWithoutRecreatingWeights); + RUN_TEST(TestCheckpointRestoreChangesOnlyLogicalPosition); + RUN_TEST(TestPreSubmitFailureIsRecoverable); + RUN_TEST(TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState); + RUN_TEST(TestSynchronizeFailurePoisonsAndClearsState); + RUN_TEST(TestPoisonedInstanceRejectsEveryLaterEntryPoint); + RUN_TEST(TestFakeTensorWindowRetainsAndPropagatesParentStorage); + RUN_TEST(TestGetKCacheGathersHeadMajorPosition); + RUN_TEST(TestGetVCacheGathersHeadMajorPosition); + RUN_TEST(TestCancellationBoundaryLeavesNoOutstandingFakeWork); + RUN_TEST(TestTwoConcurrentAie4RequestsNeverOverlapDispatch); + RUN_TEST(TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits); +#undef RUN_TEST +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp new file mode 100644 index 00000000..2bb19b7e --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_frontend.cpp @@ -0,0 +1,740 @@ +#include "test_support.hpp" +#include "gguf_fixture.hpp" +#if defined(FLM_ENABLE_CORELIB_AIE4) +#include "fake_corelib.hpp" +#endif +#include "utils/file_access.hpp" + +#include +#include "server.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +std::vector g_encoded_tokens; +std::vector g_samples; +std::vector g_opened_paths; +std::size_t g_sample_index{}; + +class FakeEngine final : public causal_lm { +public: + explicit FakeEngine(std::uint32_t limit) : max_length(limit) {} + + buffer forward(int token) override { + ++forward_calls; + forwarded.push_back(token); + if (forward_delay.count()) std::this_thread::sleep_for(forward_delay); + if (fail_forward) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + ++position; + return buffer(1); + } + buffer prefill(std::vector& tokens, void*) override { + ++prefill_calls; + if (fail_prefill) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + position += static_cast(tokens.size()); + return buffer(1); + } + void set_context_length(int value) override { position = value; } + void load_weights(Q4NX&) override {} + void update_max_length(std::uint32_t value) override { max_length = value; } + void clear_context() override { + if (poisoned_state) throw std::runtime_error("poisoned"); + position = 0; + } + buffer get_k_cache(int, int) override { return buffer(1); } + buffer get_v_cache(int, int) override { return buffer(1); } + int get_current_context_length() override { return position; } + int checkpoint() override { return position; } + int restore() override { return position; } + + std::uint32_t max_length; + int position{}; + int prefill_calls{}; + int forward_calls{}; + bool fail_prefill{}; + bool fail_forward{}; + bool poisoned_state{}; + std::chrono::microseconds forward_delay{0}; + std::vector forwarded; +}; + +struct FactoryState { + int legacy_calls{}; + int aie4_calls{}; + bool throw_for_aie4{}; + FakeEngine* engine{}; +} g_factory; + +class TempPackage final { +public: + explicit TempPackage(bool valid = true) { + static std::uint64_t serial{}; + path_ = std::filesystem::temp_directory_path() / + ("flm-task4-" + std::to_string(++serial)); + std::filesystem::create_directories(path_); + Write(path_ / "config.json", gguf_fixture::ValidConfig()); + Write(path_ / "tokenizer.json", gguf_fixture::ValidTokenizer()); + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["eos_token"] = ""; + tokenizer_config["eos_token_id"] = nlohmann::json::array({200020, 199999}); + Write(path_ / "tokenizer_config.json", tokenizer_config); + auto gguf = gguf_fixture::Builder().AddFullContractTensors(false).Write("frontend"); + std::filesystem::rename(gguf.path, path_ / "Phi-4-mini-instruct.Q8_0.gguf"); + if (!valid) { + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + Write(path_ / "config.json", config); + } + } + ~TempPackage() { + std::error_code ignored; + std::filesystem::remove_all(path_, ignored); + } + const std::filesystem::path& path() const { return path_; } +private: + static void Write(const std::filesystem::path& path, const nlohmann::json& value) { + std::ofstream out(path, std::ios::binary); + if (!out) throw std::runtime_error("cannot write package fixture"); + out << value.dump(); + } + std::filesystem::path path_; +}; + +nlohmann::ordered_json ModelInfo(std::optional backend = std::nullopt) { + nlohmann::ordered_json details = {{"family", "phi4"}}; + if (backend) details["execution_backend"] = *backend; + return {{"default_context_length", 4096}, {"details", details}}; +} + +chat_meta_info_t Meta() { + chat_meta_info_t value; + value.max_prefill_len = 64; + return value; +} + +lm_uniform_input_t Input(std::optional budget = std::nullopt) { + lm_uniform_input_t value; + value.prompt = "prompt"; + value.requested_max_new_tokens = budget; + return value; +} + +template +void ExpectRequestError(F&& action, int code, bool cleared, std::string_view text) { + try { action(); } + catch (const ModelRequestError& error) { + TEST_REQUIRE(error.http_code() == code); + TEST_REQUIRE(error.session_cleared() == cleared); + RequireContains(error.what(), text); + return; + } + throw std::runtime_error("expected ModelRequestError"); +} + +} // namespace + +Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); + is_doubled_encoded = false; +} +Tokenizer::~Tokenizer() = default; +std::vector Tokenizer::encode(const std::string&) { return g_encoded_tokens; } +std::string Tokenizer::decode(const std::vector&) { return "decoded"; } +std::string Tokenizer::run_time_decoder(int token) { return "t" + std::to_string(token); } +SafeTensors::~SafeTensors() = default; + +Sampler::Sampler(int features, sampler_config& config) + : in_features(features), rep_penalty(config.rep_penalty), + freq_penalty(config.freq_penalty), pre_penalty(config.pre_penalty), + top_k(config.top_k), top_p(config.top_p), min_p(config.min_p), + temperature(config.temperature), total_tokens(0), + freq_penalty_window(config.freq_penalty_window), + rep_penalty_window(config.rep_penalty_window), + repeat_last_n(config.repeat_last_n), + use_optimized_sampling(config.use_optimized_sampling) { + logits.resize(1); counters.resize(1); token_positions.resize(1, -1); +} +void Sampler::reset_penalties() {} +int Sampler::sample(buffer&) { + if (g_sample_index < g_samples.size()) return g_samples[g_sample_index++]; + return 7; +} + +namespace utils { +std::string get_executable_directory() { return std::filesystem::current_path().string(); } +} + +namespace flm::phi4::testing { +class Phi4FrontendTestAccess final { +public: + static void InstallFactory() { + g_factory = {}; + g_opened_paths.clear(); + flm::file_access::SetOpenObserver([](const auto& path) { + g_opened_paths.push_back(path); + }); + Phi4::engine_factory_for_testing_ = + [](bool aie4, const LM_Config&, npu_xclbin_manager*, + const std::filesystem::path&, std::uint32_t limit) { + if (aie4) { + ++g_factory.aie4_calls; + if (g_factory.throw_for_aie4) throw std::runtime_error("missing corelib"); + } else { + ++g_factory.legacy_calls; + } + auto result = std::make_unique(limit); + g_factory.engine = result.get(); + return std::unique_ptr(std::move(result)); + }; + Phi4::engine_poisoned_for_testing_ = [](const causal_lm* engine) { + return static_cast(engine)->poisoned_state; + }; + } + static void RemoveFactory() { + Phi4::engine_factory_for_testing_ = {}; + Phi4::engine_poisoned_for_testing_ = {}; + flm::file_access::SetOpenObserver({}); + } + static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } + static const std::string& EosToken(const Phi4& model) { return model.eos_token; } + static const std::vector& EosTokenIds(const Phi4& model) { return model.eos_token_ids; } + static bool HasBosToken(const Phi4& model) { return model.has_bos_token; } +#if defined(FLM_ENABLE_CORELIB_AIE4) + static void SetRuntime(Phi4& model, + std::shared_ptr runtime) { + model.corelib_runtime_ = std::move(runtime); + } +#endif +}; +} // namespace flm::phi4::testing + +namespace { +using flm::phi4::testing::Phi4FrontendTestAccess; + +struct FactoryScope { + FactoryScope() { Phi4FrontendTestAccess::InstallFactory(); } + ~FactoryScope() { Phi4FrontendTestAccess::RemoveFactory(); } +}; + +std::unique_ptr Load(const TempPackage& package, + nlohmann::ordered_json info, + int context = -1, + bool preemption = false, + flm_rt::device* device = reinterpret_cast(1)) { + auto model = std::make_unique(device); + model->load_model(package.path().string(), std::move(info), context, preemption); + return model; +} + +void TestAbsentBackendStillBuildsQ4nxPhi4Npu() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); + TEST_REQUIRE(!model->uses_corelib_aie4()); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +void TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + g_samples = {7}; + g_sample_index = 0; + auto meta = Meta(); + auto input = Input(1); + std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 1, output); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +void TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + auto meta = Meta(); + auto input = Input(1); + TEST_REQUIRE(model->insert(meta, input)); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +void TestCorelibAie4GgufBuildsOnlyTheCorelibEngine() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + TEST_REQUIRE(g_factory.legacy_calls == 0); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(model->uses_corelib_aie4()); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +#if defined(FLM_ENABLE_CORELIB_AIE4) +void TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges() { + TempPackage package; + FactoryScope scope; + fake_corelib::Reset(); + const auto dll_a = std::filesystem::absolute(package.path() / "runtime-a.dll"); + const auto dll_b = std::filesystem::absolute(package.path() / "runtime-b.dll"); + auto api = flm::corelib::CorelibApi::ResolveForTest( + fake_corelib::Resolver(), dll_a); + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(std::move(api)); + +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", dll_b.string().c_str()); +#else + setenv("FLM_AIE4_CORELIB_PATH", dll_b.string().c_str(), 1); +#endif + auto aie4 = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + Phi4FrontendTestAccess::SetRuntime(*aie4, runtime); + const auto aie4_profile = aie4->show_profile(); + RequireContains(aie4_profile, "corelib_aie4_gguf"); + RequireContains(aie4_profile, dll_a.string()); + TEST_REQUIRE(aie4_profile.find(dll_b.string()) == std::string::npos); + + auto legacy = Load(package, ModelInfo()); + const auto legacy_profile = legacy->show_profile(); + TEST_REQUIRE(legacy_profile.find("corelib_aie4_gguf") == std::string::npos); + TEST_REQUIRE(legacy_profile.find(dll_a.string()) == std::string::npos); + aie4.reset(); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); +#ifdef _WIN32 + _putenv_s("FLM_AIE4_CORELIB_PATH", ""); +#else + unsetenv("FLM_AIE4_CORELIB_PATH"); +#endif +} +#endif + +void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + TEST_REQUIRE(model->uses_corelib_aie4()); + std::vector names; + for (const auto& path : g_opened_paths) { + const auto text = path.generic_string(); + TEST_REQUIRE(text.find("manifest") == std::string::npos); + TEST_REQUIRE(text.find("onnx") == std::string::npos); + TEST_REQUIRE(text.find("converted") == std::string::npos); + TEST_REQUIRE(text.find("cache") == std::string::npos); + names.push_back(path.filename().string()); + } + std::sort(names.begin(), names.end()); + TEST_REQUIRE(names == std::vector({ + "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "config.json", + "tokenizer.json", "tokenizer.json", "tokenizer_config.json"})); +} + +void TestUnknownAndNonStringBackendAreErrors() { + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("other")); }), "Unknown"); + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo(7)); }), "string"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.aie4_calls == 0); +} + +void TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders() { +#if !defined(FLM_ENABLE_CORELIB_AIE4) + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { + (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + }), "This binary was built without Phi-4 AIE4 corelib support"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.aie4_calls == 0); +#endif +} + +void TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation() { + TempPackage package(false); + FactoryScope scope; + RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +void TestMissingCorelibFailsOnlyWhenAie4ModelLoads() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }), "missing corelib"); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestAie4SelectionWithMissingDllFailsWithoutChangingBackend() { + TempPackage package; + FactoryScope scope; + Phi4 model(nullptr); + g_factory.throw_for_aie4 = true; + RequireContains(RequireThrows([&] { + model.load_model(package.path().string(), ModelInfo("corelib_aie4_gguf")); + }), "missing corelib"); + TEST_REQUIRE(!model.uses_corelib_aie4()); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + (void)RequireThrows([&] { + (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); + }); + TEST_REQUIRE(g_factory.aie4_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_aie4 = true; + RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); }); + g_factory.throw_for_aie4 = false; + auto ordinary = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(!ordinary->uses_corelib_aie4()); +} + +void TestPreemptionIsRejectedForTheAie4Route() { + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo("corelib_aie4_gguf"), -1, true, nullptr); }), "preemption"); + TEST_REQUIRE(g_factory.aie4_calls == 0); +} + +std::unique_ptr ReadyAie4(const TempPackage& package) { + return Load(package, ModelInfo("corelib_aie4_gguf"), -1, false, nullptr); +} + +void TestRenderedPromptPlusExplicitBudgetMayEqual4095() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(95); + TEST_REQUIRE(model->insert(meta, input)); +} + +void TestRenderedPromptPlusExplicitBudgetAbove4095Is400() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(96); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 400, false, "4095"); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow() { + for (const auto requested : {std::optional{}, std::optional{0}, std::optional{-1}}) { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(requested); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); + TEST_REQUIRE(meta.stop_reason == MAX_LENGTH_REACHED); + } + + // /api/chat uses generate_with_prompt and retains 4096 only as the legacy + // loop default; omission must not become an explicit AIE4 budget. + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + (void)model->generate_with_prompt(meta, input, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); +} + +void TestCancellationBeforePrefillSubmitsNothing() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + int checks = 0; + TEST_REQUIRE(!model->insert(meta, input, [&] { return ++checks >= 2; })); + TEST_REQUIRE(checks >= 2); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); +} + +void TestCancellationBetweenDecodeStepsStopsWithCancelReason() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11, 12}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + int checks = 0; + (void)model->generate(meta, 10, output, [&] { return checks++ == 2; }); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->forward_calls == 1); +} + +void TestCancellationReturnsOnlyAfterSynchronize() { + // Fake calls are synchronous by construction: observing one completed call + // before cancellation proves no work remains outstanding at return. + TestCancellationBetweenDecodeStepsStopsWithCancelReason(); +} + +void TestNonStreamingChatGenerateWithPromptForwardsCancellation() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + std::ostringstream output; + int checks = 0; + AutoModel* endpoint_model = model.get(); + const auto response = endpoint_model->generate_with_prompt( + meta, input, 4096, output, [&] { return ++checks >= 2; }); + TEST_REQUIRE(response.empty()); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestLegacyTokenizerContractIsPreserved() { + TempPackage package; FactoryScope scope; auto legacy = Load(package, ModelInfo()); + // Main's legacy Phi-4 frontend intentionally did not pass the textual EOS + // token into minja and retained an empty eos_token string. + TEST_REQUIRE(Phi4FrontendTestAccess::EosToken(*legacy).empty()); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*legacy) == + std::vector({200020, 199999})); + + auto aie4 = ReadyAie4(package); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*aie4) == + std::vector({200020, 199999})); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasBosToken(*aie4)); +} + +void TestSamePathBackendSwitchForcesLegacyInitialization() { + TempPackage package; FactoryScope scope; + Phi4 model(reinterpret_cast(1)); + model.load_model(package.path().string(), ModelInfo("corelib_aie4_gguf")); + TEST_REQUIRE(model.uses_corelib_aie4()); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(model)); + model.load_model(package.path().string(), ModelInfo()); + TEST_REQUIRE(!model.uses_corelib_aie4()); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(model)); + TEST_REQUIRE(g_factory.legacy_calls == 1); +} + +void TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + TEST_REQUIRE(model->get_current_context_length() == 0); +} + +void TestPoisonedModelReturns500UntilReload() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 500, true, "unload/reload"); + auto reloaded = ReadyAie4(package); + TEST_REQUIRE(reloaded->insert(meta, input)); +} + +void TestEosSelfTerminatesWithoutAnExtraDecode() { + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 0); + TEST_REQUIRE(meta.stop_reason == EOT_DETECTED); +} + +void TestAie4DecodeTimeAndSpeedAreMeasured() { + // The AIE4 route has its own decode loop, so it must record DECODING_TIME + // itself. Without that the profile reports "0 us" and a nan speed, and the + // hardware acceptance record has no decode throughput to publish. + TempPackage package; FactoryScope scope; auto model = ReadyAie4(package); + g_encoded_tokens = {1}; g_samples = {11, 12, 13, 200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->forward_delay = std::chrono::microseconds(2000); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 3); + TEST_REQUIRE(meta.decoding_duration > 0); + const auto profile = model->show_profile(); + TEST_REQUIRE(profile.find("nan") == std::string::npos); + TEST_REQUIRE(profile.find("inf") == std::string::npos); + TEST_REQUIRE(profile.find("Decoding time: 0 ") == std::string::npos); +} + +void TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics() { + for (const auto raw : {std::optional{}, std::optional{0}, std::optional{-2}, std::optional{17}}) { + const auto expected = raw && *raw > 0 ? raw : std::nullopt; + for (int source = 0; source < 5; ++source) + TEST_REQUIRE(normalize_requested_max_new_tokens(raw) == expected); + } +} + +void TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint() { + TEST_REQUIRE(requires_npu_access("POST", "/v1/completions")); + for (int path = 0; path < 5; ++path) { + int releases = 0; + { + NPURequestCompletionGuard guard([&] { ++releases; }); + if (path == 0) guard.complete(); + else if (path == 1) { guard.complete(); guard.complete(); } + else if (path == 2) { NPURequestCompletionGuard moved(std::move(guard)); } + else if (path == 3) { try { throw std::runtime_error("model"); } catch (...) {} } + else { try { throw 1; } catch (...) {} } + } + TEST_REQUIRE(releases == 1); + } +} + +void TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff() { + constexpr auto cooldown = std::chrono::milliseconds(100); + + NPURequestCoordinator empty; + bool released = false; + const auto empty_start = std::chrono::steady_clock::now(); + empty.complete_current([](auto) { TEST_REQUIRE(false); }, + [&] { released = true; }, cooldown); + const auto empty_elapsed = std::chrono::steady_clock::now() - empty_start; + TEST_REQUIRE(released); + TEST_REQUIRE(empty_elapsed < std::chrono::milliseconds(50)); + + NPURequestCoordinator queued; + bool handed_off = false; + bool released_while_queued = false; + TEST_REQUIRE(queued.try_enqueue([] {})); + const auto queued_start = std::chrono::steady_clock::now(); + queued.complete_current( + [&](auto task) { + handed_off = true; + task(); + }, + [&] { released_while_queued = true; }, cooldown); + const auto queued_elapsed = std::chrono::steady_clock::now() - queued_start; + TEST_REQUIRE(handed_off); + TEST_REQUIRE(!released_while_queued); + TEST_REQUIRE(queued_elapsed >= std::chrono::milliseconds(75)); +} + +void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { + TempPackage package; + FactoryScope scope; + auto model = ReadyAie4(package); + NPURequestCoordinator coordinator(3); + bool cancelled = false; + bool capacity_failed = false; + bool queued_request_ran = false; + int completion_callbacks = 0; + int accelerator_releases = 0; + + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens = {1}; + cancelled = !model->insert(meta, input, [] { return true; }); + })); + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens.assign(4095, 1); + try { (void)model->insert(meta, input); } + catch (const ModelRequestError& error) { + capacity_failed = error.http_code() == 400; + } + })); + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens = {1}; + queued_request_ran = model->insert(meta, input); + })); + TEST_REQUIRE(!coordinator.try_enqueue([] {})); + + std::function)> execute; + const auto complete = [&] { + ++completion_callbacks; + coordinator.complete_current(execute, [&] { ++accelerator_releases; }, + std::chrono::milliseconds(0)); + }; + execute = [&](std::function task) { + NPURequestCompletionGuard completion(complete); + task(); + completion.complete(); + completion.complete(); + }; + { + NPURequestCompletionGuard active_request_completion(complete); + active_request_completion.complete(); + active_request_completion.complete(); + } + + TEST_REQUIRE(cancelled); + TEST_REQUIRE(capacity_failed); + TEST_REQUIRE(queued_request_ran); + TEST_REQUIRE(coordinator.empty()); + TEST_REQUIRE(completion_callbacks == 4); + TEST_REQUIRE(accelerator_releases == 1); +} + +} // namespace + +int main() { +#if defined(FLM_ENABLE_CORELIB_AIE4) + RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); + RunTest(TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing, "TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing"); + RunTest(TestCorelibAie4GgufBuildsOnlyTheCorelibEngine, "TestCorelibAie4GgufBuildsOnlyTheCorelibEngine"); + RunTest(TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges, "TestAie4ProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges"); + RunTest(TestNoManifestOnnxConvertedWeightOrCachePathIsOpened, "TestNoManifestOnnxConvertedWeightOrCachePathIsOpened"); + RunTest(TestUnknownAndNonStringBackendAreErrors, "TestUnknownAndNonStringBackendAreErrors"); + RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); + RunTest(TestMissingCorelibFailsOnlyWhenAie4ModelLoads, "TestMissingCorelibFailsOnlyWhenAie4ModelLoads"); + RunTest(TestAie4SelectionWithMissingDllFailsWithoutChangingBackend, "TestAie4SelectionWithMissingDllFailsWithoutChangingBackend"); + RunTest(TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback, "TestAie4SelectionCannotReachQ4nxPhi4NpuOrCpuFallback"); + RunTest(TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure, "TestOrdinaryModelLoadsAfterAnAie4RuntimeLoadFailure"); + RunTest(TestPreemptionIsRejectedForTheAie4Route, "TestPreemptionIsRejectedForTheAie4Route"); + RunTest(TestRenderedPromptPlusExplicitBudgetMayEqual4095, "TestRenderedPromptPlusExplicitBudgetMayEqual4095"); + RunTest(TestRenderedPromptPlusExplicitBudgetAbove4095Is400, "TestRenderedPromptPlusExplicitBudgetAbove4095Is400"); + RunTest(TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow, "TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow"); + RunTest(TestCancellationBeforePrefillSubmitsNothing, "TestCancellationBeforePrefillSubmitsNothing"); + RunTest(TestCancellationBetweenDecodeStepsStopsWithCancelReason, "TestCancellationBetweenDecodeStepsStopsWithCancelReason"); + RunTest(TestCancellationReturnsOnlyAfterSynchronize, "TestCancellationReturnsOnlyAfterSynchronize"); + RunTest(TestNonStreamingChatGenerateWithPromptForwardsCancellation, "TestNonStreamingChatGenerateWithPromptForwardsCancellation"); + RunTest(TestLegacyTokenizerContractIsPreserved, "TestLegacyTokenizerContractIsPreserved"); + RunTest(TestSamePathBackendSwitchForcesLegacyInitialization, "TestSamePathBackendSwitchForcesLegacyInitialization"); + RunTest(TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned, "TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned"); + RunTest(TestPoisonedModelReturns500UntilReload, "TestPoisonedModelReturns500UntilReload"); + RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); + RunTest(TestAie4DecodeTimeAndSpeedAreMeasured, "TestAie4DecodeTimeAndSpeedAreMeasured"); + RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); + RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); + RunTest(TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff, "TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff"); + RunTest(TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable, "TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable"); +#else + RunTest(TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib, "TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib"); + RunTest(TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders, "TestFeatureOffRejectsAie4TagWithoutIncludingCorelibHeaders"); +#endif + std::cout << "test_phi4_frontend: PASS\n"; +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp new file mode 100644 index 00000000..492d04f5 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_gguf.cpp @@ -0,0 +1,580 @@ +#include "gguf_fixture.hpp" +#include "fake_corelib.hpp" +#include "corelib/corelib_api.hpp" +#include "models/phi4/phi4_corelib_constants.hpp" +#include "models/phi4/phi4_corelib_gguf.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::phi4::Phi4GgufPackage; +using gguf_fixture::Builder; +using gguf_fixture::Mutation; + +std::shared_ptr Open(Builder builder, + gguf_fixture::TempFile& file, + std::string_view label) { + file = builder.Write(label); + return Phi4GgufPackage::Open(file.path); +} + +std::string OpenFailure(Builder builder, Mutation mutation, + std::string_view label) { + auto file = builder.Apply(mutation).Write(label); + return RequireThrows([&] { Phi4GgufPackage::Open(file.path); }); +} + +void RequireMismatch(std::string_view error, std::string_view field, + std::string_view actual, std::string_view expected) { + RequireContains(error, field); + RequireContains(error, "actual " + std::string(actual)); + RequireContains(error, "expected " + std::string(expected)); +} + +std::string ShapeText(const std::vector& shape) { + std::string result = "["; + for (std::size_t index = 0; index < shape.size(); ++index) { + if (index != 0) result += ','; + result += std::to_string(shape[index]); + } + return result + ']'; +} + +struct TensorRole { + std::string name; + std::vector shape; + std::uint32_t type; +}; + +std::vector RequiredTensorRoles() { + std::vector roles = { + {"token_embd.weight", {200064, 3072}, gguf_fixture::kQ8_0}, + {"output_norm.weight", {3072}, gguf_fixture::kF32}}; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + roles.push_back({prefix + ".attn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".ffn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".attn_output.weight", {3072, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_down.weight", {3072, 8192}, gguf_fixture::kQ8_0}); + } + return roles; +} + +Builder SplitFixture() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0) + .AddTensor("blk.0.ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0) + .AddTensor("f32", {48}, gguf_fixture::kF32); + return builder; +} + +void TestValidV3HeaderMetadataDirectoryAndAlignment() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "valid"); + const auto metadata = package->Metadata(); + TEST_REQUIRE(metadata.architecture == "phi3"); + TEST_REQUIRE(metadata.layer_count == 32); + TEST_REQUIRE(metadata.tokenizer_vocabulary_size == 200064); + TEST_REQUIRE(!metadata.add_bos_token); +} + +void TestOmittedAlignmentUsesGgufDefault32() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().RemoveMetadata("general.alignment"), + file, "default-alignment"); + TEST_REQUIRE(package->RequireF32( + "f32", std::array{48}).values.size() == 48); +} + +void TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().AddEverySkippableMetadataType(), file, + "metadata-types"); + TEST_REQUIRE(package->Metadata().hidden_size == 3072); +} + +void TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail() { + const auto truncated_header = std::filesystem::temp_directory_path() / "flm_phi4_short_header.gguf"; + { std::ofstream out(truncated_header, std::ios::binary | std::ios::trunc); out << "GG"; } + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(truncated_header); }), "header"); + std::error_code ignored; std::filesystem::remove(truncated_header, ignored); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedString, "truncated-string"), "string"); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedDirectory, "truncated-directory"), "tensor"); + + Builder array; + array.RemoveMetadata("tokenizer.ggml.tokens") + .AddMetadata("tokenizer.ggml.tokens", gguf_fixture::ArrayValue{ + 8, std::numeric_limits::max(), {}}) + .AddTensor("x", {32}, gguf_fixture::kQ8_0); + auto file = array.Write("truncated-array"); + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(file.path); }), "array"); +} + +void TestCountProductAlignmentAndOffsetOverflowFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::CountOverflow, "count-overflow"), "count"); + RequireContains(OpenFailure(SplitFixture(), Mutation::ProductOverflow, "product-overflow"), "overflow"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OffsetOverflow, "offset-overflow"), "overflow"); +} + +void TestPresentMalformedAlignmentFails() { + RequireContains(OpenFailure(SplitFixture(), Mutation::ZeroAlignment, "zero-align"), "alignment"); + RequireContains(OpenFailure(SplitFixture(), Mutation::NonPowerOfTwoAlignment, "bad-align"), "alignment"); + + auto wrong_type = SplitFixture().SetMetadata( + "general.alignment", std::int32_t{-32}).Write("wrong-align-type"); + RequireMismatch(RequireThrows([&] { Phi4GgufPackage::Open(wrong_type.path); }), + "general.alignment", "INT32", "unsigned integer metadata"); +} + +void TestDuplicateTensorNamesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::DuplicateName, "duplicate"), "duplicate"); +} + +void TestOutOfFileAndOverlappingTensorRangesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::OutOfFileRange, "outside"), "range"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OverlappingRanges, "overlap"), "overlap"); + RequireContains(OpenFailure(SplitFixture(), Mutation::PayloadLengthMismatch, "short-payload"), "range"); +} + +void TestUnsupportedUnskippableMetadataTypeFails() { + RequireContains(OpenFailure(SplitFixture(), Mutation::UnsupportedMetadataType, "unsupported"), "metadata type"); +} + +void TestRequireQ8AndRequireF32ReportNameActualAndExpected() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "requires"); + auto error = RequireThrows([&] { package->RequireQ8("f32", std::array{48}); }); + RequireMismatch(error, "f32", "F32", "Q8_0"); + error = RequireThrows([&] { package->RequireF32("f32", std::array{47}); }); + RequireMismatch(error, "f32", "[48]", "[47]"); +} + +void TestAttentionQkvReturnsThreeZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "qkv"); + const auto fused = package->RequireQ8("blk.0.attn_qkv.weight", std::array{5120, 3072}); + const auto views = package->AttentionQkv(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 3); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 3072 * row_bytes); + TEST_REQUIRE(views.values[2].bytes.data() == fused.bytes.data() + 4096 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({3072, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({1024, 3072})); + TEST_REQUIRE(views.values[2].logical_shape == std::vector({1024, 3072})); +} + +void TestGateUpReturnsTwoZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "gate-up"); + const auto fused = package->RequireQ8("blk.0.ffn_up.weight", std::array{16384, 3072}); + const auto views = package->GateUp(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 2); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 8192 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({8192, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({8192, 3072})); +} + +void TestSplitRejectsNonIntegralQ8RowBoundary() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3073}, gguf_fixture::kQ8_0); + auto file = builder.Write("bad-row"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->AttentionQkv(0); }), + "blk.0.attn_qkv.weight", "3073", + "Q8_0 whole-row width divisible by 32"); +} + +void TestViewsPointIntoTheReadOnlyMapping() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "mapping"); + const auto first = package->RequireF32("f32", std::array{48}); + const auto second = package->RequireF32("f32", std::array{48}); + TEST_REQUIRE(first.values.data() == second.values.data()); + TEST_REQUIRE(first.values.size() == 48); + + auto misaligned_file = Builder().AddTensor("misaligned-f32", {48}, gguf_fixture::kF32) + .Apply(Mutation::MisalignedF32).Write("misaligned-f32"); + auto misaligned = Phi4GgufPackage::Open(misaligned_file.path); + const auto error = RequireThrows([&] { + misaligned->RequireF32("misaligned-f32", std::array{48}); + }); + RequireMismatch(error, "misaligned-f32", "address", "alignment 4"); +} + +struct ContractFixture { + gguf_fixture::TempFile file; + std::shared_ptr package; + ContractFixture() { + package = Open(Builder().AddFullContractTensors(), file, "contract"); + } +}; + +void TestAcceptsExactPhi3Phi4Contract() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsWrongArchitectureAndEveryDimension() { + struct Case { + std::string field; + gguf_fixture::MetadataValue value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"general.architecture", std::string("llama"), "llama", "phi3"}, + {"phi3.block_count", std::uint32_t{31}, "31", "32"}, + {"phi3.context_length", std::uint32_t{131071}, "131071", "131072"}, + {"phi3.embedding_length", std::uint32_t{3071}, "3071", "3072"}, + {"phi3.feed_forward_length", std::uint32_t{8191}, "8191", "8192"}, + {"phi3.attention.head_count", std::uint32_t{23}, "23", "24"}, + {"phi3.attention.head_count_kv", std::uint32_t{7}, "7", "8"}, + {"phi3.rope.dimension_count", std::uint32_t{95}, "95", "96"}, + {"tokenizer.ggml.tokens", + gguf_fixture::ArrayValue{0, 200063, std::vector(200063)}, + "200063", "200064"}}; + for (const auto& test_case : cases) { + auto file = Builder().SetMetadata(test_case.field, test_case.value) + .AddFullContractTensors().Write("wrong-field"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); + } +} + +void TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole() { + ContractFixture valid; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + valid.package->RequireF32(prefix + ".attn_norm.weight", std::array{3072}); + valid.package->RequireF32(prefix + ".ffn_norm.weight", std::array{3072}); + valid.package->RequireQ8(prefix + ".attn_qkv.weight", std::array{5120, 3072}); + valid.package->RequireQ8(prefix + ".attn_output.weight", std::array{3072, 3072}); + valid.package->RequireQ8(prefix + ".ffn_up.weight", std::array{16384, 3072}); + valid.package->RequireQ8(prefix + ".ffn_down.weight", std::array{3072, 8192}); + } + const nlohmann::json unused; + for (const auto& role : RequiredTensorRoles()) { + auto missing_file = Builder().AddFullContractTensors().RemoveTensor(role.name).Write("missing-role"); + auto missing = Phi4GgufPackage::Open(missing_file.path); + RequireMismatch(RequireThrows([&] { missing->ValidatePhi4Contract(unused, unused, unused); }), + role.name, "missing", "present tensor"); + + const auto wrong_type = role.type == gguf_fixture::kQ8_0 + ? gguf_fixture::kF32 : gguf_fixture::kQ8_0; + auto type_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, wrong_type, role.shape).Write("wrong-type"); + auto type_package = Phi4GgufPackage::Open(type_file.path); + RequireMismatch(RequireThrows([&] { type_package->ValidatePhi4Contract(unused, unused, unused); }), + role.name, wrong_type == gguf_fixture::kF32 ? "F32" : "Q8_0", + role.type == gguf_fixture::kF32 ? "F32" : "Q8_0"); + + auto wrong_shape = role.shape; + --wrong_shape.front(); + auto shape_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, role.type, wrong_shape).Write("wrong-shape"); + auto shape_package = Phi4GgufPackage::Open(shape_file.path); + const auto shape_error = RequireThrows([&] { + shape_package->ValidatePhi4Contract(unused, unused, unused); + }); + RequireMismatch(shape_error, role.name, ShapeText(wrong_shape), ShapeText(role.shape)); + + auto length_file = Builder().AddFullContractTensors() + .TruncateTensorPayload(role.name).Write("wrong-length"); + const auto length_error = RequireThrows([&] { Phi4GgufPackage::Open(length_file.path); }); + RequireMismatch(length_error, role.name + " range", "out-of-file range", + "range within mapped file"); + } +} + +void TestRejectsMixedQuantizationAndOutputWeightPresence() { + auto mixed_file = Builder().AddFullContractTensors().MutateTensor("token_embd.weight", gguf_fixture::kF32, {200064,3072}).Write("mixed"); + auto mixed = Phi4GgufPackage::Open(mixed_file.path); + RequireMismatch(RequireThrows([&] { mixed->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "F32", "Q8_0"); + auto output_file = Builder().AddFullContractTensors().AddTensor("output.weight", {200064,3072}, gguf_fixture::kQ8_0).Write("output-weight"); + auto output = Phi4GgufPackage::Open(output_file.path); + const auto error = RequireThrows([&] { output->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, "output.weight", "present", "absent (tied token_embd.weight)"); +} + +void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { + auto file = Builder().AddFullContractTensors().RemoveTensor("token_embd.weight").Write("untied"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "missing", "present tensor"); +} + +void TestRequiresOriginal4096WindowAndValidatesLongRopeFactors() { + auto wrong_file = Builder().SetMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{8192}).AddFullContractTensors().Write("long-window"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "phi3.rope.scaling.original_context_length", "8192", "4096"); + + auto valid_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); + auto valid = Phi4GgufPackage::Open(valid_file.path); + valid->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); + + auto malformed_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {47}, gguf_fixture::kF32).Write("bad-long-rope"); + auto malformed = Phi4GgufPackage::Open(malformed_file.path); + RequireMismatch(RequireThrows([&] { malformed->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_long.weight", "[47]", "[48]"); +} + +void TestValidatesOptionalShortRopeFactorsAsF32Length48() { + auto absent_file = Builder().AddFullContractTensors(false).Write("no-short-rope"); + auto absent = Phi4GgufPackage::Open(absent_file.path); + absent->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + auto wrong_file = Builder().AddFullContractTensors(false).AddTensor("rope_factors_short.weight", {47}, gguf_fixture::kF32).Write("wrong-short-rope"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_short.weight", "[47]", "[48]"); +} + +void TestRejectsNonFiniteOrNonPositiveRopeValues() { + for (const auto& field : {"phi3.rope.freq_base", "phi3.rope.scaling.attn_factor"}) { + for (const float value : {0.0f, -1.0f, std::numeric_limits::infinity(), std::numeric_limits::quiet_NaN()}) { + auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("bad-rope-value"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + field, std::to_string(static_cast(value)), + "finite positive value"); + } + } +} + +void TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads() { + ContractFixture fixture; + auto config = gguf_fixture::ValidConfig(); + config.erase("head_dim"); + fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsConfigDisagreement() { + ContractFixture fixture; + struct Case { + std::string field; + nlohmann::json value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"model_type", "other", "other", "phi3"}, + {"num_hidden_layers", 31, "31", "32"}, + {"hidden_size", 3071, "3071", "3072"}, + {"intermediate_size", 8191, "8191", "8192"}, + {"num_attention_heads", 23, "23", "24"}, + {"num_key_value_heads", 7, "7", "8"}, + {"head_dim", 127, "127", "128"}, + {"vocab_size", 200063, "200063", "200064"}, + {"rms_norm_eps", 2.0e-5, "2e-05", "0.000010"}, + {"original_max_position_embeddings", 4095, "4095", "4096"}, + {"hidden_size", 3072.0, "3072.0", "integer 3072"}, + {"hidden_size", std::uint64_t{4294970368ULL}, "4294970368", "3072"}, + {"eos_token_id", 199999.0, "199999.0", "integer 199999"}}; + for (const auto& test_case : cases) { + auto config = gguf_fixture::ValidConfig(); + config[test_case.field] = test_case.value; + const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); + } +} + +void TestDerivesStopSetFromGgufConfigAndTokenizerIds() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + + auto gguf_file = Builder().SetMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{1}) + .AddFullContractTensors().Write("wrong-gguf-eos"); + auto gguf = Phi4GgufPackage::Open(gguf_file.path); + RequireMismatch(RequireThrows([&] { gguf->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.eos_token_id", "1", "200020"); + + auto config = gguf_fixture::ValidConfig(); + config["eos_token_id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "eos_token_id", "1", "199999"); + + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"][token] = 1; + for (auto& added : tokenizer["added_tokens"]) + if (added["content"] == token) added["id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + token, "1", std::to_string(expected)); + } +} + +void TestAcceptsPinnedDynamicRoleChatTemplate() { + ContractFixture fixture; + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = + "{% for message in messages %}{{ '<|' + message['role'] + '|>' + " + "message['content'] + '<|end|>' }}{% endfor %}" + "{% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %}"; + fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); +} + +void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { + ContractFixture fixture; + + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"].erase("t0"); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json distinct vocabulary ID count", "200028", "200029"); + + for (const auto& [invalid_id, actual, expected] : std::array{ + std::tuple{-1, "-1", "0..200063"}, + std::tuple{200064, "200064", "0..200063"}, + std::tuple{ + std::numeric_limits::max(), "18446744073709551615", "0..200063"}, + std::tuple{0.0, "0.0", "integer in 0..200063"}}) { + tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"]["t0"] = invalid_id; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json token ID", actual, expected); + } + + tokenizer = gguf_fixture::ValidTokenizer(); + auto& added = tokenizer["added_tokens"]; + const auto highest = std::find_if(added.begin(), added.end(), [](const auto& item) { + return item.at("id") == 200028; + }); + TEST_REQUIRE(highest != added.end()); + added.erase(highest); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json maximum vocabulary ID", "200027", "200028"); + + auto bos_file = Builder().SetMetadata("tokenizer.ggml.add_bos_token", true) + .AddFullContractTensors().Write("wrong-gguf-bos"); + auto bos = Phi4GgufPackage::Open(bos_file.path); + RequireMismatch(RequireThrows([&] { bos->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.add_bos_token", "true", "false"); + + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["add_bos_token"] = true; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "add_bos_token", "true", "false"); + tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = "<|user|><|assistant|>"; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "<|end|>", "missing from chat_template", "present in chat_template"); +} + +void TestRejectsFiniteWrongRmsValue() { + auto file = Builder().SetMetadata("phi3.attention.layer_norm_rms_epsilon", 2.0e-5f) + .AddFullContractTensors().Write("wrong-rms"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, "phi3.attention.layer_norm_rms_epsilon", + "0.000020", "0.000010"); +} + +void TestValidationCreatesNoCorelibObjects() { + fake_corelib::Reset(); + auto api = flm::corelib::CorelibApi::ResolveForTest(fake_corelib::Resolver()); + fake_corelib::GetState().call_counts.clear(); + + ContractFixture fixture; + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "hidden_size", "1", "3072"); + + for (const auto name : {"ryzenai_corelib_create_stream", + "ryzenai_corelib_create_device_tensor", + "ryzenai_corelib_create_tensor_window", + "ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized", + "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized"}) + TEST_REQUIRE(fake_corelib::GetState().call_counts[name] == 0); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + (void)api; +} + +} // namespace + +int main() { +#define RUN(name) RunTest(name, #name) + RUN(TestValidV3HeaderMetadataDirectoryAndAlignment); + RUN(TestOmittedAlignmentUsesGgufDefault32); + RUN(TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely); + RUN(TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail); + RUN(TestCountProductAlignmentAndOffsetOverflowFail); + RUN(TestPresentMalformedAlignmentFails); + RUN(TestDuplicateTensorNamesFail); + RUN(TestOutOfFileAndOverlappingTensorRangesFail); + RUN(TestUnsupportedUnskippableMetadataTypeFails); + RUN(TestRequireQ8AndRequireF32ReportNameActualAndExpected); + RUN(TestAttentionQkvReturnsThreeZeroCopyWholeRowViews); + RUN(TestGateUpReturnsTwoZeroCopyWholeRowViews); + RUN(TestSplitRejectsNonIntegralQ8RowBoundary); + RUN(TestViewsPointIntoTheReadOnlyMapping); + RUN(TestAcceptsExactPhi3Phi4Contract); + RUN(TestRejectsWrongArchitectureAndEveryDimension); + RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); + RUN(TestRejectsMixedQuantizationAndOutputWeightPresence); + RUN(TestRequiresTiedQ8TokenEmbeddingAsLmHead); + RUN(TestRequiresOriginal4096WindowAndValidatesLongRopeFactors); + RUN(TestValidatesOptionalShortRopeFactorsAsF32Length48); + RUN(TestRejectsNonFiniteOrNonPositiveRopeValues); + RUN(TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads); + RUN(TestRejectsConfigDisagreement); + RUN(TestRejectsFiniteWrongRmsValue); + RUN(TestDerivesStopSetFromGgufConfigAndTokenizerIds); + RUN(TestAcceptsPinnedDynamicRoleChatTemplate); + RUN(TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement); + RUN(TestValidationCreatesNoCorelibObjects); +#undef RUN + return 0; +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_host.cpp b/src/test/phi4_corelib_aie4/test_phi4_host.cpp new file mode 100644 index 00000000..a8ba89b3 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_host.cpp @@ -0,0 +1,185 @@ +#include "models/phi4/phi4_corelib_host.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NOMINMAX +#include + +namespace { +using namespace flm::phi4; + +void PutHalf(std::vector& bytes, std::size_t offset, std::uint16_t bits) { + bytes[offset] = static_cast(bits & 0xff); + bytes[offset + 1] = static_cast(bits >> 8); +} + +TensorView ThreeRows() { + static std::vector bytes(3 * 34, std::byte{0x7f}); + std::fill(bytes.begin(), bytes.end(), std::byte{0x7f}); + for (std::size_t row = 0; row < 3; ++row) { + const std::size_t base = row * 34; + PutHalf(bytes, base, row == 0 ? 0x3800 : row == 1 ? 0x3c00 : 0x4000); + for (std::size_t column = 0; column < 32; ++column) { + const auto value = static_cast(row == 1 ? -static_cast(column) : + static_cast(row + column)); + bytes[base + 2 + column] = static_cast(value); + } + } + return {"token_embd.weight", bytes, {3, 32}, 8}; +} + +GgufPhi4Metadata Metadata(double attention = 1.0) { + return {"phi3", 32, 3072, 8192, 24, 8, 131072, 96, 10000.0, + attention, 4096, 200064, false}; +} + +void TestLazyEmbeddingDecodesOnlyRequestedRows() { + constexpr std::size_t width = 65536; + constexpr std::size_t row_bytes = width / 32 * 34; // 17 Windows pages. + auto* mapping = static_cast(VirtualAlloc( + nullptr, 3 * row_bytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); + TEST_REQUIRE(mapping != nullptr); + for (std::size_t block = 0; block < width / 32; ++block) { + const std::uint16_t scale = 0x3c00; + std::memcpy(mapping + row_bytes + block * 34, &scale, sizeof(scale)); + std::fill_n(mapping + row_bytes + block * 34 + 2, 32, std::byte{0xff}); + } + DWORD old_protection{}; + TEST_REQUIRE(VirtualProtect(mapping, row_bytes, PAGE_NOACCESS, &old_protection)); + TEST_REQUIRE(VirtualProtect(mapping + 2 * row_bytes, row_bytes, + PAGE_NOACCESS, &old_protection)); + const TensorView embedding{"token_embd.weight", {mapping, 3 * row_bytes}, + {3, static_cast(width)}, 8}; + const std::array ids{1}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == width); + TEST_REQUIRE(decoded.front() == -1.0f && decoded.back() == -1.0f); + VirtualFree(mapping, 0, MEM_RELEASE); +} + +void TestLazyEmbeddingPreservesRequestOrderAndDuplicates() { + const auto embedding = ThreeRows(); + const std::array ids{2, 0, 2}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == 96); + TEST_REQUIRE(decoded[0] == 4.0f); + TEST_REQUIRE(decoded[32] == 0.0f); + TEST_REQUIRE(decoded[64] == 4.0f); + TEST_REQUIRE(decoded[95] == 66.0f); +} + +void TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds() { + const auto embedding = ThreeRows(); + std::array negative{-1}; + std::array too_large{3}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, negative); }), + "token id"); + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, too_large); }), + "token id"); + auto malformed = embedding; + malformed.bytes = malformed.bytes.first(malformed.bytes.size() - 1); + std::array valid{0}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(malformed, valid); }), + "Q8_0"); +} + +void TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits() { + const std::array input{ + std::bit_cast(0xBE8BBBACu), + std::bit_cast(0xBCCC9DE0u), + std::bit_cast(0xBFED682Fu), + std::bit_cast(0xC2CD01EDu)}; + const std::array scale{1.0f, 1.0f, 1.0f, 1.0f}; + std::array output{}; + HostRmsNorm(input, scale, 1, 4, 1.0e-5f, output); + constexpr std::array expected{ + 0xBBAE75DBu, 0xB9FF7820u, 0xBD143451u, 0xBFFFF50Bu}; + for (std::size_t i = 0; i < output.size(); ++i) + TEST_REQUIRE(std::bit_cast(output[i]) == expected[i]); +} + +void TestHostRmsNormRejectsZeroAndShapeErrors() { + std::array input{1.0f, 2.0f}; + std::array scale{1.0f, 1.0f}; + std::array output{}; + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 0, 2, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 0, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, std::span(scale).first(1), 1, 2, 1.0e-5f, output); }), "shape"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 2, -1.0f, output); }), "epsilon"); +} + +void TestHostRmsNormMatchesPr706Bf16BoundaryReference() { + constexpr std::size_t width = 3072; + std::vector input(width, 0.03125f); + input[0] = 1024.0f; + std::vector scale(width, 1.0f); + std::vector output(width); + HostRmsNorm(input, scale, 1, width, 1.0e-5f, output); + const auto bf16 = ConvertF32ToBf16(output); + TEST_REQUIRE(std::bit_cast(output[0]) == 0x425DB3C3u); + TEST_REQUIRE(std::bit_cast(output[1]) == 0x3ADDB3C3u); + TEST_REQUIRE(bf16[0] == 0x425e); + TEST_REQUIRE(bf16[1] == 0x3ade); +} + +void TestF32ToBf16UsesRoundToNearestEven() { + const std::array values{ + std::bit_cast(std::uint32_t{0x3f808000}), + std::bit_cast(std::uint32_t{0x3f818000}), + -2.5f, + std::numeric_limits::infinity()}; + const auto result = ConvertF32ToBf16(values); + TEST_REQUIRE(result == std::vector({0x3f80, 0x3f82, 0xc020, 0x7f80})); +} + +void TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + constexpr std::size_t i = 47; + constexpr std::size_t p = 4095; + const double inv = 1.0 / std::pow(10000.0, (2.0 * i) / 96.0); + const float expected = static_cast(std::cos(p * inv)); + TEST_REQUIRE(tables.cosine[p * 48 + i] == expected); +} + +void TestRopeTablesApplyShortFactorsAndAttentionFactor() { + std::array factors{}; + factors.fill(2.0f); + FloatTensorView factor_view{"rope_factors_short.weight", factors, {48}}; + const auto tables = BuildShortRopeTables(Metadata(1.5), factor_view); + const double inv = 1.0 / (std::pow(10000.0, 2.0 / 96.0) * 2.0); + TEST_REQUIRE(tables.cosine[48 + 1] == static_cast(std::cos(inv) * 1.5)); + TEST_REQUIRE(tables.sine[48 + 1] == static_cast(std::sin(inv) * 1.5)); +} + +void TestRopeTablesHaveShape4096By48() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + TEST_REQUIRE(tables.cosine.size() == 4096 * 48); + TEST_REQUIRE(tables.sine.size() == 4096 * 48); + TEST_REQUIRE(tables.cosine[0] == 1.0f); + TEST_REQUIRE(tables.sine[0] == 0.0f); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestLazyEmbeddingDecodesOnlyRequestedRows); + RUN_TEST(TestLazyEmbeddingPreservesRequestOrderAndDuplicates); + RUN_TEST(TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds); + RUN_TEST(TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits); + RUN_TEST(TestHostRmsNormRejectsZeroAndShapeErrors); + RUN_TEST(TestHostRmsNormMatchesPr706Bf16BoundaryReference); + RUN_TEST(TestF32ToBf16UsesRoundToNearestEven); + RUN_TEST(TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs); + RUN_TEST(TestRopeTablesApplyShortFactorsAndAttentionFactor); + RUN_TEST(TestRopeTablesHaveShape4096By48); +#undef RUN_TEST +} diff --git a/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp new file mode 100644 index 00000000..72e17aa3 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_phi4_shape_plan.cpp @@ -0,0 +1,106 @@ +#include "models/phi4/phi4_corelib_shape_plan.hpp" +#include "fake_corelib.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::phi4::Phi4ShapePlan; + +std::shared_ptr Api() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +void TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& state = fake_corelib::GetState(); + constexpr std::array buckets{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; + TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * buckets.size() + 1); + TEST_REQUIRE(state.rows_pad_calls.size() == buckets.size()); + TEST_REQUIRE(state.mha_pad_calls.size() == buckets.size()); + for (std::size_t index = 0; index < buckets.size(); ++index) { + TEST_REQUIRE(state.matmul_pad_calls[index * 3].m == buckets[index]); + TEST_REQUIRE(state.matmul_pad_calls[index * 3].group_size == 64); + TEST_REQUIRE(state.rows_pad_calls[index].m == buckets[index]); + TEST_REQUIRE(state.mha_pad_calls[index].m == buckets[index]); + } + TEST_REQUIRE(plan.ForRows(2).query_rows == 64); + TEST_REQUIRE(plan.ForRows(65).query_rows == 128); + TEST_REQUIRE(plan.ForRows(257).query_rows == 512); + TEST_REQUIRE(plan.ForRows(4095).query_rows == 4096); +} + +void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& state = fake_corelib::GetState(); + const auto& q = state.matmul_pad_calls[0]; + const auto& kv = state.matmul_pad_calls[1]; + const auto& output = state.matmul_pad_calls[2]; + TEST_REQUIRE(q.k == 3072 && q.n == 3072); + TEST_REQUIRE(kv.k == 3072 && kv.n == 1024); + TEST_REQUIRE(output.k == 3072 && output.n == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); + TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); + const auto& lm = state.matmul_pad_calls.back(); + TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); + TEST_REQUIRE(plan.lm_head_desc().k == 3072); + TEST_REQUIRE(plan.lm_head_desc().n == 200064); +} + +void TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + const auto& desc = plan.attention_desc(); + TEST_REQUIRE(desc.num_heads == 24); + TEST_REQUIRE(desc.kv_num_heads == 8); + TEST_REQUIRE(desc.head_size == 128); + TEST_REQUIRE(desc.max_seq == 4096); + TEST_REQUIRE(desc.rope_dim == 96); + TEST_REQUIRE(fake_corelib::GetState().mha_pad_calls.front().desc.rope_dim == 96); +} + +void TestShapePlanRejectsPaddedKOrNChanges() { + fake_corelib::Reset(); + fake_corelib::GetState().matmul_k_delta = 1; + RequireContains(RequireThrows([&] { Phi4ShapePlan::Build(Api()); }), "padded K/N"); + fake_corelib::Reset(); + fake_corelib::GetState().matmul_n_delta = 1; + RequireContains(RequireThrows([&] { Phi4ShapePlan::Build(Api()); }), "padded K/N"); +} + +void TestShapePlanRejectsRowsOutsideCachedRange() { + fake_corelib::Reset(); + const auto plan = Phi4ShapePlan::Build(Api()); + RequireContains(RequireThrows([&] { plan.ForRows(0); }), "1..4096"); + RequireContains(RequireThrows([&] { plan.ForRows(4097); }), "1..4096"); +} + +void TestShapePlanFailureNamesHelperAndLogicalShape() { + fake_corelib::Reset(); + auto api = Api(); + fake_corelib::GetState().statuses["ryzenai_corelib_ssmlp_bf16_pad_rows"] = + ryzenai_corelib_status_unsupported; + const auto error = RequireThrows([&] { Phi4ShapePlan::Build(api); }); + RequireContains(error, "ryzenai_corelib_ssmlp_bf16_pad_rows"); + RequireContains(error, "[1,3072,8192]"); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow); + RUN_TEST(TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions); + RUN_TEST(TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96); + RUN_TEST(TestShapePlanRejectsPaddedKOrNChanges); + RUN_TEST(TestShapePlanRejectsRowsOutsideCachedRange); + RUN_TEST(TestShapePlanFailureNamesHelperAndLogicalShape); +#undef RUN_TEST +} diff --git a/src/test/phi4_corelib_aie4/test_real_corelib.cpp b/src/test/phi4_corelib_aie4/test_real_corelib.cpp new file mode 100644 index 00000000..1febec71 --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_real_corelib.cpp @@ -0,0 +1,33 @@ +#include "corelib/corelib_api.hpp" +#include "corelib/corelib_runtime.hpp" +#include "test_support.hpp" + +#include +#include +#include + +int main() { + const char* configured = std::getenv("FLM_AIE4_CORELIB_PATH"); + if (configured == nullptr || *configured == '\0') { + std::cout << "SKIP: FLM_AIE4_CORELIB_PATH is unset\n"; + return 77; + } + + try { + const auto api = flm::corelib::CorelibApi::Load( + flm::corelib::CorelibApi::ResolveLibraryPath( + std::filesystem::current_path())); + const auto version = api->runtime_version(); + TEST_REQUIRE(version.major == 0 && version.minor == 3 && version.patch == 0); +#define FLM_ASSERT_CORELIB_SYMBOL(member, symbol) TEST_REQUIRE(api->functions().member != nullptr); + FLM_CORELIB_FUNCTIONS(FLM_ASSERT_CORELIB_SYMBOL) +#undef FLM_ASSERT_CORELIB_SYMBOL + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(api); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } + return 0; +} diff --git a/src/test/phi4_corelib_aie4/test_support.hpp b/src/test/phi4_corelib_aie4/test_support.hpp new file mode 100644 index 00000000..1065e6ea --- /dev/null +++ b/src/test/phi4_corelib_aie4/test_support.hpp @@ -0,0 +1,43 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#define TEST_REQUIRE(condition) \ + do { \ + if (!(condition)) { \ + throw std::runtime_error(std::string("requirement failed: ") + \ + #condition); \ + } \ + } while (false) + +inline void RequireContains(std::string_view text, std::string_view expected) { + if (text.find(expected) == std::string_view::npos) { + throw std::runtime_error("expected '" + std::string(text) + + "' to contain '" + std::string(expected) + "'"); + } +} + +template +std::string RequireThrows(Callable&& callable) { + try { + callable(); + } catch (const Exception& error) { + return error.what(); + } + throw std::runtime_error("expected exception was not thrown"); +} + +inline void RunTest(void (*test)(), const char* name) { + try { + test(); + std::cout << "PASS " << name << '\n'; + } catch (const std::exception& error) { + std::cerr << "FAIL " << name << ": " << error.what() << '\n'; + std::exit(1); + } +}