From d486dc272bb97b83fadc05b77aa3627f275bac5f Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Fri, 14 Aug 2026 22:56:11 -0700 Subject: [PATCH 1/4] Clarify model export and runtime ownership Document that Mobius emits faithful artifacts without gating on downstream ORT GenAI capabilities. Strengthen cache, quantization, provider isolation, and optional runtime validation guidance while preserving direct GGUF defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- .agents/skills/adding-a-new-model/SKILL.md | 3 +- .agents/skills/diffusion-models/SKILL.md | 5 +- .agents/skills/multimodal-models/SKILL.md | 7 ++- .../skills/onnx-export-quantization/SKILL.md | 18 +++++- .agents/skills/ort-genai-config/SKILL.md | 22 +++++-- .agents/skills/quality-checklist/SKILL.md | 59 +++++++++++-------- .agents/skills/writing-tests/SKILL.md | 14 +++++ 7 files changed, 88 insertions(+), 40 deletions(-) diff --git a/.agents/skills/adding-a-new-model/SKILL.md b/.agents/skills/adding-a-new-model/SKILL.md index 9ec1c8887..0069c07c0 100644 --- a/.agents/skills/adding-a-new-model/SKILL.md +++ b/.agents/skills/adding-a-new-model/SKILL.md @@ -286,7 +286,8 @@ see the [quality-checklist skill](../quality-checklist/SKILL.md). - [ ] Integration test in `tests/integration_test.py` (if small checkpoint available) - [ ] L4 golden file generated and committed (`testdata/golden/`) - [ ] L5 generation golden file generated and committed -- [ ] ORT GenAI test added to `tests/ort_genai_test.py` (text-generation and VLM models) +- [ ] Graph-derived ORT GenAI metadata is tested; downstream load/generation is + optional evidence and never an export capability gate - [ ] CLI build works (`mobius build --model ...`) - [ ] Multi-dtype correctness verified (fp32, fp16, bf16) - [ ] Pinned revision reaches every Hub/processor/weight/golden call diff --git a/.agents/skills/diffusion-models/SKILL.md b/.agents/skills/diffusion-models/SKILL.md index 959de5499..1246d68de 100644 --- a/.agents/skills/diffusion-models/SKILL.md +++ b/.agents/skills/diffusion-models/SKILL.md @@ -363,8 +363,9 @@ Before advertising runtime support, execute the complete path: Verify sample/output ranks, source-vs-target token slicing, CFG semantics, and the runtime's actual scheduler identifiers/equations. Reload final metadata and verify every file path, port, preprocessing step, loop edge, and postprocess. -If the runtime cannot express the dataflow, reject it before emitting artifacts; -keep direct component export separate. +Emit faithful component graphs and metadata even when the tested downstream +runtime cannot yet express the complete dataflow. Record that runtime/version +limitation separately; it must not become a Mobius export capability gate. Validate required VAE statistics against latent channel count before graph construction (`len(mean) == len(std) == z_dim`, positive standard deviations). diff --git a/.agents/skills/multimodal-models/SKILL.md b/.agents/skills/multimodal-models/SKILL.md index 156dff435..d3beed9e2 100644 --- a/.agents/skills/multimodal-models/SKILL.md +++ b/.agents/skills/multimodal-models/SKILL.md @@ -255,9 +255,10 @@ must scale with rows/windows, not total-patches × media-count. Keep dense masks only as a portable fallback and assert optimized CUDA/DML graphs lack the quadratic construction. -Config generation is not runtime support. Execute real media through ORT GenAI; -if it cannot supply a required encoder input or position-ID rank, reject export -before writing artifacts and use the same evidence for any Foundry waiver. +Config generation is not runtime support. Emit graph-derived metadata faithfully +and, when useful, execute real media through ORT GenAI as a downstream probe. +If it cannot supply a required encoder input or position-ID rank, record the +exact runtime/version limitation without blocking Mobius export. ## GQA for KV-shared layers (Gemma4) diff --git a/.agents/skills/onnx-export-quantization/SKILL.md b/.agents/skills/onnx-export-quantization/SKILL.md index 5d533aaab..dad952719 100644 --- a/.agents/skills/onnx-export-quantization/SKILL.md +++ b/.agents/skills/onnx-export-quantization/SKILL.md @@ -242,6 +242,16 @@ pip install cupy-cuda12x # Olive auto-detects cupy and uses GPU when available ``` +### Isolate Olive from unrelated provider DLLs + +Olive 0.13 may auto-register every provider DLL bundled in an ORT GPU wheel +even when a weight-only pass explicitly targets CPU. A missing TensorRT DLL can +then abort K-quant before the pass starts. Keep the accelerator CPU-only and, +for programmatic workflows, suppress `olive.systems.local` EP-library +registration around `olive.workflows.run`; restore it immediately afterward. +This is safe for `OnnxKQuantQuantization`, which does not create an inference +session. Still load and execute the resulting package with the intended EP. + ### Quantizing multi-model exports Quantize each sub-model independently. Typically only the decoder is @@ -425,9 +435,9 @@ compare_golden( ) ``` -### L5: End-to-end smoke test +### Optional ORT GenAI downstream smoke test -Run inference with the quantized model through ORT GenAI: +When useful, run inference with the quantized model through ORT GenAI: ```python import onnxruntime_genai as og @@ -442,6 +452,10 @@ output_ids = model.generate(params) print(tokenizer.decode(output_ids[0])) ``` +ORT GenAI acceptance is not a Mobius export gate. Always validate the final +quantized ONNX package directly; treat ORT GenAI load/generation as optional +downstream evidence and record its version/outcome without blocking export. + ### Numerical parity verification Quantized models will have some numerical divergence from the diff --git a/.agents/skills/ort-genai-config/SKILL.md b/.agents/skills/ort-genai-config/SKILL.md index cb94def19..62d0bf6b3 100644 --- a/.agents/skills/ort-genai-config/SKILL.md +++ b/.agents/skills/ort-genai-config/SKILL.md @@ -282,12 +282,22 @@ processor doesn't provide. Either: 1. Compute them externally and inject via NamedTensors, or 2. Modify the vision model to compute them from `image_grid_thw` internally -### Config writes successfully but runtime cannot execute - -Config/schema success is not runtime support. Run load plus generation through -the exported contract. If the runtime cannot route required feature inputs, -position-ID rank, cache state, scheduler, or multimodal metadata, reject export -before writing artifacts and report the exact runtime version/limitation. +### Config metadata and downstream runtime acceptance + +Mobius owns metadata correctness, not ORT GenAI capability decisions. Emit the +best accurate package from graph metadata: filenames, semantic graph inputs and +outputs, every cache template the current config schema can represent, and the +global cache-slot count. Preserve intrinsic schema/config validation, but do +not gate or reject export based on the current GenAI model registry, runtime +version, topology support, or cache executor capability. + +Runtime load/generation is an optional downstream probe. Record its exact +version and outcome, but a downstream limitation never blocks Mobius export or +requires a model-specific capability guard. For example, NemotronH mixes sparse +key/value, convolution, and SSM state. Emit key/value and convolution templates +plus the graph-derived slot count; the current schema has no `ssm_state` +template, so omit only that unrepresentable field and let ORT GenAI decide +whether it can execute the package now or in a future release. ### "input_ids size exceeds max length" diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 9f5b6453e..65c26dbe9 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -3,10 +3,10 @@ name: quality-checklist description: > Use this skill when verifying that a new model is truly done and ready to merge. Provides a Definition-of-Done checklist covering all five test - confidence levels (L1 graph build through L5 Foundry Local smoke-test), - ORT GenAI runtime validation, Olive quantization compatibility, multi-dtype - (f32/f16/bf16) and multi-EP (CPU/CUDA/DML) correctness, documentation - requirements, and code review criteria. + confidence levels (L1 graph build through L5 generation), faithful runtime + metadata, optional downstream runtime probes, Olive quantization + compatibility, multi-dtype (f32/f16/bf16) and multi-EP (CPU/CUDA/DML) + correctness, documentation requirements, and code review criteria. --- # Skill: Quality Checklist @@ -14,8 +14,8 @@ description: > ## When to use Use this checklist before marking a new model addition as **done**. -Every item must be checked — or explicitly waived with a written reason — -before the PR is merged. +Every required item must be checked — or explicitly waived with a written +reason — before the PR is merged. --- @@ -162,27 +162,33 @@ python examples/_text_generation.py --compare-hf --dtype bf16 - [ ] Final fusion claims use loaded weights/constants; no-weight graphs cannot prove initializer folding -### 9. ORT GenAI runtime +### 9. ORT GenAI metadata and downstream runtime -- [ ] Model can be loaded with `ort_genai.Model(output_dir)` without error -- [ ] Actual generation with required media/features produces coherent output; - schema/config emission alone is not runtime support -- [ ] ORT GenAI test added to `tests/ort_genai_test.py` - (or confirmed covered by an existing parametrized test) -- [ ] Structurally unsupported contracts fail before artifacts are emitted and - have a version-specific, evidence-based waiver +- [ ] Generated metadata faithfully reflects graph filenames, semantic I/O, + representable cache templates, and global cache-slot indices +- [ ] Intrinsic schema/config errors are tested without inferring downstream + runtime capability -Run the ORT GenAI integration test: +Optional downstream evidence: + +- [ ] If ORT GenAI load/generation is run, record the exact runtime version and + result. Failures document limitations but never gate Mobius export based + on the runtime registry, topology support, or cache executor capability. +- [ ] Add an ORT GenAI integration test when useful, but do not require one for + export acceptance. + +Optional ORT GenAI integration probe: ```bash python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv ``` -### 10. Foundry Local smoke test +### 10. Foundry Local package check -- [ ] Model exported package can be loaded and run in Foundry Local -- [ ] At minimum, verify that the `genai_config.json` and all ONNX files - are present and the model responds to a short prompt +- [ ] `genai_config.json` and all ONNX files are present and internally + consistent +- [ ] If Foundry Local is available, record its version and load/generation + result as optional downstream evidence; limitations do not block export > If Foundry Local is not available in the current environment, document the > skip with a `# TODO: verify with Foundry Local` comment in the PR. @@ -194,8 +200,9 @@ python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv - [ ] Quantized model produces non-degenerate output (coherent text) - [ ] Quantization uses only required execution providers if unrelated provider registration fails, and evidence includes size, load, and inference -- [ ] If quantization changes the graph structure (e.g. MatMulNBits), verify - the `genai_config.json` still loads correctly in ORT GenAI +- [ ] If quantization changes graph names (e.g. `logits_Q4`), regenerate or + verify metadata against the final graph; ORT GenAI loading remains an + optional downstream probe Run the quantization integration test suite to confirm existing patterns are not broken: @@ -223,8 +230,8 @@ novel weight layouts (e.g. fused QKV, non-standard expert routing). after optimization or rebase changes - [ ] Rebase linearly onto `origin/main`; resolve shared registries/helpers semantically, run shared-surface tests, and push with `--force-with-lease` -- [ ] After rebase, rerun config generation and actual runtime load/generation; - parity-only tests do not catch changed cache/runtime contracts +- [ ] After rebase, rerun metadata generation and intrinsic config validation. + If downstream runtime evidence is included, rerun that exact probe too. - [ ] Confirm the remote PR head SHA, mergeability, replacement lint, and architecture checks after the final push - [ ] Triage red CI at check/job/test granularity against the exact base SHA; @@ -236,17 +243,17 @@ novel weight layouts (e.g. fused QKV, non-standard expert routing). ## Waiver policy -Any item that cannot be completed must be waived explicitly in the PR +Any required item that cannot be completed must be waived explicitly in the PR description: ``` **Waivers:** - L5 golden: Model is 70B — generating golden data exceeds CI resources. skip_reason added to YAML. -- Foundry Local: Not available in this environment. Tracked in issue #NNN. ``` -Unchecked items without a waiver are grounds to request changes before merge. +Unchecked required items without a waiver are grounds to request changes +before merge. Optional downstream probes need no waiver. --- diff --git a/.agents/skills/writing-tests/SKILL.md b/.agents/skills/writing-tests/SKILL.md index 2f92dd4f0..073e542b4 100644 --- a/.agents/skills/writing-tests/SKILL.md +++ b/.agents/skills/writing-tests/SKILL.md @@ -266,6 +266,20 @@ integration test alongside any new custom function or Scan op.** runner termination from a model assertion failure. - Reference goldens must come from an independently invoked upstream pipeline, never from the implementation under test or ad-hoc intermediate features. +- When the checkpoint is too large, HTTP-range-read safetensors headers and the + exact tensors for a production-dimension reduced fixture. Cover every layer + family and cache contract, record the source layer/row derivation, and create + L4/L5 goldens from the independently invoked HuggingFace model. This is + reduced real-weight evidence, not a claim of full-checkpoint parity. +- Before accepting an architecture xfail, verify config vocabulary, epsilon, + and layer-kind translation. A stale `mlp`->`moe` mapping can look like an SSM + numerical failure while loading the wrong weights entirely. +- Compare full prefill and every token-by-token prompt/decode logit on the + target EP. Fused cache kernels can match multi-token prefill yet diverge on + the first reused-state step. If full-precision and quantized packages fail at + the same reused-state step, suspect source cache semantics rather than Olive. + Test the equivalent standard-ONNX cache graph before assigning blame; prefer + the portable graph when it restores the numeric gate. ### Recurrent state ≠ KV cache From 0aa8ccac2e33ed60d12a988cd7c40e8548635b19 Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Fri, 14 Aug 2026 23:21:50 -0700 Subject: [PATCH 2/4] Resolve optional runtime guidance contradictions Require direct ONNX Runtime validation for GGUF imports while keeping downstream GenAI and Foundry probes optional. Clarify reduced-fixture and metadata wording from review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- .agents/skills/onnx-export-quantization/SKILL.md | 7 +++++-- .agents/skills/ort-genai-config/SKILL.md | 2 +- .agents/skills/quality-checklist/SKILL.md | 3 --- .agents/skills/writing-tests/SKILL.md | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/.agents/skills/onnx-export-quantization/SKILL.md b/.agents/skills/onnx-export-quantization/SKILL.md index dad952719..132293f14 100644 --- a/.agents/skills/onnx-export-quantization/SKILL.md +++ b/.agents/skills/onnx-export-quantization/SKILL.md @@ -151,8 +151,11 @@ Before implementing or claiming a direct conversion: pinned upstream tokenizer. A self-contained package is invalid when padding, EOS, or BOS semantics disagree. 7. Require real-weight full-logit parity and deterministic multi-token - generation through both ONNX Runtime and the declared GenAI runtime. Graph, - config, and session creation are not acceptance evidence. + generation directly through ONNX Runtime. Graph, config, and session creation + alone are not acceptance evidence. +8. Optionally run the same generation through the declared GenAI runtime as + downstream evidence. Record its version and outcome, but do not gate direct + GGUF artifacts on its acceptance. Use Hub GGUF architecture metadata to fail before downloading multi-gigabyte unsupported files, then repeat the guard from the local GGUF header so local diff --git a/.agents/skills/ort-genai-config/SKILL.md b/.agents/skills/ort-genai-config/SKILL.md index 62d0bf6b3..31f713ef7 100644 --- a/.agents/skills/ort-genai-config/SKILL.md +++ b/.agents/skills/ort-genai-config/SKILL.md @@ -285,7 +285,7 @@ processor doesn't provide. Either: ### Config metadata and downstream runtime acceptance Mobius owns metadata correctness, not ORT GenAI capability decisions. Emit the -best accurate package from graph metadata: filenames, semantic graph inputs and +most accurate package from graph metadata: filenames, semantic graph inputs and outputs, every cache template the current config schema can represent, and the global cache-slot count. Preserve intrinsic schema/config validation, but do not gate or reject export based on the current GenAI model registry, runtime diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 65c26dbe9..8681a18bf 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -190,9 +190,6 @@ python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv - [ ] If Foundry Local is available, record its version and load/generation result as optional downstream evidence; limitations do not block export -> If Foundry Local is not available in the current environment, document the -> skip with a `# TODO: verify with Foundry Local` comment in the PR. - ### 11. Olive quantization compatibility - [ ] Model can be loaded from the exported ONNX package by Olive diff --git a/.agents/skills/writing-tests/SKILL.md b/.agents/skills/writing-tests/SKILL.md index 073e542b4..ab7ca84d1 100644 --- a/.agents/skills/writing-tests/SKILL.md +++ b/.agents/skills/writing-tests/SKILL.md @@ -269,8 +269,8 @@ integration test alongside any new custom function or Scan op.** - When the checkpoint is too large, HTTP-range-read safetensors headers and the exact tensors for a production-dimension reduced fixture. Cover every layer family and cache contract, record the source layer/row derivation, and create - L4/L5 goldens from the independently invoked HuggingFace model. This is - reduced real-weight evidence, not a claim of full-checkpoint parity. + L4/L5 goldens from the independently invoked HuggingFace model. Treat this as + reduced real-weight evidence, not as a claim of full-checkpoint parity. - Before accepting an architecture xfail, verify config vocabulary, epsilon, and layer-kind translation. A stale `mlp`->`moe` mapping can look like an SSM numerical failure while loading the wrong weights entirely. From e496d93938dae33657ecfafa16da0f9d01ac5c95 Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Fri, 14 Aug 2026 23:25:28 -0700 Subject: [PATCH 3/4] Keep ORT GenAI cache probes optional Move dual-head-dimension runtime configuration guidance into optional downstream evidence and state that omitted probes require no waiver or TODO. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu --- .agents/skills/quality-checklist/SKILL.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 8681a18bf..8240d5fb1 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -148,12 +148,6 @@ python examples/_text_generation.py --compare-hf --dtype bf16 opt-in via `MOBIUS_ORT_LOWER_OPSET_FOR_EP=1`) - [ ] Dead graph inputs removed after EP-aware optimization (`RemoveDeadGraphInputsPass` in Stage 4 of `optimize_model()`) -- [ ] For models with dual head_dim (e.g. Gemma4: 256 for sliding, - 512 for full attention), verify whether ORT GenAI requires - `search.past_present_share_buffer=false` because GenAI allocates - uniform KV cache shapes; if so, explicitly override the generated - `genai_config.json` before runtime validation rather than assuming - the default generated setting is correct - [ ] Vision/audio graph inputs match the real processor (normally float32); reduced-precision encoders cast once at graph entry - [ ] Representative graph evidence covers raw/post-Mobius/post-weight and @@ -174,9 +168,15 @@ Optional downstream evidence: - [ ] If ORT GenAI load/generation is run, record the exact runtime version and result. Failures document limitations but never gate Mobius export based on the runtime registry, topology support, or cache executor capability. +- [ ] If ORT GenAI validation is run for a model with dual head dimensions, + determine whether that runtime requires + `search.past_present_share_buffer=false` for its uniform KV-cache + allocation, and override the generated config only for that probe. - [ ] Add an ORT GenAI integration test when useful, but do not require one for export acceptance. +Omitting these downstream probes requires no waiver or TODO. + Optional ORT GenAI integration probe: ```bash From 87e57e763636ddfa2af06a0d51e416eaa15c1f59 Mon Sep 17 00:00:00 2001 From: Justin Chu Date: Sat, 15 Aug 2026 11:58:18 -0700 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Justin Chu Signed-off-by: Justin Chu --- .agents/skills/ort-genai-config/SKILL.md | 8 -------- .agents/skills/quality-checklist/SKILL.md | 15 ++++++++------- 2 files changed, 8 insertions(+), 15 deletions(-) diff --git a/.agents/skills/ort-genai-config/SKILL.md b/.agents/skills/ort-genai-config/SKILL.md index 31f713ef7..6ab46aa10 100644 --- a/.agents/skills/ort-genai-config/SKILL.md +++ b/.agents/skills/ort-genai-config/SKILL.md @@ -291,14 +291,6 @@ global cache-slot count. Preserve intrinsic schema/config validation, but do not gate or reject export based on the current GenAI model registry, runtime version, topology support, or cache executor capability. -Runtime load/generation is an optional downstream probe. Record its exact -version and outcome, but a downstream limitation never blocks Mobius export or -requires a model-specific capability guard. For example, NemotronH mixes sparse -key/value, convolution, and SSM state. Emit key/value and convolution templates -plus the graph-derived slot count; the current schema has no `ssm_state` -template, so omit only that unrepresentable field and let ORT GenAI decide -whether it can execute the package now or in a future release. - ### "input_ids size exceeds max length" For image prompts, the tokenized input_ids (including image_pad tokens) can diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 8240d5fb1..f539eb275 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -163,7 +163,7 @@ python examples/_text_generation.py --compare-hf --dtype bf16 - [ ] Intrinsic schema/config errors are tested without inferring downstream runtime capability -Optional downstream evidence: +Downstream evidence: - [ ] If ORT GenAI load/generation is run, record the exact runtime version and result. Failures document limitations but never gate Mobius export based @@ -177,7 +177,10 @@ Optional downstream evidence: Omitting these downstream probes requires no waiver or TODO. -Optional ORT GenAI integration probe: + +Waiver needed if any of the steps are not possible. + +Run the ORT GenAI integration test: ```bash python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv @@ -197,9 +200,8 @@ python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv - [ ] Quantized model produces non-degenerate output (coherent text) - [ ] Quantization uses only required execution providers if unrelated provider registration fails, and evidence includes size, load, and inference -- [ ] If quantization changes graph names (e.g. `logits_Q4`), regenerate or - verify metadata against the final graph; ORT GenAI loading remains an - optional downstream probe +- [ ] If quantization changes the graph structure (e.g. MatMulNBits), verify + the `genai_config.json` still loads correctly in ORT GenAI Run the quantization integration test suite to confirm existing patterns are not broken: @@ -249,8 +251,7 @@ description: skip_reason added to YAML. ``` -Unchecked required items without a waiver are grounds to request changes -before merge. Optional downstream probes need no waiver. +Unchecked items without a waiver are grounds to request changes before merge. ---