ort_genai: native processor configs for gemma-4-12B unified - #391
Merged
Merged
Conversation
The encoder-free gemma-4-12B "unified" model can now be preprocessed natively by ORT GenAI (no HuggingFace processor / set_inputs needed), so stop skipping its processor artifacts and emit the correct configs: * model.type: gemma4_unified now maps to the dedicated "gemma4_unified" ORT GenAI multimodal type (was reusing "gemma4"). * image_processor.json: DecodeImage -> Gemma4ImageTransform configured for the 48px merged-patch contract (patch_size=48, pooling_kernel_size=1, patch_dim=6912). HF builds these via 16px patchify + 3x3 patches_merge, which is identical to a direct 48px patchify, so the existing op is reused with merged geometry. * audio_feature_extraction.json: AudioDecoder -> Gemma4UnifiedAudioFrames (raw 640-sample waveform framing) instead of the 128-dim log-mel op. Requires the companion onnxruntime-extensions (Gemma4UnifiedAudioFrames) and onnxruntime-genai (gemma4_unified processor) support. Updates the unified auto_export tests to assert the emitted configs. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Updates Mobius’ ORT GenAI auto-export integration to emit native processor configs for the encoder-free Gemma-4-12B “unified” model, aligning exported artifacts with newly available ORT GenAI + ort-extensions preprocessing support.
Changes:
- Map HF
model_type="gemma4_unified"to ORT GenAImodel.type="gemma4_unified"(instead of reusinggemma4). - Emit
image_processor.jsonfor unified models usingDecodeImage → Gemma4ImageTransformwithpatch_size=48andpooling_kernel_size=1. - Emit
audio_feature_extraction.jsonfor unified models usingAudioDecoder → Gemma4UnifiedAudioFrames(raw 640-sample framing), and update tests to assert these outputs.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/mobius/integrations/ort_genai/auto_export.py | Updates ORT GenAI model-type mapping and writes unified-specific image/audio processor configs. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Adjusts expectations and adds assertions for unified processor config emission and attributes. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Follow the onnxruntime-extensions consolidation: both gemma4 audio configs now use the single Gemma4Audio op with an explicit type attribute instead of two separate op types. * gemma4 / gemma4_text -> Gemma4Audio type="log_mel" (128-dim USM log-mel) * gemma4_unified* -> Gemma4Audio type="raw_frames" (raw 640-sample frames) Updates the auto_export audio-config tests accordingly. 115 tests pass. Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…-processor # Conflicts: # src/mobius/integrations/ort_genai/auto_export.py
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
src/mobius/integrations/ort_genai/auto_export.py:783
- PR description (and the earlier module-level comment) refer to a
Gemma4UnifiedAudioFramesort-extensions op for unified audio framing, but the emitted processor config here usesGemma4Audiowithattrs.type="raw_frames". If ORT GenAI / ort-extensions expects the dedicatedGemma4UnifiedAudioFramesop name, this config will fail to load at runtime; ifGemma4Audio(raw_frames)is the intended API, the PR description (and any related docs) should be updated for consistency.
"operation": {
"name": "gemma4_audio",
"type": "Gemma4Audio",
"attrs": {
"type": "raw_frames",
"audio_samples_per_token": samples_per_token,
"sampling_rate": 16000,
"padding_value": 0.0,
},
src/mobius/integrations/ort_genai/auto_export.py:109
- The comment describing the unified Gemma4 audio preprocessing mentions a
Gemma4UnifiedAudioFramesop, but this PR’s emitted audio processor config uses theGemma4Audioop withattrs.type="raw_frames"(and there are no other references toGemma4UnifiedAudioFramesin the repo). This comment should be updated to match the actual emitted config to avoid misleading future readers.
This issue also appears on line 775 of the same file.
# These are produced natively by ort-extensions: ``Gemma4ImageTransform`` with
# patch_size=48 / pooling_kernel_size=1 (the 48px merged patch is identical to a
# direct 48px patchify) and the ``Gemma4UnifiedAudioFrames`` raw-framing op. The
# genai ``gemma4_unified`` processor consumes both (see the companion
# onnxruntime-genai / onnxruntime-extensions support).
Contributor
|
@copilot resolve the merge conflicts in this pull request |
rui-ren
approved these changes
Sep 15, 2026
…-processor Signed-off-by: GitHub <noreply@github.com> # Conflicts: # src/mobius/integrations/ort_genai/auto_export.py # src/mobius/integrations/ort_genai/auto_export_test.py Co-authored-by: rui-ren <15321482+rui-ren@users.noreply.github.com>
Contributor
Merged
|
Lock the exported vision and audio graph ABI, processor attributes, runtime model type, semantic input mappings, and multimodal token metadata to the Extensions and GenAI consumer contracts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e64e522-b07f-4c1c-abf1-3985268beb7e Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
|
|
Bring the PR onto origin/main at 8c1bf54 without rebasing while preserving the native Gemma4 unified preprocessing contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e64e522-b07f-4c1c-abf1-3985268beb7e Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
For the encoder-free gemma-4-12B "unified" model, mobius deliberately did not emit
image_processor.jsonoraudio_feature_extraction.json, and itsgenai_config.jsonmappedmodel.typetogemma4. The unified model's inputs (48px merged pixel patches / raw 640-sample waveform frames) had no native ORT GenAI preprocessing, so the only way to run it was HuggingFaceAutoProcessor+Generator.set_inputs. The emittedgenai_configreferenced processor files that were never written (a dangling reference; harmless forset_inputsbut broken forMultiModalProcessor).Change
With native ORT GenAI support now available (companion PRs below), emit the correct processor configs:
model.type:gemma4_unified→ dedicatedgemma4_unifiedORT GenAI type (wasgemma4).image_processor.json:DecodeImage → Gemma4ImageTransformatpatch_size=48, pooling_kernel_size=1(patch_dim 6912). HF produces the merged patches via 16px patchify + 3×3patches_merge, which is provably identical to a direct 48px patchify, so the existing op is reused with merged geometry.audio_feature_extraction.json:AudioDecoder → Gemma4UnifiedAudioFrames(raw 640-sample framing) instead of the 128-dim log-mel op.The text-only unified backbone (
gemma4_unified_text) still resolves togemma4_text(unchanged).Dependencies
Gemma4UnifiedAudioFrames; image reuse)gemma4_unifiedprocessor + model type)Testing
auto_export_test.py+genai_config_test.py: 115 passed. Updated the three unified tests to assert the emitted configs (48/1 image transform, raw-frame audio op,gemma4_unifiedtype).ruff,ruff format, andmypyclean on the changed source.