Skip to content

Document and test Orukeet with the existing Hugging Face backend - #1623

Merged
raivisdejus merged 2 commits into
chidiwilliams:mainfrom
Nathan-Roll1:codex/orukeet-huggingface
Sep 19, 2026
Merged

raivisdejus merged 2 commits into
chidiwilliams:mainfrom
Nathan-Roll1:codex/orukeet-huggingface

Conversation

@Nathan-Roll1

@Nathan-Roll1 Nathan-Roll1 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Adds a short Orukeet entry to the FAQ's model-selection section, plus a config-detection regression and an opt-in transcription test for Buzz's existing Hugging Face / Parakeet TDT backend. No application code or dependency changes.

I'm Nathan, the founder of Oruk, and Orukeet is our model.

The Transformers files are now available in the official Hugging Face model repository. Buzz downloads only the six required runtime files; the real config.json fetch participates in Hugging Face's normal download accounting. Cached transcription stays local. The weights are CC BY-SA 4.0 with NVIDIA attribution, and require no custom remote code.

Runtime validation at 44f6d30 on Apple silicon, CPU/FP32, with both minimum Transformers 5.15.1 and locked 5.16.1 (runtime and tests are unchanged by the documentation follow-up):

  • 122 model-loader unit tests passed on each version. Seven unrelated model-download cases were deselected.
  • The added integration test passed with BUZZ_TEST_DOWNLOAD_MODELS=1, using Buzz's actual downloader and a fresh Hugging Face cache. All six downloaded files match the qualified export hashes.
  • Buzz's actual transcriber passed repeated English, German, Spanish and French recordings, 0.1/1/5-second silence, and audio spanning two 30-second chunks on both versions, with identical transcripts and chunk boundaries. Cached inference passed with network connections blocked.

The published source-model evaluation reports 9.85% pooled FLEURS WER versus 11.01% for Parakeet v3 over 20,146 recordings in 25 languages. That is a NeMo result, not a Buzz benchmark. Separately, the Transformers export qualification matched 117/120 normalized source transcripts; its small compatibility sample measured 5.43% WER versus 5.34% for the source model (two additional word errors). No CUDA performance claim is made.

The FAQ notes the existing backend's limits: automatic language detection, no translation/prompts/word timings, and 30-second subtitle boundaries. Full GUI and CUDA validation were not run. Reviewed and tested with Codex assistance.

Documentation validation at 7d5f068: English and Chinese Docusaurus production builds passed. The generated English FAQ contains the entry under ‘What model should I use?’ with the model link, selection steps, limits and license; the File import page has no duplicate recipe. git diff --check passed. Runtime tests were not repeated for this documentation-only change.

@raivisdejus

Copy link
Copy Markdown
Collaborator

@Nathan-Roll1 Thanks for bringing this up. Looks interesting, can't wait to test :)

Tests look fine. Regarding the documentation, I think the best place to mention your model is in https://chidiwilliams.github.io/buzz/docs/faq#4-what-model-should-i-use I try to keep the documentation simple. File import section is about how to use this section, model options are currently covered in the FAQ, but maybe we will need to move them at some point to some more prominent location

@Nathan-Roll1

Copy link
Copy Markdown
Contributor Author

Thanks! Moved the short Orukeet note into the FAQ's “What model should I use?” section and removed the File import recipe. Both documentation builds pass. I also added my Oruk affiliation to the PR description.

@raivisdejus
raivisdejus enabled auto-merge (squash) September 19, 2026 12:56
@codecov

codecov Bot commented Sep 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.87%. Comparing base (7d1929c) to head (7d5f068).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1623   +/-   ##
=======================================
  Coverage   83.87%   83.87%           
=======================================
  Files         115      115           
  Lines       13605    13605           
=======================================
  Hits        11411    11411           
  Misses       2194     2194           
Flag Coverage Δ
Linux 83.13% <ø> (+<0.01%) ⬆️
macOS 83.61% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@raivisdejus
raivisdejus merged commit 81fdc26 into chidiwilliams:main Sep 19, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants