Document and test Orukeet with the existing Hugging Face backend - #1623
Conversation
|
@Nathan-Roll1 Thanks for bringing this up. Looks interesting, can't wait to test :) Tests look fine. Regarding the documentation, I think the best place to mention your model is in https://chidiwilliams.github.io/buzz/docs/faq#4-what-model-should-i-use I try to keep the documentation simple. File import section is about how to use this section, model options are currently covered in the FAQ, but maybe we will need to move them at some point to some more prominent location |
|
Thanks! Moved the short Orukeet note into the FAQ's “What model should I use?” section and removed the File import recipe. Both documentation builds pass. I also added my Oruk affiliation to the PR description. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1623 +/- ##
=======================================
Coverage 83.87% 83.87%
=======================================
Files 115 115
Lines 13605 13605
=======================================
Hits 11411 11411
Misses 2194 2194
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Adds a short Orukeet entry to the FAQ's model-selection section, plus a config-detection regression and an opt-in transcription test for Buzz's existing Hugging Face / Parakeet TDT backend. No application code or dependency changes.
I'm Nathan, the founder of Oruk, and Orukeet is our model.
The Transformers files are now available in the official Hugging Face model repository. Buzz downloads only the six required runtime files; the real
config.jsonfetch participates in Hugging Face's normal download accounting. Cached transcription stays local. The weights are CC BY-SA 4.0 with NVIDIA attribution, and require no custom remote code.Runtime validation at
44f6d30on Apple silicon, CPU/FP32, with both minimum Transformers 5.15.1 and locked 5.16.1 (runtime and tests are unchanged by the documentation follow-up):BUZZ_TEST_DOWNLOAD_MODELS=1, using Buzz's actual downloader and a fresh Hugging Face cache. All six downloaded files match the qualified export hashes.The published source-model evaluation reports 9.85% pooled FLEURS WER versus 11.01% for Parakeet v3 over 20,146 recordings in 25 languages. That is a NeMo result, not a Buzz benchmark. Separately, the Transformers export qualification matched 117/120 normalized source transcripts; its small compatibility sample measured 5.43% WER versus 5.34% for the source model (two additional word errors). No CUDA performance claim is made.
The FAQ notes the existing backend's limits: automatic language detection, no translation/prompts/word timings, and 30-second subtitle boundaries. Full GUI and CUDA validation were not run. Reviewed and tested with Codex assistance.
Documentation validation at
7d5f068: English and Chinese Docusaurus production builds passed. The generated English FAQ contains the entry under ‘What model should I use?’ with the model link, selection steps, limits and license; the File import page has no duplicate recipe.git diff --checkpassed. Runtime tests were not repeated for this documentation-only change.