Add optional TwelveLabs Pegasus transcription engine#18
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Add optional TwelveLabs Pegasus transcription engine#18mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Add an opt-in cloud transcription backend using the TwelveLabs Pegasus model alongside the default local Whisper.cpp engine. Select it from the new Engine dropdown in Optional Settings; the default behavior is unchanged. The backend returns the same result shape as transcribe_audio, so SRT export, diarization, and display all work unchanged. API key is read from the TWELVELABS_API_KEY env var or the settings field.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
This PR adds an opt-in cloud transcription engine powered by the TwelveLabs Pegasus video-understanding model, alongside the existing local Whisper.cpp engine.
What it adds
whisper.cpp(default) andtwelvelabs-pegasus.twelvelabs_backend.pymodule that uploads the selected media as a TwelveLabs asset, runs Pegasus analysis, and returns the exact same result dict shape astranscribe_audio()(raw/text/segments/audio_length/stderr/cancelled). Because of this, SRT export, diarization, and the transcript display all work unchanged.TWELVELABS_API_KEYenvironment variable), persisted inconfig.jsonlike the other settings.Why it helps
Whisper.cpp is great for local, offline transcription. Pegasus gives users an optional cloud path that handles long videos and runs without needing a local model or GPU — useful when the machine can't run the larger Whisper models comfortably.
Opt-in / non-breaking
The default engine stays
whisper.cppand that code path is untouched. The TwelveLabs code only runs when a user explicitly selects the new engine. The dependency (twelvelabs>=1.2.8) is only imported lazily inside the backend, so existing installs are unaffected until they opt in.How it was tested
python -m unittest test_twelvelabs_backend— offline unit tests for the timestamp/segment parsers, result-shape contract, and API-key resolution (added in this PR; they run without a key).TWELVELABS_API_KEY, skipped otherwise) confirming the client constructs and the analyze/assets API is wired correctly.VideoContext_AssetId+analyze); a full end-to-end Pegasus run is slow due to server-side video processing.py_compileon all changed files.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.