refactor(audio): canonical audio service with chat/API/tool/trigger wrappers#403
Open
esafwan wants to merge 27 commits into
Open
refactor(audio): canonical audio service with chat/API/tool/trigger wrappers#403esafwan wants to merge 27 commits into
esafwan wants to merge 27 commits into
Conversation
…_chat Create huf/ai/audio_service.py as the single backend home for audio: - save_audio_upload(): base64 audio upload with guardrails (25MB max, extension/MIME allowlist, strict base64 validation, clear errors) - resolve_stt_config(): STT resolution (explicit model > Agent.stt_model > provider default), now preferring AI Model records with the Transcription modality over the hardcoded provider default map - transcribe_audio_file(): pure transcription (LiteLLM transcription, Gemini/Google/Vertex multimodal adapter preserved) with no Agent Message creation and no socket events - create_audio_user_message(): Agent Message (kind=Audio, voice_message, stt_model) creation/upsert plus conversation counters and socket event sdk_tools.handle_transcribe_audio keeps its signature and return shape (text key preserved) and is now a thin wrapper: transcription via the service, message/socket side effects only when conversation_id is set. _resolve_stt_config/_get_default_stt_model remain as delegating aliases. agent_chat upload_audio_and_transcribe(_web) keep their whitelisted signatures and response shapes, delegating upload/STT/message work to the audio service.
- huf.ai.audio_api.transcribe: clean whitelisted endpoint (no guest access) accepting file_id OR b64data+filename, with agent, language, model override, and opt-in create_message against a conversation - huf/tests/test_audio_service.py: standalone unit tests (frappe and LiteLLM mocked, no bench or provider calls) covering upload guardrails, resolve_stt_config priority, transcribe side-effect-free behavior, and API input validation
…t runs Audio files attached via Agent Trigger were routed to OCR and never transcribed. Route them to audio_service.transcribe_audio_file and append transcripts to the run prompt; errors are logged and skipped.
…ption naming - speech-input: 180s default recording cap (maxDurationSeconds prop) - ChatInput: clear toasts on failed/empty transcription; accept audio files in the attachment picker - Label stt_model as 'Audio Transcription Model'; display Transcription and legacy 'Speech to Text' tool types as 'Audio Transcription'
…ging Found via live verification on a real site: - Gemini/Google/Vertex adapter now sends audio as an input_audio content block (the previous image_url hack fails in current LiteLLM) - Google-family providers default to gemini/gemini-2.5-flash; unknown providers now raise a clear error instead of falling back to whisper-1 - log_error calls use keyword args so long provider errors go to the message body, not the 140-char title column (DataError 1406)
Audio attachments in React chat were rejected by the OCR/Vision upload gate. Audio files are now allowed whenever the agent has a resolvable transcription config, and prepare/process endpoints transcribe them via the canonical audio service instead of OCR, stamping voice_message and stt_model on the message. Also fixes a gettext shadowing bug and a cross-suite double-import in the audio test mocks.
…odal TriggerDocEventExtras existed but was never rendered, so Doc Event triggers could not be configured with file/audio field attachments from the React UI. Wire it into TriggerModal for Doc Event triggers and pass prompt_field/file_attachments through the agent trigger API.
audio_api.transcribe and the transcribe_audio tool accept a file_path pointing at a server filesystem audio file that is not a Frappe File. Paths are realpath-resolved and must live inside sites/<site>/private/ audio_imports or a site-config audio_import_dirs root (symlink/traversal safe); file_path use requires System Manager. import_local_audio turns a drop into a Frappe File when message creation is requested.
-Introduced a new preferServerStt prop to prioritize server-side STT via MediaRecorder over the local browser dictation API. -Added a local visual badge and descriptive tool tips to explicitly indicate when the app falls back to unaudited browser dictation.
-Updated the SpeechInput usage inside ChatInput to pass preferServerStt={true}.
-Ensured that user voice dictations from the chat interface are processed server-side whenever possible for auditing and cost control.
-Added support for rendering voice messages originating from users within the ChatMessage component. -Implemented an AudioPlayer UI with playback controls to listen to the recorded user voice messages. -Added UI elements to display the extracted transcript, STT model used, and processing status.
-Updated the conversation item mapper to include sttModel and status properties. -Ensured that the raw audio metadata is properly normalized into the chat message format.
-Extended the MessageType interface to include sttModel and status properties. -Enabled tracking of the audio processing state directly within the frontend message types.
-Added clear UI notices to the OpenAI and Groq settings tabs. -Advised users that provider-specific transcription settings are deprecated in favor of using the global Agent STT Model configuration.
-Updated the AgentMessageDoc and ChatMessage interfaces to include stt_model and status fields. -Modified the database query payload in getDocList to fetch these new audio metadata fields for frontend consumption.
-Added logic to capture and persist the stt_model link and processing status (Completed or Failed) to the newly created Agent Message document.
-Updated the API transcribe function to automatically stamp the agent message with a "Completed" status upon successful transcription.
-Modified create_audio_user_message to accept and persist a status metadata field.
…tence -Refactored handle_transcribe_audio to act strictly as a pure SDK tool without causing database side effects. -Removed logic that previously created Agent Message records directly from within the transcription tool. -Added transcript, local_path, and provider keys to the returned tool response dictionary.
-Deprecated execute_provider_capability and handle_speech_to_text dispatchers in transcription_handler.py. -Added runtime warnings directing developers to use the unified canonical huf.ai.audio_service instead.
-Removed legacy transcribe_audio method from the GroqSettings DocType. -Deferred Groq audio transcription routing to the unified canonical audio service.
-Removed legacy transcribe_audio and text_to_speech methods from the OpenAISettings DocType. -Deferred all audio capability routing to the unified canonical audio service.
…cord - Updated upload_audio_and_transcribe_web and prepare_message_with_file_web to instantly assign the Audio kind and inject the transcript directly into the content field. - Passed skip_user_message=True during legacy API orchestration to prevent duplicated text messages. - Standardized session_id mapping natively for all audio-related database operations. - Enhanced payload returns to directly feed optimistic UI updates.
- Added support for is_audio, transcript, voice_message, and stt_model to PrepareMessageWithFileResponse. - Added file_url to TranscribeAudioResponse to support immediate UI rendering of recorded media players.
…audio elements - Forced skipUserMessage: true inside runAgentAndUpdateAssistant when firing audio payloads to prevent dual-insertion in the backend. - Upgraded the file-upload and microphone-recording flows to dynamically swap the initial userMessage UI state to kind: 'Audio' with the newly attached voiceMessage URL. - Dropped generic attachment cards for audio uploads in favor of the native audio player hook.
- Wrapped the transcribed text inside an accessible container, hiding the transcription visually by default under an expandable Transcript toggle. - Removed raw metadata text clusters from displaying below the audio player for a much cleaner UX.
Sanjusha-tridz
marked this pull request as ready for review
July 17, 2026 05:17
shzdb
approved these changes
Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unifies HUF's fragmented audio/STT support into one canonical backend service, per the refactor-audio track plan.
huf/ai/audio_service.py(new): canonical service —save_audio_upload(25MB cap, MIME/extension allowlist, strict base64),resolve_stt_config(explicit model >Agent.stt_model> provider default, prefersAI Model.modalities = Transcription),transcribe_audio_file(pure STT, no message/socket side effects; Gemini/Vertex multimodal adapter preserved),create_audio_user_message,is_audio_file.huf/ai/audio_api.py(new): clean whitelistedtranscribeAPI (file_idorb64data+filename;create_messageopt-in; no guest).sdk_tools.handle_transcribe_audio,agent_chat.upload_audio_and_transcribe(_web)keep signatures/response shapes, delegate to the service.stt_modelrelabeled "Audio Transcription Model" in Agent → Advanced.Audio entry paths
huf.ai.audio_api.transcribetranscribe(file_path=...)/ toolfile_pathparam — System Manager only, restricted toprivate/audio_imports+ site-configaudio_import_dirs, traversal/symlink safeTests
huf/tests/test_audio_service.py(25) andhuf/tests/test_doc_event_audio.py(10) — all pass.yarn typecheckpasses; no new lint issues.Track docs:
workspace/Tracks/refactor-audio/(PLAN, FINDINGS).