[codex] improve recording and editor workflows#4
Draft
sleel-sun wants to merge 127 commits into
Draft
Conversation
Decode mono 16k audio from the editor video, run Whisper via Transformers.js, and insert linked text annotations with timing and layout helpers. Adds Vite resolve shims for Node-only imports used by the model stack, optional leading-silence trim for the caption buffer, timeline gap reconciliation for auto-caption regions, and editor i18n. Raises Select content z-index so caption controls stay usable over the video surface. Co-authored-by: Cursor <cursoragent@cursor.com>
- Gate trimSec segment shift when transcription used full buffer retry - Restore UTF-8 autoCaptions strings (ar, es, fr, ja-JP, ko-KR, tr, zh-CN, zh-TW) - Dedupe caption segments after grouping only; stricter chunk dedupe in transcribe - Post-truncate duration, explicit consume merge for web demuxer, trim region shift cleanup - Tests for caption annotation pipeline Co-authored-by: Cursor <cursoragent@cursor.com>
Phrase mode with one merged span now splits wrapCaptionTextByWordBounds lines into separate CaptionSegments with even time allocation across the phrase span (fallback when duration is too short for min spans). Update unit test accordingly. Co-authored-by: Cursor <cursoragent@cursor.com>
Merge siddharthvaddem/openscreen main and resolve conflicts in VideoEditor.tsx (export diagnostics, save-diagnostic handler, auto captions, toolbar chrome) and package-lock.json. Apply Biome formatting after merge so hooks pass. Co-authored-by: Cursor <cursoragent@cursor.com>
Use one Sonner id for the full caption flow, show a distinct transcribing step, yield before Whisper so updates paint, match editor dark chrome on toasts, and keep pointer-events only on toast bodies. Add transcribing strings across locales. Co-authored-by: Cursor <cursoragent@cursor.com>
Scope withoutNodeVersion to Transformers import/pipeline only; reapply trim-region filtering after ignore-trims retries; check AbortSignal after each slice inference before chunk processing. Co-authored-by: Cursor <cursoragent@cursor.com>
- Add RowWaveform component (refactored from TrimWaveform) that decodes audio via AudioContext, caches peaks per URL, and renders a faint waveform canvas behind any timeline row using useTimelineContext() for zoom/pan alignment - Wire RowWaveform as background of the trim row in TimelineEditor - Add showTrimWaveform boolean to EditorState / ProjectEditorState with default true; persists across saves/loads - Add Timeline Settings accordion section in SettingsPanel (always visible) with a Waveform toggle to enable/disable the visualization - Bump hint text opacity from 0.12 to 0.25 so "Press T to add trim" remains readable over the waveform background Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add "Timeline" as a dedicated SettingsPanelMode with AudioWaveform icon - Waveform toggle now lives exclusively in the Timeline panel (not in every panel) - Default showTrimWaveform to false so trim row starts with visible hint text - Guard AccordionItem render with activePanelMode === "timeline" to prevent bleed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace AudioWaveform icon with Brackets on the Timeline panel tab and section header - Rename toggle label from "Waveform" to "Show Audio Waveform on Trim Track" - Make toggle full-width to accommodate the longer label Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 0.25 bump was a workaround for hint visibility over the waveform background, but is now stale — waveform defaults to off and hints only appear on empty rows anyway. Restores consistency with all other row hints. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move fetch/decode/cache logic out of RowWaveform into a standalone useAudioPeaks hook. RowWaveform is now a pure canvas renderer that accepts pre-computed peaks as a prop. TimelineEditor calls the hook and passes peaks down, skipping decoding entirely when waveform is off. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…loading Add a single exported loadFileAsArrayBuffer in streamingDecoder.ts that owns the IPC-vs-fetch branching for local and remote video URLs. StreamingVideoDecoder.loadSourceFile and useAudioPeaks both delegate to it, removing the duplicated readBinaryFile/fetch logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e with useRef - Move computePeaks into audioPeaksWorker.ts — runs off the main thread with zero-copy channel buffer transfer - Replace module-level peaksCache Map with a useRef scoped to the hook instance — no global mutable state, toggle off/on still instant Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add timeline.title and timeline.waveform keys to every settings.json locale file. Replace all hardcoded "Timeline" and "Show Audio Waveform on Trim Track" strings in SettingsPanel with t() calls. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Rename to BackgroundWaveform to reflect its visual role rather than structural placement. Remove the redundant wrapper div — Row already provides relative overflow-hidden, so the canvas can sit absolute inset-0 directly and observe itself via ResizeObserver. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Reset peaks to null immediately when an uncached URL is set, so the previous video's waveform is not shown during decode. Also clear on decode failure so stale data does not persist indefinitely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use 剪輯 (editing/clip) instead of 修剪 (trimming) to match the term already used in the trim section throughout zh-TW strings. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move the peaks null-check after clearRect so a previously drawn waveform is erased when peaks transitions to null (e.g. on URL change or decode failure) rather than lingering on screen. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add docstrings to all undocumented functions and methods across the files touched in this branch: getAudioCtx, Row component, and all StreamingVideoDecoder methods (loadSourceFile, computeSegments, getExportMetrics, splitBySpeed, getDemuxer, cancel, destroy, withTimeout). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- computePeaksInWorker now accepts an AbortSignal; terminates the worker immediately on abort and rejects with AbortError so orphaned workers cannot outlive the effect cleanup - useAudioPeaks wires an AbortController into the effect; controller.abort() fires alongside the existing cancelled flag on URL change or unmount - loadSourceFile handles data: URLs by parsing the MIME type, deriving a safe filename (e.g. video.mp4), and setting Blob type correctly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lemetry fix: resolve cursor-sampler telemetry not working
…mber-last-project-folder feat: Add projectFolder to user preferences
…p-nix-1.5.0 chore: bump nix package to v1.5.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Notes