docs: add native C++ export engine architecture plan#678
Conversation
P1-C: Prefer hardware acceleration on all platforms including Windows. Previously Windows tried software first to avoid known driver bugs, but modern hardware encoders (NVENC, VideoToolbox, VAAPI) are 5–10× faster. Software remains the fallback if hardware configure/encode throws. P1-D: Thread encoder latency mode through VideoExporterConfig. Medium and good quality presets now use latencyMode "realtime" which skips encoder lookahead for ~3–5× faster encode throughput. Source quality keeps "quality" mode for maximum compression efficiency. P1-B: Reuse the Pixi TextureSource resource across frames instead of destroying and recreating the GPU texture on every frame. Updates the backing resource and calls source.update() to re-upload, eliminating per-frame GPU alloc/free churn (~5–15 % overhead on long exports).
This reverts commit 959d3e9.
📝 WalkthroughWalkthroughNew draft doc outlines a standalone native C++ export helper process spawned from Electron, replacing the WebCodecs browser pipeline. Specifies hardware-acceleration stack (VideoToolbox/NVENC/AMF/Quick Sync/VAAPI), GPU zero-copy decode/composite/encode stages, JSON IPC contract, phased feature delivery, and performance targets (under ~15s for typical 1080p with acceleration). ChangesNative C++ Export Helper Design Plan
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/export-optimize-native-cpp-plan.md`:
- Around line 121-122: Update the docs to make stdin the canonical input channel
instead of a large JSON command-line argument: replace the line describing Input
as “a single JSON object passed as a command-line argument” with language that
the exporter reads the full JSON job description from stdin (and that argv
should only contain a tiny bootstrap token/flags), and keep Output described as
newline-delimited JSON progress events on stdout (`ready`, `progress`, `done`,
`error`); explicitly call out that large payloads (trim maps, cursor/effects)
must be provided via stdin to avoid argv length limits.
- Around line 46-49: The fenced diagram block showing the process boundary (the
three-line block that starts with "Electron Renderer ──IPC──► Electron Main
──spawn──► openscreen-export-helper") should include a language tag (e.g., use
```text) to satisfy markdownlint MD040; update that fenced code block to begin
with a language identifier like "text" so the diagram is properly tagged.
- Around line 123-126: Define and implement an explicit cancellation/error
contract for the native helper invoked by NativeExporter: ensure that on SIGTERM
or any encoding/muxing error the helper stops the pipeline, cleans up temporary
artifacts (partial files, temp dirs), and exits non‑zero; only on successful
completion write output to a temp path and publish the final file via an atomic
rename/move into the target path. Update NativeExporter (the TypeScript wrapper
around VideoExporterConfig → helper JSON) to pass a temp-output path, listen for
helper exit codes/events, delete temp artifacts on non‑zero/error/SIGTERM, and
only surface success callbacks (and call onProgress finalization) after the
atomic rename; keep VideoExporter (WebCodecs) as the documented fallback for
systems where the helper binary is unavailable.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 48382e63-d821-4b99-84b9-82c4ce9d2e50
📒 Files selected for processing (1)
docs/export-optimize-native-cpp-plan.md
| ``` | ||
| Electron Renderer ──IPC──► Electron Main ──spawn──► openscreen-export-helper | ||
| (React / UI) (Node.js) (C++ encode engine) | ||
| ``` |
There was a problem hiding this comment.
Nit: add a language tag to fenced diagram block.
markdownlint MD040 is valid here; use something like ```text for the process-boundary diagram. nit, but cleaner CI/docs hygiene.
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 46-46: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/export-optimize-native-cpp-plan.md` around lines 46 - 49, The fenced
diagram block showing the process boundary (the three-line block that starts
with "Electron Renderer ──IPC──► Electron Main ──spawn──►
openscreen-export-helper") should include a language tag (e.g., use ```text) to
satisfy markdownlint MD040; update that fenced code block to begin with a
language identifier like "text" so the diagram is properly tagged.
| - **Input**: a single JSON object passed as a command-line argument describing the full export job (paths, effects, quality, trim, cursor data, etc.) | ||
| - **Output**: newline-delimited JSON progress events on stdout (`ready`, `progress`, `done`, `error`) |
There was a problem hiding this comment.
Input transport is inconsistent and lowkey risky at scale.
Doc says “stdin/stdout JSON contract” but then defines input as a JSON command-line arg. For large jobs (trim maps, cursor/effects payloads), argv length limits (especially on Windows) can fail export startup. recommend making stdin the canonical input channel and keeping argv to a tiny bootstrap token only.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/export-optimize-native-cpp-plan.md` around lines 121 - 122, Update the
docs to make stdin the canonical input channel instead of a large JSON
command-line argument: replace the line describing Input as “a single JSON
object passed as a command-line argument” with language that the exporter reads
the full JSON job description from stdin (and that argv should only contain a
tiny bootstrap token/flags), and keep Output described as newline-delimited JSON
progress events on stdout (`ready`, `progress`, `done`, `error`); explicitly
call out that large payloads (trim maps, cursor/effects) must be provided via
stdin to avoid argv length limits.
| - **Cancellation**: SIGTERM | ||
|
|
||
| The JS side (`NativeExporter`) wraps this in a thin TypeScript class that translates the existing `VideoExporterConfig` into the helper's JSON format and maps progress events back to the `onProgress` callback. The `VideoExporter` (WebCodecs) remains as a fallback for systems where the helper binary is unavailable. | ||
|
|
There was a problem hiding this comment.
Cancellation contract needs explicit cleanup + atomic output semantics.
SIGTERM-only is kinda under-specified for long-running encode/mux. Please define required behavior on cancel/error: stop pipeline, delete temp artifacts, and only publish output via atomic rename on success. otherwise users can end up with corrupt/partial files that look “done-ish”.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/export-optimize-native-cpp-plan.md` around lines 123 - 126, Define and
implement an explicit cancellation/error contract for the native helper invoked
by NativeExporter: ensure that on SIGTERM or any encoding/muxing error the
helper stops the pipeline, cleans up temporary artifacts (partial files, temp
dirs), and exits non‑zero; only on successful completion write output to a temp
path and publish the final file via an atomic rename/move into the target path.
Update NativeExporter (the TypeScript wrapper around VideoExporterConfig →
helper JSON) to pass a temp-output path, listen for helper exit codes/events,
delete temp artifacts on non‑zero/error/SIGTERM, and only surface success
callbacks (and call onProgress finalization) after the atomic rename; keep
VideoExporter (WebCodecs) as the documented fallback for systems where the
helper binary is unavailable.
| - **Background pre-render cache.** While the user scrubs the timeline and previews effects, CapCut silently renders affected segments into a low-bitrate segment cache. When export is triggered, cached segments are assembled directly without re-rendering — reducing the export to a mux-and-encode pass over pre-computed frames. | ||
| - **On-demand decode.** Only frames that survive trim boundaries are decoded. The demuxer seeks to the nearest keyframe before each active segment and skips the rest at the packet level, so a 10-minute source with a 30-second active region decodes approximately 30 seconds of video, not 10 minutes. | ||
|
|
||
| The architecture proposed here applies the first and third strategies directly. The second (segment pre-render cache) is a longer-term addition that can layer on top once the core C++ pipeline is in place. |
There was a problem hiding this comment.
Why C++? I'm not against it, I see no argument in favor or against it.
| The helper selects the best available backend at runtime, in priority order: | ||
|
|
||
| | Platform | Decode | Composite | Encode | | ||
| |---|---|---|---| | ||
| | macOS (Apple Silicon) | VideoToolbox | Metal compute | VideoToolbox (H.264 / HEVC) | | ||
| | macOS (Intel) | VideoToolbox | Metal compute | VideoToolbox | | ||
| | Windows (NVIDIA) | NVDEC | D3D11 compute | NVENC | | ||
| | Windows (AMD) | AMF decoder | D3D11 compute | AMF encoder | | ||
| | Windows (Intel) | Quick Sync | D3D11 compute | Quick Sync | | ||
| | Linux | VAAPI / NVDEC | OpenGL compute | VAAPI / NVENC | | ||
| | All (fallback) | FFmpeg software | CPU | libx264 / libx265 | | ||
|
|
||
| The critical optimisation at each stage is **GPU zero-copy**: the decoded frame lives on a GPU surface, the compositor reads and writes GPU textures, and the encoder consumes the GPU surface directly — no pixel data crosses the CPU bus until the final muxed file is written to disk. |
There was a problem hiding this comment.
Why not using an abstraction like libavcodec library from FFmpeg project? that already maintain all these backend layer.
There was a problem hiding this comment.
the C++ helper does use FFmpeg (libavcodec / libavformat / libswscale). FFmpeg is the abstraction layer over every hardware backend listed in this document — VideoToolbox, NVENC, AMF, Quick Sync, VAAPI. The question is not whether to use FFmpeg, but where to run it.
There are two realistic ways to run FFmpeg-backed code from an Electron app:
Option A — WebAssembly (ffmpeg.wasm). Compile FFmpeg to WASM and run it inside the renderer or main process. This is a real project and works for simple transcodes. The problem is that WASM runs inside the browser's sandbox, which means it cannot open a VideoToolbox session, cannot acquire an NVENC encoder context, cannot use D3D11VA or VAAPI, and cannot share GPU surfaces with the compositor. Every frame would be a CPU copy. WASM is also single-threaded by default; SharedArrayBuffer threads are available but cannot call native OS APIs. For a pipeline whose entire performance argument is GPU zero-copy and hardware encode, WASM erases the benefit entirely.
Option B — Native process or addon. Compile FFmpeg into a native binary (or N-API addon) that runs outside the browser sandbox with full OS API access. This is what this document proposes, and it is exactly how CapCut, DaVinci Resolve, and every other professional desktop video tool works.
So the stack is: C++ process → libavcodec (FFmpeg) → platform HW API (VideoToolbox / NVENC / VAAPI). FFmpeg is not an alternative to this plan; it is a core dependency of it. The C++ layer exists specifically to host FFmpeg outside the sandbox where it can actually reach the hardware.
2 participants
Summary
It's a blueprint and a draft about the export optimization. Most optimization methods are just industry-standard, safe, conservative plays. They’re fine for quick, low-risk iterations.
Adds
docs/export-optimize-native-cpp-plan.md, a design document for a native C++ export engine intended to replace the current WebCodecs-based pipeline as the primary export path.Welcome to improve it.
What's in the doc
openscreen-export-helperC++ binary following the same child-process pattern as the existingopenscreen-screencapturekit-helperandopenscreen-wgc-capture-helper. The helper owns the full decode → GPU composite → HW encode → mux pipeline while the renderer stays untouched.