feat(screen-recording): record through simulator-server instead of host ffmpeg - #587
Draft
latekvo wants to merge 4 commits into
Draft
feat(screen-recording): record through simulator-server instead of host ffmpeg#587latekvo wants to merge 4 commits into
latekvo wants to merge 4 commits into
Conversation
…st ffmpeg simulator-server already encodes every frame of the device screen for its live stream, with the touch overlay drawn in. Recording re-decoded that MJPEG stream on the host, paced it in Node and re-encoded it through an ffmpeg child — two encodes, a JPEG round-trip in between, and an external binary users had to install. `screen-recording-start` now asks simulator-server to record instead. It paces, trims and watermarks the frames it already has and hands back a muxed mp4 at stop, which is copied out of its session directory (that directory dies with the server, and the artifact is materialized by the client afterwards). Everything the tools present is unchanged: the same admission guards, the same reminder note, the same `outputFile` from start, the same durations, warnings and `.argent/recordings` artifact. The endpoint only exists where simulator-server can encode, so a build without it answers 404 and the host pipeline in `capture.ts` runs exactly as before — which is what its Linux and Windows builds do, and they are the only place `ffmpeg` is still required.
…ng back `startServerRecording` treated any non-error reply as a started recording, diverging from its sibling `pointerPost`, which tests for `status: "ok"`. An unrecognized reply leaves it unknown whether a recording is running, and falling back to the host pipeline there would capture the screen twice and strand the server's copy with nothing left to stop it.
Nothing outside the start tool binds simulator-server's recording endpoints, and the wire functions it wraps are tested directly.
…-handling gaps
Six issues found reviewing the simulator-server recording path, each
reproduced before it was changed and each pinned by a test that fails
without the fix.
Lifecycle races in server-capture:
- A start suspended at `pointer.enable()` resumed past a dispose that had
already stopped the server's recording, reported `{status: "recording"}`
for it, and armed a cap timer no later stop or dispose could clear. The
start request's own await was already guarded; the pointer await was not.
- `serverStop` stayed set across the stop request, so a dispose landing
mid-stop issued a second, concurrent stop for the recording that call was
already finalizing - and that stop takes down the simulator-server whose
session directory the copy reads from.
- A copy that fails leaves the only finished video inside simulator-server,
where the session directory takes it when that server exits. It threw a
raw filesystem error naming two temp paths, with no failure signal. It
now names the server-side path while it is still fetchable, classified
like the empty-output failure two calls down.
Reply handling in simulator-client:
- `startServerRecording`/`stopServerRecording` had no request timeout, so a
simulator-server that accepts the connection and never answers held
`startPending`/`stopPending` - and every other recording call on the
device - for undici's 300s header timeout. Measured at 301011 ms per
call. The pointer toggles five lines away were already bounded at 2s.
- The 404 fallback keyed on the status alone while its own comment
described an empty body. A 404 carrying one came from a handler, so the
route exists and refused the command: falling back there starts a second
capture over a recording that is already running. Verified against the
shipped macOS simulator-server, which has no recording route and answers
`POST /api/recording/start` with 404 and `content-length: 0`.
- `res.json().catch(() => null)` reported a body that never finished
arriving, and any non-JSON reply, as "simulator-server rejected the
recording command" - sending the caller after a refusal reason that was
never sent. Split out the way `simulatorPost` already splits it.
The ffmpeg prose was keyed on the host OS while the code keys on the route
being present, and the two do not agree today: the shipped macOS
simulator-server has no recording route, so macOS falls back to host
encoding and does need ffmpeg. Both the tool description and SKILL.md now
describe the capability rather than predicting it from the platform.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves screen recording off the host
ffmpegpipeline and onto simulator-server's own recorder.Why
simulator-server already encodes every frame of the device screen for its live stream, with the touch overlay drawn in. Recording subscribed to that MJPEG stream, decoded it on the host, paced it in Node and re-encoded it through an ffmpeg child. So a recording cost two encodes with a lossy JPEG round-trip in between, and required an external binary the user had to install.
It now asks simulator-server to record instead: it paces to 30fps, trims static stretches, stamps the watermark and muxes the mp4 from the frames it already holds.
What stays identical
Everything the tools present. Same admission guards (double start, start-after-cap, concurrent stop), same per-call reminder note, same
outputFilereturned by start, samedurationMs/wallClockMs/trimmedMs, same warnings, same.argent/recordingsartifact with theargent-prefix stripped. The touch visualizer is still argent's to toggle for the life of a recording, since it is server-global state.Two details worth calling out:
argent linkmuch later.screen-recording-stop's job, whenever it arrives.The fallback
The endpoint exists only where simulator-server can encode: the software-encoder builds compile the routes out, and no simulator-server predating radon#155 has them either — including the one argent currently pins. Those answer 404 and
startCaptureruns the existing host pipeline unchanged. That path is untouched here, andffmpegis required exactly there.Which one you get is a property of the build, not of the host OS, so the tool description and SKILL.md say that rather than naming platforms. They would otherwise be wrong today in the most visible direction: the shipped macOS simulator-server has no recording route, so macOS falls back and does need ffmpeg.
Detecting the absent route needed care: it is answered by the router's unmatched-route fallback with an empty body, which the shared
simulatorPosthelper would surface as "non-JSON response" — indistinguishable from a server in a bad state.recordingPosttests the status and the emptiness. A 404 carrying a body came from a handler, so the route exists and refused the command; falling back there would start a second capture over a recording that is already running. Failed recording commands are the other case: HTTP 200 carrying anerrorfield.Found in review
Each reproduced before it was changed, and each pinned by a test that fails without the fix.
pointer.enable()resumed past a dispose that had already stopped the server's recording, reported{status: "recording"}for it, and armed a cap timer no later stop or dispose could clear. The start request's own await was guarded; the pointer await, the one suspension point after the session is stamped, was not.serverStopstayed set across the stop request, so a dispose landing mid-stop issued a second, concurrent stop for the recording that call was already finalizing — and that stop takes down the simulator-server whose session directory the copy reads from.startPending/stopPending— and so every other recording call on the device — for undici's 300 s header timeout. Measured at 301011 ms per call, twice per recording; the pointer toggles five lines away were already bounded at 2 s.POST /api/recording/startanswers 404 withcontent-length: 0.res.json().catch(() => null)reported a body that never finished arriving, and any non-JSON reply, as "simulator-server rejected the recording command" — sending the caller after a refusal reason that was never sent. Split out the waysimulatorPostalready splits it.Verification
3095 tool-server tests pass (295 files), plus tsc for
srcandtsconfig.test.json, eslint and prettier.25 new tests in
screen-recording.test.ts(62 → 87). Each of these mutations turns the suite red, so none of them is passing on incidental behaviour:serverStopleft set across the stop awaitErrorres.json().catch(() => null)End-to-end on an iOS 18.6 simulator, tool-server driving a real simulator-server (native framebuffer 1179×2556):
Server path (
--features argentbuild of radon#155): start → appearance flip →gesture-tap→ 5s static → stop.durationMs: 2500,wallClockMs: 9908,trimmedMs: 7408— trimming engagedr_frame_rate=30/1, 75 frames / duration 2.500000 — exactly the reporteddurationMssaveDir: ".argent/recordings"and the prefix-stripped filenameFallback path (today's shipped
simulator-server-argent-macos,strings … api/recording= 0): same tool calls, start fell through the 404 to the host pipeline → h264 1178×2556, 30/1, 176 frames,duration=5.866667matchingdurationMs: 5867.