Skip to content

feat(screen-recording): record through simulator-server instead of host ffmpeg - #587

Draft
latekvo wants to merge 4 commits into
mainfrom
feat/screen-recording-simserver
Draft

feat(screen-recording): record through simulator-server instead of host ffmpeg#587
latekvo wants to merge 4 commits into
mainfrom
feat/screen-recording-simserver

Conversation

@latekvo

@latekvo latekvo commented Jul 29, 2026

Copy link
Copy Markdown
Member

Moves screen recording off the host ffmpeg pipeline and onto simulator-server's own recorder.

Blocked on software-mansion/radon#155, which adds the endpoint and enables it in the argent build variant. Until a release carries it, every simulator-server answers 404 and this branch behaves exactly like main — that fallback is deliberate and covered below, but the win only lands once #155 ships.

Why

simulator-server already encodes every frame of the device screen for its live stream, with the touch overlay drawn in. Recording subscribed to that MJPEG stream, decoded it on the host, paced it in Node and re-encoded it through an ffmpeg child. So a recording cost two encodes with a lossy JPEG round-trip in between, and required an external binary the user had to install.

It now asks simulator-server to record instead: it paces to 30fps, trims static stretches, stamps the watermark and muxes the mp4 from the frames it already holds.

What stays identical

Everything the tools present. Same admission guards (double start, start-after-cap, concurrent stop), same per-call reminder note, same outputFile returned by start, same durationMs/wallClockMs/trimmedMs, same warnings, same .argent/recordings artifact with the argent- prefix stripped. The touch visualizer is still argent's to toggle for the life of a recording, since it is server-global state.

Two details worth calling out:

  • The video is copied out of simulator-server's session directory rather than referenced there. That directory is wiped when the server exits, and the artifact is materialized by the client afterwards — possibly downloaded over argent link much later.
  • The time limit is enforced by the server, and a host timer mirrors it purely as bookkeeping: it flips the reminder to "ended, still to retrieve" and makes a start-after-cap fail instead of discarding the finished video. It must not stop the recording — that is screen-recording-stop's job, whenever it arrives.

The fallback

The endpoint exists only where simulator-server can encode: the software-encoder builds compile the routes out, and no simulator-server predating radon#155 has them either — including the one argent currently pins. Those answer 404 and startCapture runs the existing host pipeline unchanged. That path is untouched here, and ffmpeg is required exactly there.

Which one you get is a property of the build, not of the host OS, so the tool description and SKILL.md say that rather than naming platforms. They would otherwise be wrong today in the most visible direction: the shipped macOS simulator-server has no recording route, so macOS falls back and does need ffmpeg.

Detecting the absent route needed care: it is answered by the router's unmatched-route fallback with an empty body, which the shared simulatorPost helper would surface as "non-JSON response" — indistinguishable from a server in a bad state. recordingPost tests the status and the emptiness. A 404 carrying a body came from a handler, so the route exists and refused the command; falling back there would start a second capture over a recording that is already running. Failed recording commands are the other case: HTTP 200 carrying an error field.

Found in review

Each reproduced before it was changed, and each pinned by a test that fails without the fix.

  • A start suspended at pointer.enable() resumed past a dispose that had already stopped the server's recording, reported {status: "recording"} for it, and armed a cap timer no later stop or dispose could clear. The start request's own await was guarded; the pointer await, the one suspension point after the session is stamped, was not.
  • serverStop stayed set across the stop request, so a dispose landing mid-stop issued a second, concurrent stop for the recording that call was already finalizing — and that stop takes down the simulator-server whose session directory the copy reads from.
  • A failed copy lost the video silently. It threw a raw filesystem error naming two temp paths, with no failure signal, while the only finished video sat in simulator-server's session directory waiting to be deleted with it. It now names that path while it is still fetchable, classified like the empty-output failure two calls down.
  • The recording commands had no request timeout. A simulator-server that accepts the connection and never answers held startPending/stopPending — and so every other recording call on the device — for undici's 300 s header timeout. Measured at 301011 ms per call, twice per recording; the pointer toggles five lines away were already bounded at 2 s.
  • The 404 fallback keyed on the status alone while its own comment described an empty body. Verified against the shipped macOS simulator-server, which has no recording route: POST /api/recording/start answers 404 with content-length: 0.
  • res.json().catch(() => null) reported a body that never finished arriving, and any non-JSON reply, as "simulator-server rejected the recording command" — sending the caller after a refusal reason that was never sent. Split out the way simulatorPost already splits it.
  • The ffmpeg prose was keyed on the host OS while the code keys on the route being present, and the two disagree today (above).

Verification

3095 tool-server tests pass (295 files), plus tsc for src and tsconfig.test.json, eslint and prettier.

25 new tests in screen-recording.test.ts (62 → 87). Each of these mutations turns the suite red, so none of them is passing on incidental behaviour:

mutation tests failing
server path never chosen 10
stop dispatches to the host path 8
dispose leaves the server recording running 1
post-dispose start race guard removed 1
post-pointer dispose guard removed 1
serverStop left set across the stop await 1
copy failure throws a plain Error 1
recording commands sent without a timeout 1
any 404 falls back 1
any empty-bodied non-2xx falls back 1
back to res.json().catch(() => null) 2
trim fields always reported 1
server's copy of the video not released 1
cap timer stops the recording early 1
ambiguous start reply treated as success 1

End-to-end on an iOS 18.6 simulator, tool-server driving a real simulator-server (native framebuffer 1179×2556):

Server path (--features argent build of radon#155): start → appearance flip → gesture-tap → 5s static → stop.

  • durationMs: 2500, wallClockMs: 9908, trimmedMs: 7408 — trimming engaged
  • mp4: h264, 1178×2556 (even-cropped), r_frame_rate=30/1, 75 frames / duration 2.500000 — exactly the reported durationMs
  • zero mentions of ffmpeg in the tool-server log; no encoder was spawned
  • watermark legible in an extracted frame, contrast-matched to the UI underneath
  • the tap's translucent bubble is present in the video (frame 31), so the touch overlay still lands
  • simulator-server's media directory is empty afterwards — the video was moved out, not left behind
  • the artifact carries saveDir: ".argent/recordings" and the prefix-stripped filename

Fallback path (today's shipped simulator-server-argent-macos, strings … api/recording = 0): same tool calls, start fell through the 404 to the host pipeline → h264 1178×2556, 30/1, 176 frames, duration=5.866667 matching durationMs: 5867.

latekvo added 4 commits July 29, 2026 18:28
…st ffmpeg

simulator-server already encodes every frame of the device screen for its
live stream, with the touch overlay drawn in. Recording re-decoded that
MJPEG stream on the host, paced it in Node and re-encoded it through an
ffmpeg child — two encodes, a JPEG round-trip in between, and an external
binary users had to install.

`screen-recording-start` now asks simulator-server to record instead. It
paces, trims and watermarks the frames it already has and hands back a
muxed mp4 at stop, which is copied out of its session directory (that
directory dies with the server, and the artifact is materialized by the
client afterwards). Everything the tools present is unchanged: the same
admission guards, the same reminder note, the same `outputFile` from start,
the same durations, warnings and `.argent/recordings` artifact.

The endpoint only exists where simulator-server can encode, so a build
without it answers 404 and the host pipeline in `capture.ts` runs exactly as
before — which is what its Linux and Windows builds do, and they are the
only place `ffmpeg` is still required.
…ng back

`startServerRecording` treated any non-error reply as a started recording,
diverging from its sibling `pointerPost`, which tests for `status: "ok"`.
An unrecognized reply leaves it unknown whether a recording is running, and
falling back to the host pipeline there would capture the screen twice and
strand the server's copy with nothing left to stop it.
Nothing outside the start tool binds simulator-server's recording endpoints,
and the wire functions it wraps are tested directly.
…-handling gaps

Six issues found reviewing the simulator-server recording path, each
reproduced before it was changed and each pinned by a test that fails
without the fix.

Lifecycle races in server-capture:

- A start suspended at `pointer.enable()` resumed past a dispose that had
  already stopped the server's recording, reported `{status: "recording"}`
  for it, and armed a cap timer no later stop or dispose could clear. The
  start request's own await was already guarded; the pointer await was not.
- `serverStop` stayed set across the stop request, so a dispose landing
  mid-stop issued a second, concurrent stop for the recording that call was
  already finalizing - and that stop takes down the simulator-server whose
  session directory the copy reads from.
- A copy that fails leaves the only finished video inside simulator-server,
  where the session directory takes it when that server exits. It threw a
  raw filesystem error naming two temp paths, with no failure signal. It
  now names the server-side path while it is still fetchable, classified
  like the empty-output failure two calls down.

Reply handling in simulator-client:

- `startServerRecording`/`stopServerRecording` had no request timeout, so a
  simulator-server that accepts the connection and never answers held
  `startPending`/`stopPending` - and every other recording call on the
  device - for undici's 300s header timeout. Measured at 301011 ms per
  call. The pointer toggles five lines away were already bounded at 2s.
- The 404 fallback keyed on the status alone while its own comment
  described an empty body. A 404 carrying one came from a handler, so the
  route exists and refused the command: falling back there starts a second
  capture over a recording that is already running. Verified against the
  shipped macOS simulator-server, which has no recording route and answers
  `POST /api/recording/start` with 404 and `content-length: 0`.
- `res.json().catch(() => null)` reported a body that never finished
  arriving, and any non-JSON reply, as "simulator-server rejected the
  recording command" - sending the caller after a refusal reason that was
  never sent. Split out the way `simulatorPost` already splits it.

The ffmpeg prose was keyed on the host OS while the code keys on the route
being present, and the two do not agree today: the shipped macOS
simulator-server has no recording route, so macOS falls back to host
encoding and does need ffmpeg. Both the tool description and SKILL.md now
describe the capability rather than predicting it from the platform.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant