feat(screen-recording): record sim-remote devices over the MoQ video track - #552
Draft
latekvo wants to merge 20 commits into
Draft
feat(screen-recording): record sim-remote devices over the MoQ video track#552latekvo wants to merge 20 commits into
latekvo wants to merge 20 commits into
Conversation
… stop reminders Adds a screen-recording tool pair following the native-profiler session pattern: a per-device ScreenRecordingSession blueprint owns the capture child (simctl io recordVideo on iOS simulators, on-device screenrecord + adb pull on Android), with a time-limit cap, unexpected-exit recovery, and dispose-on-shutdown cleanup. While a recording is live (or ended on its own but was never retrieved), every tool result carries a note reminding the agent to call screen-recording-stop - riding the same note field the update reminder uses, but per-call and unsuppressed. Ships an argent-screen-recording skill (plus routing stanza) that tells the agent to schedule its own wakeup for the expected end of recording right after starting one.
…yable, fix durationMs Review findings from the adversarial pass on #517: - startPending/stopPending flags close the check-then-stamp TOCTOU that let two overlapping starts (or stops) on one device both pass the session guards and cross-corrupt shared state; cap timers and exit handlers now carry an ownership guard so a superseded capture's callbacks can't touch the newer capture's session. - Android: the start timeout is disarmed once READY:<pid> is parsed (a PID landing near the deadline no longer fails as a bogus timeout); a failed start best-effort reaps the device-side screenrecord and its file; a start that supersedes a finalized-but-unretrieved capture removes the stale /sdcard mp4 instead of leaking it (dispose too); a clean exit long before the cap is classified as unexpected (legacy adb always reports 0); a failed pull keeps the session retrievable so stop can be retried instead of stranding the video; pull timeout raised to 10 min for wireless adb. - durationMs now reports capture length (recording end stamp), not wall clock since start, when stop arrives after the cap fired. - Reminder note carries a shared-server caveat so foreign agents don't stop captures they don't own; SKILL.md remote-simulator prefix corrected to remote:.
…rsede rm, reap mid-start children on dispose Second adversarial pass on #517: - assertStoppableSession now rejects while a start is mid-readiness (startPending): a recovery stop admitted in that window used to wipe the superseding capture's freshly stamped session, leaving its recorder unmanageable (no cap SIGINT source, invisible to dispose). - The Android supersede rm of a finalized-but-unretrieved capture's /sdcard file now runs only AFTER the new capture is live, so a failed start stays side-effect-free and the previous video remains retrievable. - iOS start's pending flag moved into a wrapper try/finally (matching the Android shape) so a synchronous spawn throw can no longer wedge the session behind a stuck startPending. - New pendingChild session field: dispose() reaps a child whose start is still mid-readiness at shutdown (captureProcess is stamped success-only, so dispose couldn't see it). Plus regression tests for each, an Android durationMs-after-cap test, and an Android double-start-in-flight test.
…ed on-device cleanup retrievable Two findings from adversarial E2E on #517: - A start suspended at a pre-spawn await could spawn a capture AFTER dispose() already ran at shutdown, orphaning a device-side recorder (Android screenrecord running to its cap / iOS recordVideo) that the pendingChild reaping can no longer see. dispose() now sets a `disposed` flag before any await, and both start paths check it synchronously immediately before spawn (no await between the check and the pendingChild stamp), aborting with SCREEN_RECORDING_SERVER_SHUTTING_DOWN instead of launching an orphan. This completes the pendingChild-reaping design for the Android resolveAndroidBinary window it previously missed. - Android stop: when the best-effort `rm -f` after a successful pull failed, the already-delivered mp4 was orphaned on /sdcard permanently because its path was nulled. Keep the path on the session when the rm did not succeed so the next start's stale sweep removes it; the reminder is still cleared since the video was delivered. Adds regression tests: iOS/Android start-after-dispose aborts without spawning, and a delivered stop whose on-device rm fails preserves the file for the next start's stale sweep.
…termark (opt-out flag)
`simctl io recordVideo` and `adb screenrecord` both emit VARIABLE-framerate
video — frames land only when the screen changes, so static spans collapse and
the irregular timestamps stutter on playback (this, plus over-compression, was
the "laggy" capture). Neither path uses the simulator-server. On stop the raw
capture now runs one ffmpeg finishing pass that:
- normalizes to a constant 30fps (smooth playback everywhere), and
- overlays a corner watermark — for now a translucent placeholder square,
bottom-left, standing in for assets/watermark.svg ("Argent / By @swmansion")
until the artwork is final.
The pass is best-effort: if ffmpeg is missing or fails, the raw capture is
returned untouched with a warning, so a recording is never lost. The watermark
is on by default and disableable with `argent disable video-watermark`
(re-enable with `argent enable video-watermark`); 30fps normalization always
applies when ffmpeg is present. The flag is read live per stop, so toggling it
takes effect without restarting the tool-server.
Supporting the opt-out flag needed the flag system to learn about default-ON
flags: FlagDefinition gains `defaultEnabled`, a registry-aware `isFeatureEnabled`
resolves an unset flag to its declared default, and `argent disable` persists an
explicit `false` for such a flag (an unset would revert it to its ON default).
Opt-in flags are unchanged (no `defaultEnabled` -> same enable/disable behavior).
Tests: opt-out flag semantics (core + CLI, incl. `argent disable video-watermark`
writes false), the 30fps/watermark filter chain, and the shipped registry entry.
E2E on an iOS sim: default -> constant 30fps + watermark; flag off -> 30fps,
no watermark.
…ptive contrast Replace the placeholder square with the actual Argent wordmark + mark and a "By @swmansion" tagline. The artwork lives in assets/watermark.svg and is rasterized to a PNG embedded as base64 (watermark-asset.ts), so no asset file ships next to the compiled server and no SVG rasterizer runs at runtime. The finishing ffmpeg pass now ffprobes the frame size and builds a numeric filter_complex that, per pixel, keeps the white logo over dark background and a runtime-derived near-black copy over light background (maskedmerge driven by a background-luma mask) for maximum contrast against any UI. The stamp is faded to 20% opacity (80% transparent) and pinned bottom-left. Details that only surface on real yuv420p video: box dimensions are snapped even so the 4:2:0 crop and the rgba logo scale to identical sizes (maskedmerge aborts otherwise); the still logo is looped so there is a logo frame per video frame, and overlay shortest=1 bounds the output. Degrades gracefully - no ffprobe still normalizes to 30fps but skips the watermark, no ffmpeg returns the raw capture - and warning says which. Still opt-out via `argent disable video-watermark`.
…mark + grow watermark 10%
Replaces the two per-platform capture backends with a single path built on simulator-server — the same backend `screenshot` and every interaction tool already drive. simulator-server publishes the device screen as an MJPEG stream (multi-subscriber, so a preview window can stay open on the same device); we subscribe to it, pace the frames onto a fixed 30fps timeline and pipe them into one ffmpeg process that encodes — and watermarks — straight to the final mp4. Gone with the old backends: `xcrun simctl io recordVideo`, the on-device `adb shell screenrecord` + `/sdcard` pull + PID juggling, screenrecord's 180s hard cap, the separate finishing pass and its `ffprobe` dependency. Pacing the frames here rather than pointing ffmpeg at the stream is what makes the timeline honest. A device only emits a frame when the screen CHANGES, so a still screen used to collapse to a fraction of a second of video (a documented wart of the old iOS path), and ffmpeg — blocked on a silent socket — would not even answer a stop signal promptly. Re-emitting the last frame on a wall-clock schedule keeps video duration equal to real elapsed time; duplicate frames cost almost nothing once encoded. Behaviour changes: - video duration now tracks wall-clock on every platform, static screens included, and stop returns in well under a second - no Android length cap; `timeLimitSeconds` (max 600) is the only limit - physical Android devices are recordable through the same path - ffmpeg is now required (it is the encoder) and the start fails up front with an install hint instead of degrading at stop time - the watermark is stamped during capture, so the flag is read at start - tvOS simulators are rejected explicitly: no simulator-server backend
…mp file Two gaps from a self-review of the capture rework: - when the frame size cannot be read from the stream's first JPEG the recording proceeds without the watermark, which was silent; stop now says so in `warning`, keeping the contract the finishing pass used to have - the logo temp file is removed by stop, but a tool-server shutdown abandons that path; session dispose now removes it too
A recording left running across waits, slow steps, or thinking time used to come back padded with dead air. Now the pump collapses stretches where the screen does not change: it keeps the first second of each still stretch (so pauses read naturally) then drops the duplicate frames until the screen moves again — a change of even a single pixel counts, since frames are compared byte- exact (stronger than a hash, and native-fast behind a reference short-circuit). Active stretches still play back at real time: the wall-clock pacer is re- anchored whenever a dead stretch is skipped, so the gap contributes no output frames instead of a catch-up burst of the next frame. On by default; `trimStatic: false` keeps a faithful real-time recording. stop now returns the trimmed video length as durationMs plus wallClockMs/trimmedMs when trimming actually removed anything. Verified: 39 unit tests (7 new trim cases incl. resume-after-dead, byte- identical-new-frame-stays-trimmed, duration reporting) and E2E on an iOS 18.6 sim — a 23s idle-heavy session returned a 1.47s clip with the real transitions intact (44 distinct frames, not the 30-frame grace floor), an 8s burst-motion session returned 2.1s across multiple bursts, and a trimStatic:false control returned the full 23s. Cross-checked against ffmpeg mpdecimate on the same footage: the trim keeps at least as much as ffmpeg's own duplicate detector.
Enable simulator-server's built-in touch visualizer for the duration of a recording so taps, swipes, drags, pinches and rotates are drawn straight into the video: a pulse for a tap, a fading comet trail for a swipe/drag, and paired markers for a two-finger gesture. This makes a recording self-explanatory — a viewer sees where each interaction landed instead of watching the UI react to an invisible finger. The overlay is rendered by simulator-server into the same MJPEG stream the recording already consumes (nothing is composited host-side), driven by the touch events argent already sends. `screen-recording-start` turns it on right after the encoder goes live and restores it to off when the capture ends (stop, time-limit cap, encoder crash, or session teardown), so the visualizer never outlives the recording it was enabled for. On by default; pass `showTouches: false` to capture the raw screen. Enabling is best-effort — a build without the pointer feature just yields a warning at stop and a normal, overlay-free video.
…rdings Screen recordings now land durably under `.argent/recordings/` on the client, instead of the disposable temp-artifact cache, and — for a remote `argent link` tool-server — on the client host rather than the server. Mechanism: a new optional `saveDir` hint on the artifact wire contract. `screen-recording-stop` tags its video with `saveDir: ".argent/recordings"` (and drops the internal `argent-` temp prefix from the filename); the client materializer honours it, copying the file when the server is co-located and downloading it over `/artifacts/:id` when it is remote. Either way the bytes end up in the durable directory, so the video is always where the user can find it regardless of where it was produced. The hint resolves against the client's project root (nearest ancestor with `.git`/`package.json`/`.argent`), falling back to the global `~/.argent` when run outside a project — matching how the rest of argent scopes its per-project config. Resolution is hardened: absolute paths and `..` escapes are rejected, and directory bundles are excluded, so a compromised tool-server can't steer a write outside the base directory. Adds `findProjectRoot` to configuration-core (a null-returning variant of `resolveProjectRoot`) to drive the project-vs-global choice.
…track Remote (`sim-remote`) iOS simulators can now be screen-recorded. They have no local HTTP MJPEG stream — their screen is published as an H.264 (Annex-B) elementary stream over MoQ/WebTransport — so recording them previously failed the capability gate. This wires that MoQ "video" track into the same recording pipeline the local path uses. - `moq-video-stream.ts`: subscribes to the server's `simulator`/`video` track, strips hang's per-frame microsecond-timestamp VarInt off each MoQ frame, and hands the raw Annex-B access units to the capture code. Buffers frames until a consumer attaches so the leading keyframe (carrying SPS/PPS) is never dropped; waits for the first keyframe before proceeding. - `moq-capture.ts`: pipes those frames into `ffmpeg -f h264` — the same encoder and adaptive-contrast watermark backend the MJPEG path uses. There is no Node-side pump: the server timestamps frames in real time, so they are pushed as they arrive and `-use_wallclock_as_timestamps` + a constant output rate reconstruct an honest real-time timeline. Watermark dimensions come from the keyframe's in-band SPS via ffprobe. - `screen-recording-start` routes `ios-remote` devices to the MoQ path (new `appleRemote` capability on start and stop); the session blueprint accepts the platform. The stop path (host-side, source-agnostic) retrieves the video unchanged. - The touch visualizer is not available over MoQ (the control channel carries no pointer command and a remote sim exposes no dialable `/api/pointer`), so a requested `showTouches` surfaces a clear warning at stop rather than silently doing nothing. `moq-client.ts` gains a small `establishMoqSimulator` helper (extracted from the screenshot client) so the connection/cert-pinning/lease-token handling lives in one place, shared by the screenshot client and the video stream. Verified end-to-end against a locally-built `--features moq` simulator-server: start → stop produces a valid watermarked H.264 mp4 with an honest real-time timeline and the touches-unavailable warning. Live radon-cloud sim-remote verification is out of reach locally, but the orchestrator path is a thin shell over the verified transport.
Member
Author
|
This has to be tested against a real sim-remote environment before being merged. Thus skipping this PR in 0.17.0, might land in 0.18.0. |
Resolve conflicts from the screen-recording stack being squash-merged into main (#517/#548/#549). Screen-recording files that both sides added are taken from main's more-refined version, with the PR's save-dir + MoQ delta layered on top. In capture.ts the two refactors are reconciled: superviseCapture is kept (so the MoQ path can reuse the cap + exit-handler) and is invoked before the pointer-enable await, preserving main's fix that arms the exit handler before that await. flags.ts collapses to the identical findProjectRoot both sides introduced. package-lock.json reconciled so tools-client's new configuration-core dependency is recorded.
The merge resolution mistakenly restored packages/tool-server/node_modules, a self-referential ELOOP symlink that main deliberately deleted in #537. The PR branch never modified it, so the merge must follow main's deletion.
Pick up #551 (save recordings to .argent/recordings), squash-merged into main after the first merge. Same squash pattern: artifacts.ts/artifacts.test.ts take main's canonical #551 version (the MoQ commit never touched them), and screen-recording-stop.ts keeps main's save-dir logic plus the PR's appleRemote capability. Lock left consistent.
Two places borrow a rule from the local MJPEG path where the codec makes it mean something different. Back-pressure. Both paths stop writing when ffmpeg falls behind, which is right. But the MJPEG pump resumes with the next frame it sees, and that is only safe because JPEGs are independent — each one decodes on its own. An H.264 access unit is not: every following P-frame is coded against the units before it, so resuming mid-GOP feeds the decoder references that were dropped, and the output stays broken until the next keyframe. Skip forward to that boundary instead: a lost second of video rather than a corrupt stretch. Pre-consumer buffering. The stream holds frames until the capture code attaches, so the leading keyframe is never lost while ffprobe reads the dimensions and ffmpeg starts. That queue had no bound, and how fast it fills is the remote screen's business, not this module's. Bound it, and trim by dropping whole GOPs: whatever replays first is what ffmpeg decodes from, so the trim has to land on a keyframe or it hands the encoder a P-frame whose references are gone. A buffer holding a single GOP has nothing droppable and is left alone. Both rules are now small exported functions rather than closures inside the start path, so each is pinned by a test that fails when the rule is reverted to the MJPEG one.
The MoQ path claimed to reconstruct "an honest real-time timeline". Measured against the actual ffmpeg invocation, it does not: `-fps_mode cfr` can hold a picture only up to the next packet, and this transport sends nothing while the screen is still, so the video ends at the last change. A recording stopped a few seconds after the UI settled — the normal case, since tool-call latency alone is seconds — returned a 1s file reporting 5s of `durationMs`, a field documented as "length of the returned video". Interior still stretches are held correctly; it is only the tail that has no packet to extend to. Track when the last frame reached the encoder and report that, with a warning naming the gap. The paced local path leaves the new field null and keeps reporting wall clock, which its video genuinely covers. Three transport failures were also indistinguishable from a quiet device: - A session dropping right after connect left `waitForFirstFrame` to burn its full 10s and then blame the device's screen, discarding the real cause. - A server that stops publishing mid-recording closes the track cleanly, which went unrecorded — so stop returned a video that freezes early with no warning and a duration covering time it never captured. - `close()` during a pending wait left the caller waiting on a stream that was already gone. Each now fails with the cause, mirroring `mjpeg-stream`'s `fail`. The connect race also abandoned its loser: a session completing after the timeout fired produced a live WebTransport connection nobody owned, for the life of the tool-server. It is closed now, and the timer cleared on the success path. Fixes the pre-consumer buffer trim from the previous commit, which dropped to the newest GOP and would have taken the head with it. The head is where the SPS/PPS live on a producer that sends its parameter sets once, so it is kept and the middle goes instead. Docs: both SKILL.md copies still listed remote simulators as unrecordable and promised a wall-clock-accurate timeline; `trimStatic` was silently ignored on remote while its description promised trimming; and the start tool's frame-stream error still blamed a remote transport it can no longer reach.
Member
Author
|
Once #587 lands this PR has to be either re-done or moved to Radon. Keeping it open as pending feature reminder |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Screen recording now works on remote (
sim-remote) iOS simulators. They used to fail the capability gate: a remote sim has no local HTTP MJPEG stream, because its screen is published as an H.264 (Annex-B) elementary stream over MoQ/WebTransport. This wires that MoQvideotrack into the same recording pipeline the local path uses.That answers the question that prompted it - "we still have access to the video stream, so what's stopping us from using it the same way as the local one?" Nothing fundamental; the track just wasn't being consumed.
How
moq-video-stream.tssubscribes to the server'ssimulator/videotrack, strips hang's per-frame microsecond-timestamp VarInt off each MoQ frame, and hands the raw Annex-B access units to the capture code. It buffers frames until a consumer attaches so the leading keyframe (SPS/PPS) is never dropped, and waits for the first keyframe before proceeding. That buffer is bounded: past the cap it keeps the head (the parameter sets) and the newest GOP and drops the middle, so what replays is still decodable. A dropped session, a server that stops publishing, and a close mid-connect each fail with the cause rather than leaving the caller to time out.moq-capture.tspipes those frames intoffmpeg -f h264, reusing the local path's encoder and adaptive-contrast watermark backend. There's no Node-side pump: frames are pushed as they arrive, and-use_wallclock_as_timestampsplus a constant output rate hold each picture until the next one lands. Under back-pressure it stops writing and resumes at the next keyframe rather than mid-GOP, since a dropped access unit breaks every P-frame that references it. Watermark dimensions come from the keyframe's in-band SPS via ffprobe.screen-recording-startroutesios-remoteto the MoQ path (newappleRemotecapability on start and stop), and the session blueprint accepts the platform.stopCaptureis host-side and source-agnostic, so it retrieves the video unchanged.moq-client.tsgains a smallestablishMoqSimulatorhelper (connection, cert pinning, lease token) shared by the screenshot client and the video stream.Parity note
The touch visualizer isn't available over MoQ: the control channel's protobuf has no pointer command, and a
moq+remote://sim exposes no dialable/api/pointer. A requestedshowTouchestherefore surfaces a clear warning at stop rather than silently doing nothing. The watermark carries over. Static-frame trimming doesn't apply, since the on-change encoder never sends the duplicate frames trimming collapses.trimStaticis documented as ignored here rather than silently dropped.Verification
End to end against a locally-built
--features moqsimulator-server driving a booted iOS 18.5 sim:startMoqCapturethenstopCaptureproduces a valid watermarked H.264 mp4 (660×1434), plus the touches-unavailable warning. Watermark render confirmed by extracting a frame.stripHangTimestampacross 1/2/4/8-byte VarInts,isKeyframeSPS/IDR detection), the buffer trim, the back-pressure resync, the three transport-failure paths driven through a fake MoQ session, and the duration reporting.tsc --build,tsc -p tsconfig.test.json, prettier and eslint clean.Live radon-cloud
sim-remoteverification is out of reach locally, butopenMoqVideoStream(udid)is a thin shell (asim-remote moq-infolookup) over the transport proven above.Timeline caveat
-fps_mode cfrholds a picture only up to the next packet, and this transport sends nothing while the screen is still — so a still stretch between two changes is reconstructed at its true length, but one that runs to the end of the recording is not: the video ends at the last change. Measured against the real ffmpeg invocation, a recording stopped ~4s after the UI settled came back as a ~1s file.That is inherent to a pushed H.264 source (there is no frame to re-emit the way the MJPEG pump re-emits its last JPEG), so
stopreports it instead of hiding it:durationMsis the length of the file, not the wall clock, and a warning names the gap. The paced local path is unaffected and still reports wall clock, which its video genuinely covers. BothSKILL.mdcopies previously stated the opposite (remote unsupported, timeline always wall-clock accurate) and are updated.