fix(live-proof): only publish recordings that demonstrate a change - #1199
Merged
Conversation
Every published proof was useless: terminal clips ended before the command produced output (expect_output captured the pane once, with no polling, and matched the echoed command), and browser clips showed a page that never changed because plans asserted static text already rendered at load. Nothing gated attachment, so failed drives shipped anyway. Terminal steps now wait for real output and match only post-command content; both drivers settle between steps, hold the end state, and honour a minimum length. Attachment requires an expectation that was absent at plan start and satisfied afterwards, so a demonstration that proves nothing publishes nothing. Pixel-difference gating was evaluated and rejected: ambient page animation scores higher (SSIM 0.914) than a real blade-opening demo (0.980), so no threshold separates them. The review prompt now requires a state-changing action plus an assertion on content absent beforehand, with real selectors.
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. |
Contributor
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(live-proof): only publish recordings that demonstrate a change This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
This was referenced Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every live proof published so far was useless, and I only caught it by watching the videos rather than checking that the URLs resolved.
expect_outputcaptured the pane exactly once with no polling immediately after Enter, so it threw instantly; it also grepped the whole pane including the echoed command, so other plans could match the command the model had just typed."press /", the finder's own hint) and nothing scrolled the changed region into frame.drive_statuswas recorded but unused, so failed drives were uploaded and posted to public review comments.Fixes: terminal
runwaits for real output andexpect_outputpolls with a timeout against post-command content only; both drivers settle between steps, hold the end state, and honour a minimum length; browser steps scroll their target into view.The substance gate is semantic, not pixel-based. Attachment now requires an expectation that was absent at plan start and satisfied afterwards — a demonstration that verifies nothing publishes nothing. Pixel-difference gating was implemented, measured against real artifacts, and rejected: the junk frozen-page recording scores SSIM 0.914 between first and last frame while a genuine blade-opening demo scores 0.980, because ambient page animation swamps any threshold. The review prompt now requires a state-changing action plus an assertion on content that is absent beforehand, and real selectors rather than prose.
Validation
Run locally against the two real plans:
skip: plan verified nothing that changedand writes no manifest — nothing would be attached;"Found openclaw/clawsweeper#1178"(absent at load) attaches, and its extracted frames show the input filling, the status flipping, and the toast appearing.pnpm buildclean; live-proof suite 33/33; full unit suite 2,326 passed / 0 failed (parallel with review, exit 0). Autoreview (Codex, gpt-5.6-sol, high): clean, "patch is correct (0.98)".Follow-up
The five PRs that received junk videos need cleanup once this lands: re-dispatching the same heads overwrites the head-scoped R2 objects and record block, and any plan that legitimately has nothing to show needs its stale Live Proof block stripped from the record.