Skip to content

Fail demo video rendering when the recorded command reports an error or a warning #3057

Description

@AlexSkrypnyk

Summary

A demo video published in the documentation is a claim that the command it records runs clean. Nothing currently checks that claim, so a recording that shows an error or a warning can be rendered and published unnoticed. This happened: the lint demo published an ESLint warning that stayed in the documentation until someone read the video.

The check belongs at recording time, in ahoy update-videos, so a bad recording never becomes an artifact in the first place.

Details

While recording, decode each cast to plain text and refuse to render the SVG when that text reports an error, a warning or a failure.

  • Decode the cast to plain text, stripping ANSI escape sequences, including sequences split across two recorded chunks.
  • Match the decoded text for reported errors, warnings and failures.
  • Match failures explicitly. A command that fails inside a recording still leaves asciinema exiting 0, so the exit code proves nothing.
  • Exempt fragments that carry a marker word without reporting anything, before matching: paths, URLs, package names such as symfony/error-handler, command line options such as --no-error-on-unmatched-pattern, and zero counts such as 0 errors, warnings: 0 and --max-warnings=0.
  • Stop the render when anything is found, and report every offending line with its line number so the source is identifiable.
  • Write the decoded text to .artifacts/videos/<name>.txt for inspection. It is working material, not a published artifact, so it is not committed.
  • Group repeated identical lines in the report rather than printing each occurrence.

Scope and constraints:

  • The check runs during recording only. It does not run in CI and does not re-check committed casts.
  • No override flag. Output that cannot be fixed in this repository is fixed at the command that emits it, not waved through.
  • Every existing recording passes the check once the sources of its output are fixed.

An earlier implementation was built and validated against all nine recordings in #3047, then dropped when that branch was rebuilt around the ESLint 9 migration. That work can be used as a starting point.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A2Working clone index A2

    Type

    No type

    Projects

    • Status
      Released in 1.41.0

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions