Skip to content

fix(video): validate pass-through fixed GOP cadence - #380

Open
victorwon2001 wants to merge 2 commits into
Hebbian-Robotics:mainfrom
victorwon2001:fix/issue-376
Open

victorwon2001 wants to merge 2 commits into
Hebbian-Robotics:mainfrom
victorwon2001:fix/issue-376

Conversation

@victorwon2001

@victorwon2001 victorwon2001 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add video-keyframe-cadence to hflow doctor, comparing structurally valid video keyframes with the message-index grid implied by stamped gop_seconds and the channel's measured frame rate.
  • Preserve current pass-through transform behavior: irregular GOP cadence is not a SourceNotConforming refusal, including when the existing lossless AUD repair is needed.
  • Add regression coverage proving AUD repair preserves the original H.264 payload suffix while the repaired irregular stream is diagnosed by hflow doctor; non-finite GOP-frame calculations are reported as cadence errors rather than raising.

Refs #376

Why

Real-corpus evidence showed that enforcing fixed GOP during pass-through ingest would reject a recording used to validate the existing lossless AUD-repair path. The change is therefore limited to diagnosis; pass-through provenance/v1.gop_seconds remains the configured target, and a separate provenance revision can distinguish configured and measured cadence without changing canonical bytes in this PR.

Validation

The focused processing/doctor suite passed with 74 tests, and the full root suite passed with 1728 tests and 6 skips. Ruff, formatting, typing, documentation links, and diff whitespace validation also passed.

env CARGO_TARGET_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.cargo-target' XDG_CACHE_HOME='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/xdg' GOCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-build' GOMODCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-mod' PIP_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/pip' UV_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/uv' npm_config_cache='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/npm' YARN_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/yarn' CYPRESS_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/cypress' CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 MAKEFLAGS=-j2 NINJAFLAGS=-j2 GOMAXPROCS=2 RAYON_NUM_THREADS=2 NODE_OPTIONS=--max-old-space-size=3072 uv run pytest -q tests/test_processing_regressions.py tests/test_doctor.py
env CARGO_TARGET_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.cargo-target' XDG_CACHE_HOME='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/xdg' GOCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-build' GOMODCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-mod' PIP_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/pip' UV_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/uv' npm_config_cache='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/npm' YARN_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/yarn' CYPRESS_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/cypress' CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 MAKEFLAGS=-j2 NINJAFLAGS=-j2 GOMAXPROCS=2 RAYON_NUM_THREADS=2 NODE_OPTIONS=--max-old-space-size=3072 uv run ruff check --fix
env CARGO_TARGET_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.cargo-target' XDG_CACHE_HOME='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/xdg' GOCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-build' GOMODCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-mod' PIP_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/pip' UV_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/uv' npm_config_cache='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/npm' YARN_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/yarn' CYPRESS_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/cypress' CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 MAKEFLAGS=-j2 NINJAFLAGS=-j2 GOMAXPROCS=2 RAYON_NUM_THREADS=2 NODE_OPTIONS=--max-old-space-size=3072 uv run ruff format
env CARGO_TARGET_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.cargo-target' XDG_CACHE_HOME='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/xdg' GOCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-build' GOMODCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-mod' PIP_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/pip' UV_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/uv' npm_config_cache='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/npm' YARN_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/yarn' CYPRESS_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/cypress' CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 MAKEFLAGS=-j2 NINJAFLAGS=-j2 GOMAXPROCS=2 RAYON_NUM_THREADS=2 NODE_OPTIONS=--max-old-space-size=3072 uv run ty check
env CARGO_TARGET_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.cargo-target' XDG_CACHE_HOME='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/xdg' GOCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-build' GOMODCACHE='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/go-mod' PIP_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/pip' UV_CACHE_DIR='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/uv' npm_config_cache='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/npm' YARN_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/yarn' CYPRESS_CACHE_FOLDER='/home/victor/문서/ossbot2/data/worktrees/pr-39/hflow/.dogood-cache/cypress' CARGO_BUILD_JOBS=2 RUST_TEST_THREADS=2 CMAKE_BUILD_PARALLEL_LEVEL=2 MAKEFLAGS=-j2 NINJAFLAGS=-j2 GOMAXPROCS=2 RAYON_NUM_THREADS=2 NODE_OPTIONS=--max-old-space-size=3072 uv run pytest -q
lychee --no-progress --include-fragments --exclude '^https://github\.com/Hebbian-Robotics/hflow/(issues|security/advisories/new)$' --exclude-path references/mcap-spec.md --exclude-path references/foxglove-CompressedVideo.proto .
git diff --check

Checklist

  • I added or updated outcome-focused tests for changed business logic.
  • I updated documentation for changed behavior, flags, formats, or requirements.
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding this on #376 rather than on your code. The gate is clean and the derivation is right.

#376 is open on what to do, and this implements one of its three options. The author measured nominal-io/xplane-mcap 36 minutes after you opened: 9 off-grid intervals in 102, on a conforming pass-through stream that video.py:513 cites as validating our lossless-suffix guarantee. Your change would refuse it.

The derivation and the finding survive whichever way it lands; the SourceNotConforming raise is the contingent part.

Separately: _validate_passthrough_video_payload returning bool gives a validator a second job in its return type.

@kstonekuan

kstonekuan commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Holding this on #376 rather than on your code. The gate is clean and the cadence derivation is right.

#376 is open on what to do, not just how, and this implements one of its three options. The issue author measured nominal-io/xplane-mcap 36 minutes after you opened: 24,689 messages, no AUDs, 9 off-grid intervals in 102. Your _validate_passthrough_video_payload change would refuse a recording video.py:513 cites as validating our own lossless-suffix guarantee.

The derivation and the finding survive whichever way #376 lands; the SourceNotConforming raise is the contingent part. I will say here when it has a direction.

Separately: _validate_passthrough_video_payload returning bool gives a validator a second job in its return type.

@kstonekuan

kstonekuan commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

#376 has a direction: measure and stamp the observed interval, report irregular cadence as a doctor finding, refuse nothing.

Your derivation and the video-keyframe-cadence finding survive. The SourceNotConforming raise comes out.

What goes in is the other half: provenance stamps the requested gop_seconds on pass-through as though it had been applied. Stamp the interval you derived instead.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reshape is right: the raise is gone and the cadence check is a finding.

What blocks it is the other half of #376. Provenance stamps gop_seconds at transform.py:900 from the resolved config, through one writer both paths use, with nothing recording which ran. So on pass-through this check reads a requested value as ground truth, and nominal-io/xplane-mcap (9 off-grid intervals in 102) produces errors against a GOP nobody applied.

For encoded video it is provably right: scenecut=0:keyint=min-keyint=gop_frames. video.py:670-683 already asserts that, so consider calling it.

Two smaller things: fps = 1.0 for a single-message channel is invented, and _check_video_payload now returns bool | None where None means four things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants