Skip to content

feat(stereo-split): correct right-eye colour in joined H.264 conversion - #229

Merged
shark0F0497 merged 1 commit into
main-v2from
feat/stereo-split-color-consistency
Sep 16, 2026
Merged

shark0F0497 merged 1 commit into
main-v2from
feat/stereo-split-color-consistency

Conversation

@shark0F0497

Copy link
Copy Markdown
Collaborator

What

Fit one right-to-left photometric correction from the joined H.264 source and apply it to the right eye before encoding, so both eyes of a stereo-split derivative agree in colour.

  • jobs/stereo-split/color_consistency.py (new) owns sampling, SIFT matching, exposure-outlier rejection, luminance-binned gain curves, a smooth spatial gain field, held-out block validation, and the apply/skip decision behind a small interface: feed frame pairs, get a frozen plan, apply it.
  • convert_mcap_stereo_h264.py adds a calibration pass for joined H.264 input and corrects the right-eye crop inside the existing frame callback, so the encoder pipeline, GOP/bitrate settings, IMU preservation, timestamp repair, and topic copying are untouched.
  • run_processing.py publishes a lean color_consistency summary in the manifest and keeps the full fitted model in metadata.yaml.

Scope and compatibility

  • Joined H.264 only. Split H.264 inputs keep their current behaviour: payloads are preserved and only timestamps are repaired. No colour logic is wired into that path.
  • The correction is fitted once and frozen for the whole recording, so it cannot flicker over time.
  • A recording whose eyes are already consistent, or whose features are too sparse, is reported as not_needed / insufficient and the job still succeeds with the right eye untouched. Only structural failures (decode, frame counts, encoding, model invariants) fail the job.
  • processing_mode and manifest schema_version are unchanged. The new fields are additive and the currently deployed Keystone ignores unknown JSON keys, so no backend change is required to roll this out; rollback is the existing image-digest switch.

Evidence

Tests: python3 -m unittest discover -s jobs/stereo-split/tests -p 'test_*.py'71 tests pass (20 new).

Replayed on a real 1106-frame DECXIN capture (toledo regression clip, joined H.264, 76 MB, 37.5 s):

Check Result
Decision applied, corrected_frames = 1106 = right_videos
Matched median CIEDE2000 13.3 → 4.0 (capture-level, 56 sampled frames, 15 867 correspondences)
Per-frame check on the produced right eye frames 0/500/1100: 14.03→4.60, 13.64→4.24, 13.76→4.40
/decxin/imu and /decxin/serial_number byte-for-byte identical to the source
Video log_time/publish_time/sequence identical to the current production output
Determinism two independent runs produced identical bytes

Performance

On the regression clip the job goes from ~47 s to ~108 s. The added cost is one extra decode pass for calibration (~21 s) and ~24 ms per frame for the correction (~27 s). Identified follow-ups, deliberately out of scope here: sample only IDR access units during calibration (the source has an IDR every 30 frames, so this alone removes most of the extra decode), vectorise the match filtering loop, lower match_scale, and hand writable frames out of the decoder.

Test plan

  • Unit tests for the fitting/decision logic (synthetic gain mismatch, consistent eyes, flat frames, rejection, highlight protection, determinism, CIEDE2000 reference pairs)
  • Converter-level integration tests (joined H.264 corrected, disabled path unchanged, insufficient path, spatial field path)
  • Runner-level manifest/metadata projection tests
  • Full replay of the real regression clip with byte-level checks on non-video messages and timestamps

Fit one right-to-left photometric correction from the joined H.264 stream and
apply it to the right eye before encoding, so both eyes of a stereo-split
derivative agree in colour.

- color_consistency.py owns sampling, SIFT matching, exposure outlier
  rejection, luminance-binned gain curves, a smooth spatial gain field,
  held-out block validation, and the apply/skip decision.
- The correction is fitted once and frozen for the whole recording, and it is
  only wired into the joined path; split H.264 inputs still keep their payloads
  untouched.
- Replaying a real 1106-frame DECXIN capture moves the matched median
  CIEDE2000 from 13.3 to 4.0 while preserving IMU, serial number, and every
  video timestamp.
- The manifest gains a lean colour summary and the output metadata keeps the
  full fitted model, so the processing mode and manifest schema are unchanged.
@shark0F0497
shark0F0497 merged commit 58a10cd into main-v2 Sep 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant