feat(stereo-split): correct right-eye colour in joined H.264 conversion - #229
Merged
Merged
Conversation
Fit one right-to-left photometric correction from the joined H.264 stream and apply it to the right eye before encoding, so both eyes of a stereo-split derivative agree in colour. - color_consistency.py owns sampling, SIFT matching, exposure outlier rejection, luminance-binned gain curves, a smooth spatial gain field, held-out block validation, and the apply/skip decision. - The correction is fitted once and frozen for the whole recording, and it is only wired into the joined path; split H.264 inputs still keep their payloads untouched. - Replaying a real 1106-frame DECXIN capture moves the matched median CIEDE2000 from 13.3 to 4.0 while preserving IMU, serial number, and every video timestamp. - The manifest gains a lean colour summary and the output metadata keeps the full fitted model, so the processing mode and manifest schema are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fit one right-to-left photometric correction from the joined H.264 source and apply it to the right eye before encoding, so both eyes of a stereo-split derivative agree in colour.
jobs/stereo-split/color_consistency.py(new) owns sampling, SIFT matching, exposure-outlier rejection, luminance-binned gain curves, a smooth spatial gain field, held-out block validation, and the apply/skip decision behind a small interface: feed frame pairs, get a frozen plan, apply it.convert_mcap_stereo_h264.pyadds a calibration pass for joined H.264 input and corrects the right-eye crop inside the existing frame callback, so the encoder pipeline, GOP/bitrate settings, IMU preservation, timestamp repair, and topic copying are untouched.run_processing.pypublishes a leancolor_consistencysummary in the manifest and keeps the full fitted model inmetadata.yaml.Scope and compatibility
not_needed/insufficientand the job still succeeds with the right eye untouched. Only structural failures (decode, frame counts, encoding, model invariants) fail the job.processing_modeand manifestschema_versionare unchanged. The new fields are additive and the currently deployed Keystone ignores unknown JSON keys, so no backend change is required to roll this out; rollback is the existing image-digest switch.Evidence
Tests:
python3 -m unittest discover -s jobs/stereo-split/tests -p 'test_*.py'→ 71 tests pass (20 new).Replayed on a real 1106-frame DECXIN capture (
toledoregression clip, joined H.264, 76 MB, 37.5 s):applied,corrected_frames = 1106 = right_videos14.03→4.60,13.64→4.24,13.76→4.40/decxin/imuand/decxin/serial_numberlog_time/publish_time/sequencePerformance
On the regression clip the job goes from ~47 s to ~108 s. The added cost is one extra decode pass for calibration (~21 s) and ~24 ms per frame for the correction (~27 s). Identified follow-ups, deliberately out of scope here: sample only IDR access units during calibration (the source has an IDR every 30 frames, so this alone removes most of the extra decode), vectorise the match filtering loop, lower
match_scale, and hand writable frames out of the decoder.Test plan