fix(robodojo): decode JPEG frames with the recorder's channel order

[wayrise]

Closes #13. Thanks @rakhimovv — you called this correctly from the asymmetry alone, without the data to confirm it.

I had RoboDojo trajectories on hand, so I settled the question you left open, then fixed it.

It is the cv2.imencode case

Your "if" holds. Each corpus ships preview_video/*.mp4 next to data/, produced by the recorder itself for humans to look at, on a path independent of both candidate decodings. Over saturated pixels (neutral areas are invariant under an R/B swap and just dilute the numbers), the preview matches cv2.imdecode and inverts PIL's — on both the sim and the real corpora, every episode checked:

real corpus, 10 episodes   mean |preview - cv2| =  7.57    mean |preview - PIL| = 45.34
sim corpus,  4 samples     mean |preview - cv2| =  1.7     mean |preview - PIL| = 40..57

The sim corpus makes it unarguable by eye: the scene is a rendered wooden desk with yellow number tiles, and PIL's reading makes the wood grain blue and the tiles cyan.

This is not circular — had the preview been made by decoding these JPEGs with cv2 and writing them straight out, it would match PIL instead. It does not, which means the recorder held true RGB, converted correctly for the video, and skipped the conversion for cv2.imencode. Exactly what robotwin.py describes.

The fix

_decode_jpeg now uses cv2.imdecode, with robotwin.py's rationale carried over. After it:

sim  corpus, production config   |preview - reader|  40..57  ->  3..16   (spread is the configured colour jitter)
real corpus, no jitter           |preview - reader|  31.74   ->   4.92   (= the JPEG vs H.264 floor)

Per-channel means went from transposed to aligned.

Why the tests did not catch it

The fixture encoded with PIL.Image.save(format="JPEG") — the opposite convention to the recorder — so a wrong decoder round-tripped cleanly. It now encodes the way RoboDojo does, which makes the existing three-camera assertions (head R-dominant, left_wrist G, right_wrist B) pin the real contract instead of a fictional one, and a new test states the property outright. Reverting the decoder to PIL fails 4 tests.

Scope

I walked the whole path rather than assuming, in both directions:

• Training — injecting a channel swap at _decode_jpeg flips the dataloader's output exactly, in both single-view and multiview mode, so nothing between the decode and the model touches channel order. crop_and_resize, assemble_multiview_layout and VideoColorJitter are all channel-agnostic.
• Deploy — obs_preprocess PIL-decodes the client's bytes and applies the same geometry helpers; feeding it these raw HDF5 bytes reproduces the PIL reading byte for byte. It performs no conversion, so eval colour is whatever the client encodes, and every in-repo client (benchmarks/utils/client.py, mock_genmanip_server.py) encodes true RGB. That is the train/eval mismatch you described.

RoboDojo's own eval client lives in XPolicyLab, so I could not verify that half from here; if it also sends true RGB, XPolicyLab/XPolicyLab#116 is this same bug seen downstream and should resolve with this.

full suite : 1982 passed, 20 skipped
ruff check / ruff format --check : clean

Heads-up for anyone holding RoboDojo checkpoints: this changes the training input distribution. Weights trained before it were fitted on channel-swapped frames and will not transfer cleanly.

🤖 Generated with Claude Code