robodojo and robotwin dataloaders decode JPEG frames with opposite channel conventions

[rakhimovv]

The two HDF5 dataloaders decode their JPEG camera buffers with opposite channel conventions. robotwin.py uses cv2.imdecode and documents exactly why; robodojo.py uses PIL and says nothing about it. I could not settle which is right for RoboDojo from the repo alone, so this is a question rather than a bug report — but the asymmetry looked worth raising.

The two decoders

openwam/dataloader/robotwin.py:624-634 (line numbers on main):

def _decode_jpeg(self, jpeg_bytes) -> Image.Image:
    """Decode JPEG bytes from HDF5 to a PIL RGB image.

    RoboTwin encodes frames by passing RGB arrays directly to
    ``cv2.imencode`` (which expects BGR), so R and B channels are
    swapped inside the JPEG. Using ``cv2.imdecode`` reverses this
    swap, giving back the original RGB order — no further
    conversion needed.
    """
    arr = cv2.imdecode(np.frombuffer(bytes(jpeg_bytes), np.uint8), cv2.IMREAD_COLOR)

openwam/dataloader/robodojo.py:359-365:

def _decode_jpeg(value: Any, *, source: str) -> Image.Image:
    ...
    with Image.open(io.BytesIO(encoded)) as image:
        return image.convert("RGB").copy()

PIL.Image.open(...).convert("RGB") returns channels in stored order; cv2.imdecode returns BGR, which is what undoes a cv2.imencode of an RGB array. The two paths therefore disagree by an R/B swap whenever both datasets were written by the same kind of encoder.

Why it may matter

git grep -nE 'COLOR_BGR2RGB|COLOR_RGB2BGR|\[\.\.\., ::-1\]|\[:, :, ::-1\]' -- openwam/dataloader/ returns nothing on main, so neither loader corrects afterwards, and the deploy path does no conversion either — openwam/deploy/obs_preprocess.py:142,148 decodes incoming frames with Image.open(...).convert("RGB") and hands them on. So a model trained through robodojo sees whatever PIL yields, and at eval sees whatever the client encoded, with nothing in between to reconcile them.

If RoboDojo's vision/<camera>/colors buffers were written the way robotwin.py describes for RoboTwin — an RGB array passed to cv2.imencode — then robodojo.py trains on channel-swapped frames and evaluates on correct ones.

What I could not determine

Whether that "if" holds. OpenWAM ships no RoboDojo encoder — grep -rn 'imencode|imwrite|encode_jpeg' --include='*.py' openwam/ scripts/ benchmarks/ finds only robotwin.py's comment, benchmarks/ebench/mock_genmanip_server.py (which encodes with turbojpeg TJPF_RGB, i.e. RGB-correct), and one cv.imwrite in the RoboTwin benchmark client that converts to BGR first. So the RoboDojo recorder is external to this repo and I have no RoboDojo trajectories here to test against.

If the RoboDojo buffers are RGB-correct, robodojo.py is right and the only thing missing is a comment saying so — which would still be worth having next to a sibling that documents the opposite convention.

Anyone with a trajectory file can settle it in a few lines:

import io, numpy as np, cv2
from PIL import Image

buf = ...  # one entry from vision/<camera>/colors
pil = np.asarray(Image.open(io.BytesIO(buf)).convert("RGB"))
cv  = cv2.imdecode(np.frombuffer(buf, np.uint8), cv2.IMREAD_COLOR)
print(pil[0, 0], cv[0, 0])   # equal only if the frame is grey at that pixel

then check which ordering matches the scene — a wooden table or a red block makes it obvious by eye.

Related

The same decode is reported downstream against the XPolicyLab adapter in XPolicyLab/XPolicyLab#116, where the eval side is known to be RGB. The code there is a vendored copy of this loader, and robodojo.py's own module docstring notes that the Isaac eval runtime keeps pinned copies of these contracts — so if the swap is real, it is real in both places and worth fixing at the source.

Verified on Python 3.10.20, torch 2.7.1+cu128, at main (f6bae3e). Happy to send a PR — either the cv2.imdecode treatment robotwin.py already uses, or just the clarifying comment — once someone with the data says which way it goes.

[wayrise]

Settled: your "if" holds. The buffers are channel-swapped, robodojo.py was training on them, and I've opened #17.

You were right to file this as a question — but you were also right about the answer, and the asymmetry was enough to get there without the data.

How it was settled, since you asked for a method

I had RoboDojo trajectories on hand. The pixel comparison you suggested shows the two decoders disagree but can't say which is true, and the scenes turned out to be unhelpfully brand-dependent: the real corpus is bottled water and pastel bowls, and a red-capped bottle and a blue-capped one are both entirely plausible. Skin tone would have settled it, but there are no hands in frame.

What settled it is something neither of us mentioned: each corpus ships its own preview videos, at <task>/<embodiment>/preview_video/<episode>_<camera>.mp4, frame-aligned with the HDF5. Those are produced by the recorder for humans to look at, on a path independent of both candidate decodings, which makes them usable as ground truth.

Over saturated pixels only — neutral areas are invariant under an R/B swap and just dilute the numbers:

real corpus (RoboDojo_Real), 10 episodes, both tasks
    mean |preview - cv2.imdecode| =  7.57      <- residual is JPEG vs H.264
    mean |preview - PIL|          = 45.34

sim corpus (benchmark_data/robodojo), production config
    mean |preview - cv2.imdecode| =  1.7
    mean |preview - PIL|          = 40..57

Every episode agrees, on both corpora.

This is not circular. Had the preview been generated by decoding these JPEGs with cv2 and writing them straight out, it would match PIL's reading instead. It matches the other one, which means the recorder held true RGB, converted correctly on the way to the video, and skipped the conversion on the way to cv2.imencode. Precisely the pipeline robotwin.py describes.

The sim corpus also removes the need for any of this arithmetic: the scene is a rendered wooden desk with yellow number tiles, and PIL's reading makes the wood grain blue and the tiles cyan.

Both halves of your train/eval claim check out

I walked each direction rather than reasoning about it:

• Training — injecting a channel swap at _decode_jpeg flips the dataloader's output exactly, in single-view and multiview alike, so nothing between decode and model touches channel order. crop_and_resize, assemble_multiview_layout and VideoColorJitter are all channel-agnostic.
• Deploy — obs_preprocess PIL-decodes the client's bytes and applies the same geometry helpers. Feeding it these raw HDF5 bytes reproduces the PIL reading byte for byte: it performs no conversion of its own, so eval colour is entirely whatever the client sends. Every in-repo client encodes true RGB.

One correction to the write-up, minor and in your favour: obs_preprocess._as_pil doesn't accept ndarray at all — it takes PIL.Image, raw bytes or base64 and rejects everything else. So "pre-decoded RGB arrays used unchanged" describes the XPolicyLab side rather than this server, which reaches the same conclusion by a slightly different route.

What I could not verify

RoboDojo's eval client. benchmarks/robodojo/ is a README pointing at XPolicyLab, so that half is out of reach from here. If it sends true RGB — which the repo's client contract asks for — then XPolicyLab/XPolicyLab#116 is this same bug seen downstream and should clear with #17.

Thanks for filing it as carefully as you did, and for being explicit about what you hadn't verified. That's what made it worth chasing rather than second-guessing.

[wayrise]

Thanks again for raising this.

We test the two sides separately, and the evaluation numbers looked reasonable enough, so the channel-order problem slipped past us. We can no longer reconstruct whether the colleague running our XPolicyLab evaluations aligned the image handling on that side, so we will re-run the full RoboDojo training and evaluation shortly.

What genuinely surprised us is that the model performed as well as it did under OOD conditions this severe. Perhaps we can look forward to better numbers once the fix lands.

Thanks again!