Skip to content

fix(ingest): CRC-validate the primary ingest read, not just the resume path - #462

Merged
kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Sravan1011:fix/431-ingest-crc-validation
Sep 7, 2026
Merged

kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Sravan1011:fix/431-ingest-crc-validation

Conversation

@Sravan1011

@Sravan1011 Sravan1011 commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Closes #431.

Summary

#429 fixed the LeRobot resume path. The same hole was open one stage earlier, on the primary path every ingest takes: transform.py's read of the source file (open_reader) never asked for CRC validation, so a structurally valid MCAP with a damaged chunk payload transcoded without complaint and got a fresh, true receipt over corrupt bytes — #426's finding, one stage upstream.

What changed

  • open_reader / PythonMcapEpisodeReader (reader.py) gain validate_crcs: bool = False, threaded straight into the existing make_reader(..., validate_crcs=...) call — the same shape doctor.py and fix(import): CRC-validate reused landing episodes before stamping receipts #429's fix already use.
  • Default stays False, so app.py:1390, episode.py:311, and runtime/_templates.py:717 are unaffected: those read canonical files HFlow already produced and already identifies by content hash. Only transform.py's ingest read opts in with validate_crcs=True, since that's the one call reading someone else's file for the first time.
  • ingest_ledger.classify_ingest_failure gets a new branch for mcap.stream_reader.CRCValidationError. This is not cosmetic: CRCValidationError subclasses ValueError, not McapError, so without this branch a payload-damaged source would silently fall through to IngestFailureKind.INFRASTRUCTURE — blaming the platform for a bad recording. It classifies to the same SOURCE_UNREADABLE kind as the not-MCAP case; error_type (InvalidMagic vs CRCValidationError) already stays stored verbatim beside the classification, so the two stay distinguishable in the ledger without a new IngestFailureKind member.

Cost

The transcode already decodes every chunk over one full read (reader.iter_batches()), so validate_crcs=True checks a CRC on a pass that was already happening rather than adding one. Measured directly on ingest-scale sources (not inherited from #429's smaller fixture number):

Source size validate_crcs=False validate_crcs=True Ratio
30 MB (2 cameras, 120s, 640x480 @ 15Hz) 269.7 ms 300.1 ms 1.113x
75 MB (2 cameras, 300s, 640x480 @ 15Hz) 783.3 ms 810.7 ms 1.035x

Best-of-5 wall time, filesystem cache warmed identically before each series.

Tests

  • tests/test_ingest_ledger.py::test_classify_crc_validation_error_as_source_unreadable — pins the classifier branch directly.
  • tests/test_ingest_ledger.py::test_ingest_refuses_a_source_with_a_damaged_chunk_payload — a source with a damaged chunk payload (via the shared flip_chunk_payload_bytes fixture from fix(import): CRC-validate reused landing episodes before stamping receipts #429) raises through write_canonical_episode and produces no output file.
  • tests/test_ingest_in_process.py::test_a_payload_damaged_source_is_classified_the_same_as_unreadable — end-to-end through hflow ingest: ledger row lands with failure_kind = source-unreadable, error_type = CRCValidationError.
  • The existing not-an-MCAP tests (InvalidMagic → source-unreadable) are unchanged and still pass, confirming the structurally-invalid case keeps its current classification.

Validation

uv sync --locked --all-extras
uv run ruff check                 # clean
uv run ruff format --check        # clean
uv run ty check                   # clean
uv run pytest -q                  # 1623 passed, 6 skipped

…e path

Hebbian-Robotics#429 fixed the LeRobot resume path; the same hole was open one stage
earlier, on the primary path every ingest takes. transform.py's read of
the source file (open_reader) never asked for CRC validation, so a
structurally valid MCAP with a damaged chunk payload transcoded without
complaint and got a fresh, true receipt over corrupt bytes.

open_reader and PythonMcapEpisodeReader gain validate_crcs (default
False); only the ingest read in transform.py opts in, so the other three
call sites reading HFlow's own canonical output are unaffected. The
transcode already decodes every chunk in one pass, so this piggybacks a
check on a read that already happens: measured 1.113x on a 30MB source
and 1.035x on a 75MB one.

CRCValidationError subclasses ValueError, not McapError, so
classify_ingest_failure needed its own branch for it -- without one it
silently fell through to INFRASTRUCTURE, blaming the platform for a
damaged recording. It classifies to the same SOURCE_UNREADABLE kind as
the not-MCAP case; error_type (InvalidMagic vs CRCValidationError) keeps
the two distinguishable in the ledger without a new enum member.

Refs Hebbian-Robotics#431

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

👋 Hi @Sravan1011 — thank you so much for your first contribution to HFlow!

A maintainer will review your pull request as soon as possible. In the meantime:

💡 Tip: one open pull request per contributor at a time. Issues with an assignee are taken; everything else is fair game.

We are excited to have you here and appreciate your help making the project better! 🙌

@kstonekuan kstonekuan left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, merging.

Checked the MRO rather than taking it: CRCValidationError subclasses ValueError, not McapError, so the classification branch is needed rather than defensive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ingest transcodes a payload-damaged source without noticing, so the canonical episode gets a valid receipt for corrupt bytes

2 participants