Skip to content

fix: refuse canonicals with invalid zstd chunks - #516

Merged
kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Kaileshwar16:fix/506-canonical-decompression
Sep 13, 2026
Merged

kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
Kaileshwar16:fix/506-canonical-decompression

Conversation

@Kaileshwar16

Copy link
Copy Markdown
Contributor

Fixes #506.

A canonical with a corrupted zstd-compressed chunk currently escapes verify_canonical_integrity() as ZstdError, escapes App.process(), and is eventually recorded as an infrastructure failure.

I reproduced the failure first and confirmed the actual exception is:

zstandard.backend_c.ZstdError:
error determining content size from frame header

MCAP does not wrap it.

This change handles that precise failure at the canonical integrity boundary and refuses the episode with:

canonical-decompression-failed

The refusal uses the same framework-owned integrity/canonical path introduced for CRC failures in #502.

Behavior

After this change:

  • undecompressable zstd chunks are treated as damaged canonical data rather than infrastructure failures
  • refusal_reason is set to canonical-decompression-failed
  • an integrity/canonical error row is recorded
  • user checks and enrichments do not run
  • CRC corruption behavior is unchanged
  • unrelated filesystem/read failures still propagate as infrastructure failures

The implementation catches only zstandard.ZstdError; there is no broad exception handling.

Reproduction

The regression fixture starts from a valid canonical and changes only the first byte of the first compressed zstd frame:

0x28 -> 0x29

The stored CRC, MCAP structure, indexes, footer, lengths, and compression metadata remain unchanged, so the failure reaches the actual zstd decompressor.

Tests

Added coverage for:

  • zstd decompression corruption producing the named refusal
  • no user check or enrichment running after the integrity refusal
  • CRC and decompression corruption both being discoverable through check_runs.error
  • damaged canonicals not producing infrastructure ledger entries
  • missing canonicals still remaining failure_kind='infrastructure'

Mutation validation was also run by temporarily removing the new decompression branch; the new regression test failed with the original ZstdError, then passed again after restoring the branch.

Validation

New regression tests:                  3 passed
Adjacent ingest suites:                26 passed
#502 integrity tests:                  4 passed
ruff check:                            passed
ruff format --check:                   passed
ty check:                              passed
full test suite:                       1,922 passed, 7 skipped
git diff --check:                      passed

@kstonekuan kstonekuan left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, merging.

One mutation I am merging past: widening the catch to except Exception passes all 7 tests. Path.open sits outside the try, so test_missing_canonical_remains_an_infrastructure_failure cannot constrain the catch width. Worth a cheap pin next time you are in the file.

#521 is assigned to you.

@kstonekuan
kstonekuan merged commit c79d910 into Hebbian-Robotics:main Sep 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A canonical whose chunk zstd cannot decompress is still an undiagnosed exception, not a named refusal

2 participants