fix(checks): validate canonical CRCs once at lane entry with memoization - #502
Merged
kstonekuan merged 1 commit intoSep 11, 2026
Merged
Conversation
Every post-sync read ran with chunk CRC validation off, so a canonical that decayed on disk after sync was re-certified by the lanes that exist to judge it: fresh measured findings stamped over bytes the file's own integrity stamp refuses (Hebbian-Robotics#474). App.process now verifies the canonical's chunk CRCs once per run at the front door every consuming lane shares, before any step runs. A file that fails its own stamp is refused with the named reason canonical-crc-mismatch: one report field, one framework-owned check_runs row (critical, so the curation views read the episode as unverified), and no check, enrichment, or media step. The refusal covers the meta lane, the relabel lane, and the production meta task an online re-check flows through; a run with no step work to do pays no strict read. The verdict is memoized on the App keyed by path plus a size+mtime witness, so repeated stage runs and retried batches in one process pay for one strict read, and bytes that change under the cache are re-validated rather than trusted. Replays of the same refusal dedupe through the existing run fingerprint, keeping one record per episode. Episode._reader's default and the snapshot verifier are untouched.
kstonekuan
approved these changes
Sep 11, 2026
Contributor
There was a problem hiding this comment.
LGTM, merging.
assert refused.checks == [] makes this the fix I asked for: zero tracebacks because nothing ran, not because something caught them. And asserting episodes.status == 'unverified' from the catalog rather than trusting critical=True.
I owe you a measurement, since the front-door shape was my call and costs a second read: 0.5% of a process call at 0.4 MB, 0.2% at 6.3 MB. It gets cheaper relative to ffmpeg as episodes grow.
Filed the zstd-refusing half as #506.
Contributor
Author
|
On the corpus invitation: Discussion #503 (the field report I just posted) is exactly that work. 128 real episodes, 3 factories, 6 stress passes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #474.
Bug: post-sync reads open canonicals with CRC validation off, so a canonical that decayed on disk gets re-measured and re-certified. Checks stamp fresh
measuredrows over bytes the file's own integrity stamp refuses.Fix: a front-door guard, not a reader flag flip.
verify_canonical_integrity(path)in reader.py does one strict read withvalidate_crcs=Trueand catchesCRCValidationErrorby its precise type.App.processcalls it once, before any step runs, in every lane that spends work on canonical bytes: META, relabel, and the production re-check (process_stage_batch). On mismatch the episode is refused with ONE diagnosis:TestReport.refusal_reasonplus one framework-ownedcheck_runsrow (error = 'canonical-crc-mismatch', critical, curation views read it asunverified). No check runs, so zero tracebacks is structural.hflow ingestexits 1, the runtime-finding family.Decisions worth your eye:
Validation: ruff, format, ty clean on the three changed files. Full suite 1871 passed, 6 skipped on the rebased commit. Mutation red then green. ASCII swept.