Snapshot integrity receipts are written but nothing reads them: no way to verify a delivered snapshot

[kstonekuan]

#401 landed per-file integrity receipts in a snapshot's format.json (#397). Nothing consumes them. The receipt is the half of the feature that survives a transfer; the half that checks it does not exist yet, and the how-to says so in as many words: "there is no public verify_dataset_snapshot API or CLI yet".

Current state

Written at src/hflow/snapshot.py:187:

"integrity": {
  "tables":  {"<name>": {"path": ..., "size_bytes": ..., "sha256": ...}},
  "assets":  [{"path": ..., "size_bytes": ..., "sha256": ...}],
  "content_id": "<sha256 of the normalized inventory>"
}

Read nowhere. The only reader of format.json is the overwrite guard at :595-622, and it looks at format and format_version only. So today the failure this feature exists to catch, a snapshot that arrived truncated or short a table, is silent: DuckDB opens what is there and returns fewer rows.

What to build

A verify_dataset_snapshot(snapshot_directory) beside export_dataset_snapshot, exported from hflow/__init__.py, plus an hflow verify snapshot CLI command. It re-reads the delivered directory and compares it against the receipt.

The helpers are already there and are the right shape to reuse: _sha256_hex at :116, _file_integrity_record at :125, _inventory_content_id at :134.

Four distinct failures worth separating, because a caller does different things about each:

1. A file's bytes changed. sha256 mismatch on a table or an asset.
2. A file is missing. Named in the receipt, absent on disk.
3. A file arrived that the receipt does not name. Extra members under assets/.
4. No receipt at all. A format.json from before feat(snapshot): record table and copied-asset integrity in format.json (#397) #401. This is a valid v1 snapshot and must not be reported as corrupt. Say the receipt is absent and stop.

size_bytes is not a separate failure. It is a cheap pre-filter: compare it first and skip the hash when it already differs, which matters on a copy-mode snapshot holding video.

What it is not

Not a tamper defence, and the API should not imply one. The receipt travels unsigned inside the format.json it describes, so anyone who can rewrite a table can rewrite the hashes to match. This catches corruption, truncation, and partial transfer. If the naming or the docstring makes it sound like a signature, that is a bug in this issue's output.

Design questions worth answering in the PR

• Return or raise. export_dataset_snapshot returns a DatasetSnapshotReport. A verifier that raises on the first mismatch is less useful than one that returns every mismatch it found, because a partial transfer usually damages more than one file and the caller wants the list. A report object with a boolean and a list of typed findings fits the repo better than an exception, and gives the CLI something to print. Argue for whichever you pick.
• Exit code. The CLI convention here is 2 for bad input. A snapshot that fails verification is not bad input, it is a true answer to the question asked. Pick a code, say why.
• Cost. Verification is a full re-read of every delivered byte. On a copy-mode snapshot that is the media again. Consider whether a --quick mode that checks sizes and the member set without hashing is worth having, and whether its result is honest enough to report as "verified".

Definition of done

1. The four failure classes above are each covered by a test that fails for that reason alone.
2. A clean snapshot verifies clean, in both references and copy mode.
3. A pre-feat(snapshot): record table and copied-asset integrity in format.json (#397) #401 format.json with no integrity key is reported as unverifiable, not as corrupt, and does not raise.
4. The how-to's "no public API or CLI yet" paragraph is replaced with usage.
5. Public API additions are exported and appear in __all__.

Validation

uv sync --locked --all-extras
uv run ruff check
uv run ruff format --check
uv run ty check
uv run pytest -q

Markdown changes also want the lychee command from CONTRIBUTING.md.

[Sagar-024]

Good issue, I will take this one. We have to focus on testing every different type of failure so not a single one remains hidden, and the customer either gets their receipt or at least knows the exact reason.

[Sravan1011]

@Sagar-024 can I work on this ?

[kstonekuan]

Assigned to @Sagar-024, who claimed it first and with a plan. In this repo the assignee is the only thing that reserves an issue, so that settles it.

@Sravan1011 sorry, this one is taken. Two honest options, because I would rather tell you the real state than invent something to hand you:

The good first issue pool is empty right now. There were two open this morning and both were finished within hours. I am not going to file filler to have something on the shelf, so the next one appears when something worth doing turns up, usually out of a review. Watching the good first issue label is the fastest way to see one land.

In the meantime the most useful thing anyone can do here needs no issue at all: #287. Point HFlow at a real corpus such as Egocentric-10K or Egocentric-100K, and report what breaks, what is slow, and what is awkward. Nothing is assigned, nothing is a race, and a specific failure with a reproduction is worth more to us than an issue we can write from the inside, because we cannot see those failures from here. Several of this week's merged fixes started as exactly that kind of report.

If you would rather wait for a starter issue, that is fine too. Say so and I will point you at the next one.

[Sravan1011]

should I go ahead with 287?

[Sagar-024]

@Sravan1011 yes go ahead, but keep it small. The original #287 fix already merged in #334, so you are not fixing that. You are doing what Galabavamsi did: run the current pipeline on a few real episodes and report what still breaks, is slow, or feels awkward. The corpora are gated on Hugging Face, so request access first and start with one small shard, not the whole 10K. Post your report here and I will read it and help you separate real bugs from config issues.