fix(data): detect unusable dataset media before training#84
Merged
Conversation
A dataset checkout without Git LFS materialized leaves small text pointer files where the audio should be. Reading one previously failed as an opaque decode error deep in feature extraction, which is a confusing way to learn the checkout is incomplete. Detect the pointer prefix before decoding and raise AudioIntegrityError naming the repair steps. Decode failures now raise AudioDecodeError chained to the underlying librosa/soundfile cause instead of discarding it. Both subclass OSError, so existing handling is unaffected.
Validates every candidate WAV before it reaches a manifest: rejecting LFS pointers, empty and implausibly small files, and anything whose metadata will not decode. Registered datasets are re-validated when consumed, so a checkout that degrades after download is still caught. Preparation now also requires git and git-lfs up front, rather than failing later with an unclear error once the audio turns out to be pointers.
jsugg
force-pushed
the
fix/dataset-media-integrity
branch
from
July 15, 2026 21:14
b6c5d28 to
a410d8f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes an incomplete or damaged dataset checkout fail early, at preparation time,
with a message that names the fix — instead of surfacing as an opaque decode
error deep inside feature extraction, hours into a run.
Git LFS pointers (
_internal/utils/audio_utils.py)A checkout without Git LFS materialized leaves ~130-byte text pointer files
exactly where the audio should be. These are perfectly readable files, so the
failure previously looked like corrupt audio rather than a setup problem.
version https://git-lfs.github.com/spec/v1prefix before decodingand raise
AudioIntegrityErrornaming the repair (git lfs pull/checkout).AudioDecodeErrorchained (raise ... from) to theunderlying librosa/soundfile cause, which was previously discarded.
OSError, so existing callers keep working unchanged. Thistyping is what lets the readiness work distinguish a sample-local decode
failure (quarantinable) from a broken checkout (must abort).
CREMA-D validation (
adapters/crema_d.py,strategies/default.py,dataset_prepare.py)empty files, files below a plausible WAV header size, and anything whose
metadata will not decode or reports no frames.
an empty manifest silently.
degrades after download is still caught.
gitandgit-lfsare required up front with an actionable message, ratherthan failing later once the audio turns out to be pointers.
Verification
make lint,make type(mypy + pyright clean), and the full suite — 1077 tests— pass locally. Pre-push gate passed.