Skip to content

fix(data): detect unusable dataset media before training#84

Merged
jsugg merged 2 commits into
mainfrom
fix/dataset-media-integrity
Jul 15, 2026
Merged

fix(data): detect unusable dataset media before training#84
jsugg merged 2 commits into
mainfrom
fix/dataset-media-integrity

Conversation

@jsugg

@jsugg jsugg commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Makes an incomplete or damaged dataset checkout fail early, at preparation time,
with a message that names the fix — instead of surfacing as an opaque decode
error deep inside feature extraction, hours into a run.

Git LFS pointers (_internal/utils/audio_utils.py)

A checkout without Git LFS materialized leaves ~130-byte text pointer files
exactly where the audio should be. These are perfectly readable files, so the
failure previously looked like corrupt audio rather than a setup problem.

  • Detect the version https://git-lfs.github.com/spec/v1 prefix before decoding
    and raise AudioIntegrityError naming the repair (git lfs pull / checkout).
  • Decode failures now raise AudioDecodeError chained (raise ... from) to the
    underlying librosa/soundfile cause, which was previously discarded.
  • Both subclass OSError, so existing callers keep working unchanged. This
    typing is what lets the readiness work distinguish a sample-local decode
    failure (quarantinable) from a broken checkout (must abort).

CREMA-D validation (adapters/crema_d.py, strategies/default.py, dataset_prepare.py)

  • Every candidate WAV is validated before it reaches a manifest: LFS pointers,
    empty files, files below a plausible WAV header size, and anything whose
    metadata will not decode or reports no frames.
  • An empty glob is now an error — previously an empty CREMA-D tree could produce
    an empty manifest silently.
  • Registered datasets are re-validated when consumed, so a checkout that
    degrades after download is still caught.
  • git and git-lfs are required up front with an actionable message, rather
    than failing later once the audio turns out to be pointers.

Verification

make lint, make type (mypy + pyright clean), and the full suite — 1077 tests
— pass locally. Pre-push gate passed.

jsugg added 2 commits July 15, 2026 18:14
A dataset checkout without Git LFS materialized leaves small text pointer files
where the audio should be. Reading one previously failed as an opaque decode
error deep in feature extraction, which is a confusing way to learn the
checkout is incomplete.

Detect the pointer prefix before decoding and raise AudioIntegrityError naming
the repair steps. Decode failures now raise AudioDecodeError chained to the
underlying librosa/soundfile cause instead of discarding it. Both subclass
OSError, so existing handling is unaffected.
Validates every candidate WAV before it reaches a manifest: rejecting LFS
pointers, empty and implausibly small files, and anything whose metadata will
not decode. Registered datasets are re-validated when consumed, so a checkout
that degrades after download is still caught.

Preparation now also requires git and git-lfs up front, rather than failing
later with an unclear error once the audio turns out to be pointers.
@jsugg
jsugg force-pushed the fix/dataset-media-integrity branch from b6c5d28 to a410d8f Compare July 15, 2026 21:14
@jsugg
jsugg merged commit 63ba606 into main Jul 15, 2026
12 checks passed
@jsugg
jsugg deleted the fix/dataset-media-integrity branch July 15, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant