feat(training): add mandatory training readiness and preparation contract#85
Merged
Conversation
Schema v2 keeps v1 JSONL readable and makes categorical emotion optional, so a corpus can contribute dimensional or auxiliary targets without being forced into an eight-class label it does not have. Rows can now carry normalized VAD, social attitude, binary affect, language, transcript, per-target source and confidence, speaker/session identity, native split, segment bounds, normalized-PCM SHA-256, dataset revision, license, and source provenance. Callers that require a categorical label ask for it through require_label() rather than assuming it is present.
Concatenating every configured manifest silently mixes incompatible label ontologies and leaks speakers across splits. A recipe instead declares, per corpus, which labels are exact enough for the public eight-class head, which are only approximate, and which auxiliary tasks a corpus can serve. Approximate labels route to raw_emotion and never reach the primary head, so calm is never inferred from neutral or sleepy. The audit deduplicates by normalized-PCM hash, unions speaker and session ids into identity components, assigns deterministic grouped splits, and refuses to let a group or a content hash appear in more than one supervised split. Rows with no defensible group become ssl_only and cannot enter validation or test. Recipe, manifest, and split-ledger SHA-256 digests make a benchmark reproducible; strict mode rejects missing revisions or hashes, duplicate content, leakage, and a primary task with fewer than two populated classes.
Adds the settings the recipe and readiness work reads: dataset recipe selection and strict auditing, bias-aware quarantine budgets, and a development split size. Strict auditing defaults on whenever a recipe is configured, since a recipe exists to produce a defensible benchmark. The quarantine budgets are deliberately several narrow limits rather than one global ratio: a single ratio can be satisfied while silently deleting one corpus or one class. Public API surface additions are config attributes only.
split_utterances resolved partition membership with `utterance in <list>`, making speaker partitioning quadratic. Across the ~152k-utterance inventory this extrapolated to roughly nine hours and read as a hang rather than as slow code. Membership now tests against a hashed set. Also adds split_utterances_three_way, which preserves complete native train/dev/test assignments when a manifest carries them and otherwise reserves a deterministic, speaker-isolated development partition.
Persists what a run was actually trained from: recipe and split-ledger digests, model revision, task heads, sampling policy, seed, and evaluation summary. All are validated on read, so a malformed digest or a non-JSON policy is rejected rather than silently trusted. The loader accepts both v2 and v3 envelopes and the public eight-emotion inference schema is unchanged. Newly written artifacts are v3, so an older release cannot read them; reading existing v2 artifacts is unaffected.
Training could previously fail most of the way through a long run for reasons knowable up front: missing media, an infeasible split, no write permission, a full disk. Readiness checks those first and reports every problem at once rather than failing on the first. Failures carry a stable scope, reason, severity, and disposition. Broken checkouts, invalid configuration, leakage, insufficient class support, non-finite backend output, and resource shortages always abort. Only known sample-local decode failures or a missing path proven inside a registered root may be quarantined, and every proposed exclusion must satisfy the absolute, global, per-corpus, per-class, per-reason, and minimum-remaining budgets, so containment cannot silently delete a class or a corpus. Reports publish atomically; no partial document is ever treated as ready.
Routes every training entrypoint through one readiness contract, then lets the expensive part be done once and reused. Preparation writes a non-pickle NPZ feature payload and publishes a plan only after the payload lands, so an interrupted run never leaves a plan pointing at a partial payload. A prepared plan is only accepted when it still describes reality: canonical plan and payload digests, profile and settings, backend and model revision, device and dtype, registry, manifests, media fingerprints, recipe, split and quarantine ledgers, cache namespace, and the actual sample and window ledgers. Backend plans require the exact commit resolved by the checked backend, so a mutable tag cannot mint a reusable plan. Any mismatch is a validation failure rather than a silently different run. Also wires dataset recipe routing into utterance loading and reports bounded progress with a credible ETA during preparation.
Lets an expensive training run be validated before it starts and its preparation reused afterwards. --dry-run checks readiness and never fits, caches, or writes artifacts. --prepare-only materializes reusable features. --prepared-plan consumes a digest-validated plan. --repair is accepted only alongside the first two. Conflicting or non-training options now fail during argument validation, before any registry, backend, or feature work. Validation and policy failures exit 2 and unexpected failures exit 1 with a traceback, so a caller can distinguish "you asked for something impossible" from "we broke". The traceback is taken from the classified exception rather than from ambient exception state.
Surfaces registry and training readiness findings through `ser doctor` and the training preflight, so the checks are reachable before committing to a run. --preflight off still disables only the general startup preflight; it never disables the training readiness contract itself.
Documents the readiness contract and its side-effect boundaries, the repair allowlist, the fault-containment budgets, and the versioned recipe workflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The foundation for reproducible, restartable training runs: versioned dataset
recipes, a leakage-safe split audit, a mandatory readiness contract, and
reusable prepared plans.
This is the large one, and it is deliberately atomic — readiness, orchestration
and the entrypoint wiring are one behavioural change, and splitting the contract
from its wiring would produce something that cannot be evaluated. It is
organized into 10 reviewable commits; the smaller, separable pieces already
landed ahead of it as #81, #82, #83 and #84.
Why
Training could fail most of the way through a long run for reasons knowable up
front — missing media, an infeasible split, no write permission, a full disk —
and concatenating every configured manifest silently mixed incompatible label
ontologies and leaked speakers across splits.
Dataset recipes and auditing
binary affect, transcript, speaker/session identity, native split, content
hash, revision and license. v1 manifests still load.
eight-class head, which are approximate, and which auxiliary tasks it serves.
Approximate labels route to
raw_emotionand never reach the primary head, socalmis never inferred from neutral or sleepy.identity components, assigns deterministic grouped splits, and refuses to let a
group or content hash appear in two supervised splits. Rows with no defensible
group become
ssl_onlyand cannot enter validation or test.Opt-in:
--dataset-recipe research-v1/SER_DATASET_RECIPE. The default pathis unchanged.
Readiness and containment
Readiness reports every problem at once rather than failing on the first.
Failures carry a stable scope, reason, severity and disposition. Broken
checkouts, invalid config, leakage, insufficient class support, non-finite
backend output and resource shortages always abort. Only known sample-local
decode failures — or a missing path proven inside a registered root — may be
quarantined, and every exclusion must satisfy the absolute, global, per-corpus,
per-class, per-reason and minimum-remaining budgets. Several narrow budgets
rather than one global ratio, because a single ratio can be satisfied while
silently deleting one class or one corpus.
Modes
--dry-runvalidates and never fits, caches, or writes artifacts.--prepare-onlywrites a non-pickle NPZ payload, then publishes a plan onlyafter the payload lands, so an interrupt never leaves a plan pointing at a
partial payload.
--prepared-planre-verifies every digest, the backend commit, and the actualsample/window ledgers. Any mismatch exits
2rather than silently runningsomething different.
--repairis allowlisted, idempotent and recorded; accepted only with thefirst two.
--preflight offdisables only the general startup preflight; it never disablesthe readiness contract.
Compatibility
new artifact.
2; unexpected failures still exit1.Notable fix included
split_utterancesresolved partition membership withutterance in <list>,making speaker partitioning quadratic. Across the ~152k-utterance inventory that
extrapolated to roughly nine hours and read as a hang. Now linear.
Verification
make lint,make type(mypy + pyright clean across 412 files) and the fullsuite — 1207 tests — pass locally. Pre-push gate passed.
Known follow-ups (not in scope here)
call sites do not populate them yet; digests currently flow into the prepared
plan.
audit_dataset_recipequarantines whole duplicate clusters rather thanresolving them per cluster.