data: add released v23 holdout datasets to public registry (18 mirrors) - #146
Merged
Conversation
The 18 v23 holdouts that had no usable upstream repo and were mirrored to
34data (bmcore PR #359). Uploaded and verified: 18/18 repos hold data,
56.92 GB across 326k source files.
real_images +7 medical imaging (kidney-stone-ct, malaria, pulmonary
chest x-ray, dermnet, coronahack, lits-256, tcga-coad)
synthetic_images +3 synthetic-human, ai-generated-ecommerce x2
synthetic_videos +7 VideoGen-RewardBench x5 (easyanimate/opensora1.2/
qingying/tongyi/vidu) + edtalk, float-talking-head
synthetic_audio +1 indic-tts-urdu
Archive bounds are per-run download limits, not sample counts — the per-dataset
sample cap comes from calculate_weighted_dataset_sampling. In full mode the YAML
values are respected (DOWNLOAD_SIZE_OVERRIDES is debug/small only), so -1/-1
would fetch each dataset in full every run to use a few hundred samples, and
would also lose variety: _select_files_to_download draws a seeded random subset
when the count is bounded, but returns everything when it is -1. Bounded values
give per-run economy and rotate archives across rounds as the seed changes.
image 1000/1 -> 1000 available vs 355 cap
video 500/1 -> 500 available vs 149 cap
audio 500/2 -> 1000 available vs 336 cap
For these 18 that is 56.9 GB -> 20.6 GB per full run (64% less), concentrated in
synthetic-human (16.6 -> 2.4), float-talking-head (13.1 -> 4.4), tcga (5.4 ->
1.8) and edtalk (5.3 -> 1.8). Video entries declare source_format: mp4 but hold
zips; the loader's fallback recalculates n_files on the archive branch, so
archives_per_dataset governs.
Raise full-mode targets so 18 new datasets do not dilute the existing ones:
image 69000->72500, video 33000->34000, audio 47000->47500. Per-dataset caps are
held at 355 / 149 / 336 (audio +1), unchanged from before this wave.
edtalk and float-talking-head keep media_type: semisynthetic, joining the 29
existing semisynthetic video entries.
The other 33 v23 entries are not here: 22 defer to existing ungated upstream
repos and will land separately with hf_revision pinned, 7 have gated upstreams
and still need mirroring, and 4 wild-* stay withheld.
Registry loads clean at 565 datasets, no duplicate names or paths, and all 18
paths resolve on HuggingFace with data.
This was referenced Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds 18 v23 datasets to the public registry, mirrored to
34data/*.real_images.yamlsynthetic_images.yamlsynthetic_videos.yamlsynthetic_audio.yamlArchive bounds are per-run download limits; the per-dataset sample cap comes from
calculate_weighted_dataset_sampling. Values follow the registry convention — image1000/1, video500/1, audio500/2— each clearing its modality's cap.Full-mode targets scale with the registry so added datasets don't reduce existing per-dataset representation: image 69000→72500, video 33000→34000, audio 47000→47500. Per-dataset caps are unchanged at 355 / 149 / 336.
edtalkandfloat-talking-headkeepmedia_type: semisynthetic.Verified: registry loads at 565 datasets with no validation failures, no duplicate names or paths, and all 18 paths resolve on HuggingFace with data.