From 6ad80dba9cefe4b4d75d9ca1045b8634429cfe7b Mon Sep 17 00:00:00 2001 From: dylan Date: Sat, 5 Sep 2026 00:29:52 +0000 Subject: [PATCH] data: add released v23 holdout datasets to public registry (18 mirrors) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 18 v23 holdouts that had no usable upstream repo and were mirrored to 34data (bmcore PR #359). Uploaded and verified: 18/18 repos hold data, 56.92 GB across 326k source files. real_images +7 medical imaging (kidney-stone-ct, malaria, pulmonary chest x-ray, dermnet, coronahack, lits-256, tcga-coad) synthetic_images +3 synthetic-human, ai-generated-ecommerce x2 synthetic_videos +7 VideoGen-RewardBench x5 (easyanimate/opensora1.2/ qingying/tongyi/vidu) + edtalk, float-talking-head synthetic_audio +1 indic-tts-urdu Archive bounds are per-run download limits, not sample counts — the per-dataset sample cap comes from calculate_weighted_dataset_sampling. In full mode the YAML values are respected (DOWNLOAD_SIZE_OVERRIDES is debug/small only), so -1/-1 would fetch each dataset in full every run to use a few hundred samples, and would also lose variety: _select_files_to_download draws a seeded random subset when the count is bounded, but returns everything when it is -1. Bounded values give per-run economy and rotate archives across rounds as the seed changes. image 1000/1 -> 1000 available vs 355 cap video 500/1 -> 500 available vs 149 cap audio 500/2 -> 1000 available vs 336 cap For these 18 that is 56.9 GB -> 20.6 GB per full run (64% less), concentrated in synthetic-human (16.6 -> 2.4), float-talking-head (13.1 -> 4.4), tcga (5.4 -> 1.8) and edtalk (5.3 -> 1.8). Video entries declare source_format: mp4 but hold zips; the loader's fallback recalculates n_files on the archive branch, so archives_per_dataset governs. Raise full-mode targets so 18 new datasets do not dilute the existing ones: image 69000->72500, video 33000->34000, audio 47000->47500. Per-dataset caps are held at 355 / 149 / 336 (audio +1), unchanged from before this wave. edtalk and float-talking-head keep media_type: semisynthetic, joining the 29 existing semisynthetic video entries. The other 33 v23 entries are not here: 22 defer to existing ungated upstream repos and will land separately with hf_revision pinned, 7 have gated upstreams and still need mirroring, and 4 wild-* stay withheld. Registry loads clean at 565 datasets, no duplicate names or paths, and all 18 paths resolve on HuggingFace with data. --- src/gasbench/dataset/config.py | 2 +- src/gasbench/dataset/configs/real_images.yaml | 72 +++++++++++++++++++ .../dataset/configs/synthetic_audio.yaml | 12 ++++ .../dataset/configs/synthetic_images.yaml | 32 +++++++++ .../dataset/configs/synthetic_videos.yaml | 72 +++++++++++++++++++ 5 files changed, 189 insertions(+), 1 deletion(-) diff --git a/src/gasbench/dataset/config.py b/src/gasbench/dataset/config.py index a814e2d..5c5c2b3 100644 --- a/src/gasbench/dataset/config.py +++ b/src/gasbench/dataset/config.py @@ -25,7 +25,7 @@ BENCHMARK_TOTAL_OVERRIDES = { "debug": {"image": 100, "video": 50, "audio": 50}, "small": {"image": 5000, "video": 5000, "audio": 5000}, - "full": {"image": 69000, "video": 33000, "audio": 47000}, + "full": {"image": 72500, "video": 34000, "audio": 47500}, } # Per-dataset download limits (only applied in debug/small modes for faster testing) diff --git a/src/gasbench/dataset/configs/real_images.yaml b/src/gasbench/dataset/configs/real_images.yaml index 3a9afac..39f68f3 100644 --- a/src/gasbench/dataset/configs/real_images.yaml +++ b/src/gasbench/dataset/configs/real_images.yaml @@ -1151,3 +1151,75 @@ datasets: generator_family: real content_category: documents notes: Released from v22 bmcore holdouts + + # ── Released v23 holdouts ───────────────────────────────────────────────── + + - name: kidney-stone-ct + path: 34data/kidney-stone-ct + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: malaria-bounding-boxes + path: 34data/malaria-bounding-boxes + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: pulmonary-chest-xray-abnormalities + path: 34data/pulmonary-chest-xray-abnormalities + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: dermnet + path: 34data/dermnet + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: coronahack-chest-xray + path: 34data/coronahack-chest-xray + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: lits-256 + path: 34data/lits-256 + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical + + - name: tcga-coad-msi-mss + path: 34data/tcga-coad-msi-mss + modality: image + media_type: real + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: real + content_category: medical diff --git a/src/gasbench/dataset/configs/synthetic_audio.yaml b/src/gasbench/dataset/configs/synthetic_audio.yaml index c285ba2..da74040 100644 --- a/src/gasbench/dataset/configs/synthetic_audio.yaml +++ b/src/gasbench/dataset/configs/synthetic_audio.yaml @@ -796,3 +796,15 @@ datasets: generator_family: mixed-generation content_category: speech notes: Synthetic subset of the official RuASD anti-spoofing dataset + + # ── Released v23 holdouts ───────────────────────────────────────────────── + + - name: indic-tts-urdu + path: 34data/indic-tts-urdu + modality: audio + media_type: synthetic + source_format: zip + media_per_archive: 500 + archives_per_dataset: 2 + generator_family: mixed-generation + content_category: speech diff --git a/src/gasbench/dataset/configs/synthetic_images.yaml b/src/gasbench/dataset/configs/synthetic_images.yaml index a031471..ef12539 100644 --- a/src/gasbench/dataset/configs/synthetic_images.yaml +++ b/src/gasbench/dataset/configs/synthetic_images.yaml @@ -942,3 +942,35 @@ datasets: notes: PICA-100K is purely synthetic (PICABench; video-derived src and edited tgt). Not split — src is not camera-real. generator_family: mixed-generation content_category: diverse + + # ── Released v23 holdouts ───────────────────────────────────────────────── + + - name: synthetic-human + path: 34data/synthetic-human + modality: image + media_type: synthetic + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: mixed-generation + content_category: faces + + - name: ai-generated-ecommerce-damaged-product + path: 34data/ai-generated-ecommerce-damaged-product + modality: image + media_type: synthetic + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: mixed-generation + content_category: diverse + + - name: ai-generated-ecommerce-fake-logistics + path: 34data/ai-generated-ecommerce-fake-logistics + modality: image + media_type: synthetic + source_format: parquet + media_per_archive: 1000 + archives_per_dataset: 1 + generator_family: mixed-generation + content_category: diverse diff --git a/src/gasbench/dataset/configs/synthetic_videos.yaml b/src/gasbench/dataset/configs/synthetic_videos.yaml index 7ed8cb6..eb997b8 100644 --- a/src/gasbench/dataset/configs/synthetic_videos.yaml +++ b/src/gasbench/dataset/configs/synthetic_videos.yaml @@ -1266,3 +1266,75 @@ datasets: generator_family: mixed-generation content_category: diverse notes: Released from v22 bmcore holdouts + + # ── Released v23 holdouts ───────────────────────────────────────────────── + + - name: videogen-rewardbench-easyanimatev4 + path: 34data/videogen-rewardbench-easyanimatev4 + modality: video + media_type: synthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: easyanimate + content_category: diverse + + - name: videogen-rewardbench-opensora1-2 + path: 34data/videogen-rewardbench-opensora1-2 + modality: video + media_type: synthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: opensora + content_category: diverse + + - name: videogen-rewardbench-qingying + path: 34data/videogen-rewardbench-qingying + modality: video + media_type: synthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: qingying + content_category: diverse + + - name: videogen-rewardbench-tongyi + path: 34data/videogen-rewardbench-tongyi + modality: video + media_type: synthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: tongyi + content_category: diverse + + - name: videogen-rewardbench-vidu + path: 34data/videogen-rewardbench-vidu + modality: video + media_type: synthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: vidu + content_category: diverse + + - name: edtalk + path: 34data/edtalk + modality: video + media_type: semisynthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: edtalk + content_category: faces + + - name: float-talking-head + path: 34data/float-talking-head + modality: video + media_type: semisynthetic + source_format: mp4 + media_per_archive: 500 + archives_per_dataset: 1 + generator_family: float + content_category: faces