data: register 22 v23 holdouts against upstream repos (no 34data mirror) - #147
Merged
Conversation
Second half of the v23 release. These 22 holdouts already exist in public upstream datasets, so instead of copying Wasabi -> 34data they point at the upstream repo directly. Same approach already used for cml-tts-german/spanish, ai4bharat-speecharenabench-kn/mr and indic-tts-kannada/malayalam. ylacombe/cml-tts 4 dutch, italian, polish, portuguese Sumsub/Swappir 5 GPEN x2, roop x2, simswap scaledf/ScaleDF 4 FFpp Deepfakes/Face2Face, E4E, DiffusionCLIP RekaAI/RekaDaily-10k-raw 3 phone shards 00000-00002 obscure-entropy/conceptual_captions_hu_filtered 1 eltorio/ROCOv2-radiology 1 rshaojimmy/DGM4 1 manipulation/StyleCLIP.zip MCG-NJU/TimeLens2-93K 1 videos-00003-of-00018.tar mvp-lab/LLaVA-OneVision-2-Data 1 mid_training_video/30s WenhaoWang/TIP-I2V 1 i2vgenxl_videos_subset_1.tar Upstream filenames match the Wasabi path segments exactly, so these resolve to the same material that was held out. Every repo is ungated and every entry pins hf_revision, so an upstream force-push cannot silently change what the benchmark reads. include_paths is a substring match, which is a trap here: 'CelebA_HQ_roop.zip' also matches 'GPEN_CelebA_HQ_roop.zip', and 'fairface_roop.zip' also matches 'GPEN_fairface_roop.zip'. Both carry exclude_paths: ['GPEN_'] so they resolve to one file each. Every include_paths was checked against the pinned revision's actual file listing: 20 resolve to exactly 1 archive, rocov2 to its 27 train shards, cml-tts to 373/61/12/42 per language. ScaleDF is pinned to scaledf/ScaleDF (the org repo) rather than WenhaoWang/ScaleDF; both carry all four tars. Archive bounds follow the same reasoning as the mirrored half — image 1000/1, video 500/1, audio 500/2, all clearing the per-dataset caps, which land at 335 / 145 / 327 with these 22 added. Registry loads clean at 587 datasets, no validation failures, no duplicate names and no duplicate (path, include_paths) pairs introduced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Registers 22 v23 datasets against their existing upstream HuggingFace repos instead of mirroring them to
34data/*. Same pattern already used forcml-tts-german/spanish,ai4bharat-speecharenabench-kn/mrandindic-tts-kannada/malayalam.ylacombe/cml-ttsSumsub/Swappirscaledf/ScaleDFRekaAI/RekaDaily-10k-rawobscure-entropy/conceptual_captions_hu_filteredeltorio/ROCOv2-radiologyrshaojimmy/DGM4MCG-NJU/TimeLens2-93Kmvp-lab/LLaVA-OneVision-2-DataWenhaoWang/TIP-I2VEvery entry pins
hf_revision. Archive bounds match the convention used elsewhere in the registry: image1000/1, video500/1, audio500/2.include_pathsis a substring match, so the two swappir entries whose names are prefixes of other files carryexclude_paths: ['GPEN_'].Verified: registry loads at 587 datasets with no validation failures, no duplicate names or
(path, include_paths)pairs, and everyinclude_pathsresolves to the intended files at the pinned revision.