Skip to content

Write pushed features to card when re-pushing a split - #8609

Open
belambert wants to merge 1 commit into
huggingface:mainfrom
belambert:fix-stale-features-on-repush
Open

Write pushed features to card when re-pushing a split#8609
belambert wants to merge 1 commit into
huggingface:mainfrom
belambert:fix-stale-features-on-repush

Conversation

@belambert

@belambert belambert commented Sep 12, 2026

Copy link
Copy Markdown

Fixes #8608.

_get_updated_dataset_card updates the splits and sizes of the existing dataset_info when
a config is re-pushed, but never assigns the pushed features to it. Re-pushing a dataset's
only split with different columns therefore leaves the card declaring the previous schema
while the shards hold the new one, and the dataset fails to cast on load:

CastError: Couldn't cast ... because column names don't match

The existing mismatch guard doesn't catch this because it only runs when other splits are
present (any(s != split for s in repo_info.splits)), which is never true for a single-split
dataset being replaced. The card ends up half-updated - num_examples from the new push,
features from the old one - so the push looks like it worked.

The change

            repo_info.splits[split] = split_info
+       repo_info.features = features
        info_to_dump = repo_info

Reaching that line with other splits on the repo means the features already matched, or the
guard above would have raised, so the assignment is a no-op in that case. The only behaviour
change is the case the guard deliberately skips: a push that replaces every split the card
describes, where the pushed features are the authoritative ones.

Both Dataset.push_to_hub and DatasetDict.push_to_hub go through this function, so one
change covers both.

Tests

Two unit tests alongside the existing _get_updated_dataset_card tests in
tests/test_buckets.py, using the same in-memory filesystem pattern:

  • test_get_updated_dataset_card_updates_features_when_replacing_sole_split - the new
    features reach the card, and the split count still updates. Fails on main, passes here.
  • test_get_updated_dataset_card_rejects_features_mismatch_against_other_splits - the
    existing ValueError still raises when a second split would be left disagreeing.

pytest tests/test_buckets.py tests/test_info.py tests/test_metadata_util.py passes (33
tests), and ruff check / ruff format --check are clean on both files. I have not run
tests/test_upstream_hub.py, which needs the CI Hub token.

Open question

I went with silently writing the new features, since that matches what the shards actually
contain. If you'd rather the user were told that the declared schema changed under an
existing config, a warning at that point would be easy to add - happy to adjust.

_get_updated_dataset_card updated the splits and sizes of the existing
dataset_info but never the features, so re-pushing a dataset's only split
with different columns left the card declaring a schema the new shards
don't have, and the dataset failed to cast on load.

Reaching the assignment with other splits on the repo means the features
already matched, so this only takes effect when the push replaces them all.
@belambert belambert changed the title Write the pushed features to the card when re-pushing a split Write pushed features to card when re-pushing a split Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

push_to_hub() keeps stale dataset_info.features when re-pushing a single split with different columns

1 participant