Skip to content

model_specs: declare every file the safetensors sources require - #550

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
christopherthompson81:specs/safetensors-files-lists
Sep 15, 2026
Merged

0xShug0 merged 1 commit into
0xShug0:mainfrom
christopherthompson81:specs/safetensors-files-lists

Conversation

@christopherthompson81

@christopherthompson81 christopherthompson81 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Six safetensors packages declare fewer files than their own source requires. Each installs cleanly and then fails at load:

missing model package file 'bigvgan_44k_config': <models>/SeedVC-MLX/bigvgan/v2_44khz_128band_512x/config.json

files is an explicit list rather than a filter — the package manager iterates package.files for snapshot kinds — so a downloader has no way to discover the rest. CONTRIBUTING's "a package should not advertise a model that the CLI/server cannot actually load" is the rule being broken.

package source requires package declared
seed_vc_mlx_safetensors 28 3
index_tts2_safetensors 20 3
supertonic_3_safetensors 13 3
chatterbox_safetensors 9 4
qwen3_tts_1_7b_base_safetensors 8 6
omnivoice_safetensors 7 3

No new hosting

Every file added here is already published, at exactly the path the spec declares, in the repository the package already downloads from. I checked all 37 missing paths against the HuggingFace API; 36 were present, and the one that was not is why voxcpm2 is left out (below).

So this is a declaration fix. Existing entries keep their order and the new ones follow in the order the source lists them, so the diff is additions only.

What this costs to download

These packages get materially bigger, because the engine requires the files and they were simply never fetched. Sizes from the HuggingFace blob API:

package before after
seed_vc_mlx_safetensors 712 MB 7.0 GB
index_tts2_safetensors 3.5 GB 8.1 GB
chatterbox_safetensors 3.2 GB 7.5 GB
omnivoice_safetensors 2.5 GB 3.3 GB
qwen3_tts_1_7b_base_safetensors 4.54 GB 4.54 GB
supertonic_3_safetensors 397 MB 400 MB

The comparison is a little unfair to the "before" column — those packages do not load at all today, so the smaller number buys nothing. But seed_vc going 712 MB → 7.0 GB is a real change in what a user gets when they click install, and worth a deliberate decision rather than a side effect of this PR.

Which raises a question I cannot answer from outside. seed_vc's safetensors source lists the v1 stack (v1/svc, v1/whisper_bigvgan, v1/xlsr_hift) and the v2 stack, plus whisper-small, hubert-large-ll60k and wav2vec2-xls-r-300m — all in required files/tensors, none in optional_files/optional_tensors. If a given conversion route only ever touches one of those paths, the better fix is to move the unused ones to the optional maps rather than to declare them all here, and this PR would shrink to the genuinely-required set.

I have declared exactly what the source currently marks as required, because that is what the loader enforces and what makes the packages load today. If some of those should be optional instead, say which and I will redo it that way — that is a statement about the model, not something the spec makes visible.

Two judgement calls, flagged rather than made quietly

voxcpm2_safetensors is deliberately untouched. It has the same shape — 6 required, 4 declared — but one of the two missing files, audiovae.safetensors, is not published by OpenBMB/VoxCPM2 at all. Adding only special_tokens_map.json would leave the package still failing, one clue poorer. That one looks like a hosting gap rather than a declaration gap, and I did not want to paper over it.

index_tts2 goes 3 → 21, not 3 → 20. It already declared bpe.model, which the safetensors source names in neither files nor tensors. I kept it: it ships today and is published in the repo, and removing a file a package already fetches is a different decision from adding ones it needs.

The check that found it

For each package, take the source whose format matches, collect every model:-rooted path in that source's files and tensors, and assert each appears in the package's files (allowing for strip_prefix). Worth having at build time — it is the same shape as the consistency check that would have caught this class originally.

Two things it has to get right, both found by getting them wrong first:

  • Read only files and tensors. optional_files and optional_tensors are separate keys loaded via add_optional_resource_map (src/framework/model_spec/package.cpp:378,390); only the required maps throw, at package.cpp:283. Including the optional ones gives false positives.
  • Be format-aware. A GGUF source names the same config files, but the GGUF embeds them, so the package correctly ships one .gguf. Run the check format-blind and it reports 173 packages — three quarters of the catalogue — every one of them false.

With both accounted for it reports exactly the seven packages above and nothing else across all 237.

What is not verified

The evidence here is static: the spec says the file is required, and the repo publishes it at that path. I have not installed and loaded each patched package end to end. Happy to do that for any of them if you would rather see it before merging — supertonic is the cheapest at ~414 MB.

Six safetensors packages list fewer files than their own source needs, so they
install cleanly and then fail at load with

    missing model package file '<id>': <models>/...

`files` is an explicit list rather than a filter -- the package manager iterates
it for snapshot downloads -- so a downloader has no way to discover the rest.
`optional_files` and `optional_tensors` exist for genuinely optional resources
and load through add_optional_resource_map; everything in `files` and `tensors`
is required and throws at package.cpp:283 when absent.

    chatterbox_safetensors             requires  9, declared 4
    index_tts2_safetensors             requires 20, declared 3
    omnivoice_safetensors              requires  7, declared 3
    qwen3_tts_1_7b_base_safetensors    requires  8, declared 6
    seed_vc_mlx_safetensors            requires 28, declared 3
    supertonic_3_safetensors           requires 13, declared 3

Every file added here is already published, at exactly the path the spec
declares, in the repository the package already downloads from -- so this is a
declaration fix and needs no new hosting. Existing entries keep their order and
the new ones follow in the order the source lists them.

voxcpm2_safetensors has the same shape but is deliberately left alone: it also
requires `audiovae.safetensors`, which OpenBMB/VoxCPM2 does not publish, so
completing its `files` list would not make it loadable and the gap there is
hosting rather than declaration.

Found by checking, for every package, that each `model:`-rooted path in its
format's source appears in the package's `files`. Worth having as a build-time
check -- with the caveat that it has to be format-aware, since a GGUF source
names the same config files and the GGUF embeds them, so a format-blind version
reports three quarters of the catalogue.
@0xShug0
0xShug0 merged commit 5db449e into 0xShug0:main Sep 15, 2026
6 checks passed
@0xShug0

0xShug0 commented Sep 15, 2026

Copy link
Copy Markdown
Owner

@christopherthompson81 PR merged and thanks for the consistency pass! The inconsistency is mainly because we decided to switch to GGUF as the default before implementing spec v1. Safetensors sources are no longer required and many models don't handle safetensor weights, so many of them may just be placeholders in the specs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants