From 6251965983e14519ed8d2a6b39d55c469bd82cd9 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 29 Jul 2026 21:56:28 +0200 Subject: [PATCH] chore(zarr-metadata): build 0.4.0 changelog Consume the pending news fragments (the #4119 model layer's feature and removal notes, plus a new doc fragment for the standalone documentation site and justfile from #4208/#4210) into CHANGELOG.md via towncrier for the zarr_metadata-v0.4.0 release. Assisted-by: ClaudeCode:claude-fable-5 --- packages/zarr-metadata/CHANGELOG.md | 153 ++++++++++++++++++ .../zarr-metadata/changes/4119.feature.md | 92 ----------- .../zarr-metadata/changes/4119.removal.md | 41 ----- 3 files changed, 153 insertions(+), 133 deletions(-) delete mode 100644 packages/zarr-metadata/changes/4119.feature.md delete mode 100644 packages/zarr-metadata/changes/4119.removal.md diff --git a/packages/zarr-metadata/CHANGELOG.md b/packages/zarr-metadata/CHANGELOG.md index a3ff1177a0..981c7706bf 100644 --- a/packages/zarr-metadata/CHANGELOG.md +++ b/packages/zarr-metadata/CHANGELOG.md @@ -2,6 +2,159 @@ +## 0.4.0 (2026-07-29) + +### Features + +- Added `zarr_metadata.model`: frozen-dataclass models (`ZarrV2ArrayMetadata`, + `ZarrV3ArrayMetadata`, `ZarrV2GroupMetadata`, `ZarrV3GroupMetadata`, + `ZarrV2ConsolidatedMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV3NamedConfig`) + that are canonical, semantically lossless representations of Zarr metadata + documents, plus structural validators (`validate_*` / `is_*` / `parse_*`). + Every v3 extension point (data type, chunk grid, chunk key encoding, codecs, + storage transformers) is held as `ZarrV3NamedConfig`: a name, configuration, + and `must_understand` obligation; nothing is interpreted. On the wire, an + empty configuration with the default obligation uses the spec's plain-string + shorthand. Model fields are annotated with the role alias + `ZarrV3MetadataField` (today exactly `ZarrV3NamedConfig`), so annotations + convey the logical meaning and stay put if the spec adds another field form. + + Validation is strict about what the types declare: v2 `dtype` / `order` / + `compressor` / `filters` / `dimension_separator` shapes and the fixed + `zarr_format` / `node_type` literals are all enforced. Every + `ValidationProblem` carries a machine-readable `kind` + (`missing_key` / `invalid_type` / `invalid_value` / `invalid_json`) so + consumers can dispatch on the failure mode without matching message strings, + and every ingestion failure — including missing store keys and undecodable + bytes in `from_key_value` — surfaces as `MetadataValidationError`. An + adversarial review added further structural checks: JSON booleans are not + accepted as dimension lengths, dimensions are non-negative, + `dimension_names` must have one entry per dimension of `shape`, `attributes` + and `configuration` values are JSON-checked recursively (like `fill_value`), + non-finite floats and non-standard JSON constants are rejected, abstract + mappings and sequences normalize to encoder-safe canonical containers, + v2 `shape` and `chunks` must have the same rank, non-null v2 filter pipelines + contain at least one filter, document `TypeIs` guards only narrow values that + already use the declared canonical containers, + and the inline consolidated-metadata envelope and entries are deep-validated + so the group validator's verdict always agrees with the model constructor. + + The v3 models expose `must_understand_fields`: the subset of `extra_fields` + not explicitly waived with `must_understand: false` (fields are implicitly + must-understand per the spec). Readers discharge the spec's fail-to-open + duty by subtracting the extension names they recognize; the model only + partitions by obligation, since recognition is reader-specific. + + Optional pydantic integration ships as `zarr_metadata.pydantic` (importing it + requires pydantic 2.13 or newer; the core package does not depend on it): one + `Annotated` + field type per model, validating raw documents through `from_json`, passing + core-model instances through unchanged, serializing via `to_json`, and + publishing JSON Schemas derived from private constrained document types that + mirror the independently expressible runtime rules. Cross-field cardinality + relations still require runtime validation. The instances are the core model + classes, so values interoperate freely with non-pydantic code. + + `create_default` keeps its output self-consistent: overriding `shape` without + a chunk grid derives one regular chunk covering the array (v3 + `chunk_shape == shape`; v2 `chunks == shape`) instead of silently keeping the + scalar default's 0-d grid. + + A v2 `.zarray` that omits `dimension_separator` is interpreted with the v2 + convention's default `"."` (the model previously normalized absence to `"/"`, + which would misaddress the chunks of real-world default-separator arrays). + The value is never null: absent, `"."`, or `"/"` are the only spellings. + + Optional document keys use `UNSET` — a PEP 661 sentinel + (`typing_extensions.Sentinel`), usable directly in type expressions — never + `None`: in a model, `None` always corresponds to a JSON `null` in the + document (a v2 `compressor`, an unnamed dimension inside `dimension_names`), + and `UNSET` always means the key is absent. Checker note: ty types the + sentinel exactly; pyright needs `<= 1.1.404` until microsoft/pyright#11115 + is fixed (this package's CI pins it); mypy users need a `cast` or + `type: ignore` at narrowing sites until python/mypy#21647 merges. This keeps semantically distinct spellings + distinct — an absent `dimension_names` ("there are no dimension names") and + an explicit `[null, null]` ("every dimension has a name, which is null") are + different documents and round-trip as such. The `consolidated_metadata: null` + written by a historical zarr-python bug is the one deliberate exception to + faithful round-tripping: those stores remain readable, but the bug spelling + is repaired to absence on read and never written back. + + The v2 models treat the `.zattrs` file's presence as part of the store: + `attributes` is `UNSET` when no `.zattrs` file exists (and `to_key_value` + emits none), while an explicit empty `.zattrs` is `{}` and round-trips as a + file. Previously `to_key_value` always emitted `.zattrs`, silently adding a + file to stores that never had one. + + The store-key `Literal` aliases (`ZarrV2ArrayMetadataStoreKey`, + `ZarrV2AttributesStoreKey`, ...) are exported from `zarr_metadata.model` + alongside their constants, and each `to_key_value` return type is keyed by + them, so the set of store keys a model can emit is visible in its signature. + `from_key_value` deliberately keeps `Mapping[str, bytes]` input: it accepts + any string-keyed store mapping and ignores unrelated keys. + + `to_json` returns a document that shares no mutable state with the model: + every value that can hold a mutable container (attributes, configurations, + extra fields, v2 codec configurations, fill values, consolidated entries) is + deep-copied on the way out, so editing a serialized document can never + silently mutate the frozen model that produced it. ([#4119](https://github.com/zarr-developers/zarr-python/issues/4119)) + +### Improved Documentation + +- `zarr-metadata` now has a standalone documentation site at + , with a comprehensive API reference + covering every public module, versioned by this package's release tags. The + package also gained a `justfile` collecting its development commands + (`test`, `lint`, `typecheck`, `docs-check`, `docs-serve`, `changelog-draft`), + which the package CI workflow now delegates to. ([#4208](https://github.com/zarr-developers/zarr-python/issues/4208)) + +### Deprecations and Removals + +- The document (TypedDict) types are renamed to put the format version at the + front of the name and to mark the JSON-document form with a `JSON` suffix, + so a format version can never be misread as a class revision and the bare + entity names are reserved for the `zarr_metadata.model` dataclasses: + + - `ArrayMetadataV2` → `ZarrV2ArrayMetadataJSON` (and `...Partial` accordingly) + - `ArrayMetadataV3` → `ZarrV3ArrayMetadataJSON` (and `...Partial` accordingly) + - `GroupMetadataV2` → `ZarrV2GroupMetadataJSON` (and `...Partial` accordingly) + - `GroupMetadataV3` → `ZarrV3GroupMetadataJSON` (and `...Partial` accordingly) + - `ConsolidatedMetadataV2` → `ZarrV2ConsolidatedMetadataJSON` + - `ConsolidatedMetadataV3` → `ZarrV3ConsolidatedMetadataJSON` + - `NamedConfigV3` → `ZarrV3NamedConfigJSON` + - `MetadataV3` → `ZarrV3MetadataFieldJSON` (the union of the bare-name and + named-configuration spellings of one metadata field) + - `ExtensionFieldV3` → `ZarrV3ExtensionField` + - `CodecMetadataV2` → `ZarrV2CodecMetadata` + - `DataTypeMetadataV2` → `ZarrV2DataTypeMetadata` + - `ArrayOrderV2` → `ZarrV2ArrayOrder` + - `ArrayDimensionSeparatorV2` → `ZarrV2ArrayDimensionSeparator` + - `ZArrayMetadata` → `ZarrV2ZArrayJSON` (the strict on-disk `.zarray` document) + - `ZGroupMetadata` → `ZarrV2ZGroupJSON` (the strict on-disk `.zgroup` document) + - `ZAttrsMetadata` → `ZarrV2ZAttrsJSON` (the `.zattrs` document) + + The old names are removed, not aliased. The `zarr_metadata.pydantic` field + types take the bare entity names (`ZarrV3ArrayMetadata`, ...), matching the + model classes they validate into. + + The conventions, stated once for future additions: CamelCase type names put + the format version first (`ZarrV2ArrayMetadataJSON`, + `ZarrV3ArrayMetadataStoreKey`), while SCREAMING_SNAKE constants and + snake_case functions put it last (`ARRAY_METADATA_STORE_KEY_V2`, + `validate_array_metadata_v3`). The `JSON` suffix marks a raw-document type + whose bare name is taken by (or reserved for) a `zarr_metadata.model` + dataclass; raw field-level types the models hold verbatim + (`ZarrV2CodecMetadata`, `ZarrV3ExtensionField`) keep their bare names. + Extension-entity types put the registered entity name first and end in + exactly one role suffix (`BloscCodecMetadata`, `Uint8DataTypeName`) — the + `V2` in `V2ChunkKeyEncodingMetadata` is that encoding's entity name, not a + format version, which is always spelled `ZarrV2`/`ZarrV3`. Every public + type name is checked against this grammar by + `tests/test_public_api.py::test_public_type_names_comply_with_naming_grammar`. + + ([#4119](https://github.com/zarr-developers/zarr-python/issues/4119)) + + ## 0.3.0 (2026-06-19) ### Deprecations and Removals diff --git a/packages/zarr-metadata/changes/4119.feature.md b/packages/zarr-metadata/changes/4119.feature.md deleted file mode 100644 index b9d0bb508c..0000000000 --- a/packages/zarr-metadata/changes/4119.feature.md +++ /dev/null @@ -1,92 +0,0 @@ -Added `zarr_metadata.model`: frozen-dataclass models (`ZarrV2ArrayMetadata`, -`ZarrV3ArrayMetadata`, `ZarrV2GroupMetadata`, `ZarrV3GroupMetadata`, -`ZarrV2ConsolidatedMetadata`, `ZarrV3ConsolidatedMetadata`, `ZarrV3NamedConfig`) -that are canonical, semantically lossless representations of Zarr metadata -documents, plus structural validators (`validate_*` / `is_*` / `parse_*`). -Every v3 extension point (data type, chunk grid, chunk key encoding, codecs, -storage transformers) is held as `ZarrV3NamedConfig`: a name, configuration, -and `must_understand` obligation; nothing is interpreted. On the wire, an -empty configuration with the default obligation uses the spec's plain-string -shorthand. Model fields are annotated with the role alias -`ZarrV3MetadataField` (today exactly `ZarrV3NamedConfig`), so annotations -convey the logical meaning and stay put if the spec adds another field form. - -Validation is strict about what the types declare: v2 `dtype` / `order` / -`compressor` / `filters` / `dimension_separator` shapes and the fixed -`zarr_format` / `node_type` literals are all enforced. Every -`ValidationProblem` carries a machine-readable `kind` -(`missing_key` / `invalid_type` / `invalid_value` / `invalid_json`) so -consumers can dispatch on the failure mode without matching message strings, -and every ingestion failure — including missing store keys and undecodable -bytes in `from_key_value` — surfaces as `MetadataValidationError`. An -adversarial review added further structural checks: JSON booleans are not -accepted as dimension lengths, dimensions are non-negative, -`dimension_names` must have one entry per dimension of `shape`, `attributes` -and `configuration` values are JSON-checked recursively (like `fill_value`), -non-finite floats and non-standard JSON constants are rejected, abstract -mappings and sequences normalize to encoder-safe canonical containers, -v2 `shape` and `chunks` must have the same rank, non-null v2 filter pipelines -contain at least one filter, document `TypeIs` guards only narrow values that -already use the declared canonical containers, -and the inline consolidated-metadata envelope and entries are deep-validated -so the group validator's verdict always agrees with the model constructor. - -The v3 models expose `must_understand_fields`: the subset of `extra_fields` -not explicitly waived with `must_understand: false` (fields are implicitly -must-understand per the spec). Readers discharge the spec's fail-to-open -duty by subtracting the extension names they recognize; the model only -partitions by obligation, since recognition is reader-specific. - -Optional pydantic integration ships as `zarr_metadata.pydantic` (importing it -requires pydantic 2.13 or newer; the core package does not depend on it): one -`Annotated` -field type per model, validating raw documents through `from_json`, passing -core-model instances through unchanged, serializing via `to_json`, and -publishing JSON Schemas derived from private constrained document types that -mirror the independently expressible runtime rules. Cross-field cardinality -relations still require runtime validation. The instances are the core model -classes, so values interoperate freely with non-pydantic code. - -`create_default` keeps its output self-consistent: overriding `shape` without -a chunk grid derives one regular chunk covering the array (v3 -`chunk_shape == shape`; v2 `chunks == shape`) instead of silently keeping the -scalar default's 0-d grid. - -A v2 `.zarray` that omits `dimension_separator` is interpreted with the v2 -convention's default `"."` (the model previously normalized absence to `"/"`, -which would misaddress the chunks of real-world default-separator arrays). -The value is never null: absent, `"."`, or `"/"` are the only spellings. - -Optional document keys use `UNSET` — a PEP 661 sentinel -(`typing_extensions.Sentinel`), usable directly in type expressions — never -`None`: in a model, `None` always corresponds to a JSON `null` in the -document (a v2 `compressor`, an unnamed dimension inside `dimension_names`), -and `UNSET` always means the key is absent. Checker note: ty types the -sentinel exactly; pyright needs `<= 1.1.404` until microsoft/pyright#11115 -is fixed (this package's CI pins it); mypy users need a `cast` or -`type: ignore` at narrowing sites until python/mypy#21647 merges. This keeps semantically distinct spellings -distinct — an absent `dimension_names` ("there are no dimension names") and -an explicit `[null, null]` ("every dimension has a name, which is null") are -different documents and round-trip as such. The `consolidated_metadata: null` -written by a historical zarr-python bug is the one deliberate exception to -faithful round-tripping: those stores remain readable, but the bug spelling -is repaired to absence on read and never written back. - -The v2 models treat the `.zattrs` file's presence as part of the store: -`attributes` is `UNSET` when no `.zattrs` file exists (and `to_key_value` -emits none), while an explicit empty `.zattrs` is `{}` and round-trips as a -file. Previously `to_key_value` always emitted `.zattrs`, silently adding a -file to stores that never had one. - -The store-key `Literal` aliases (`ZarrV2ArrayMetadataStoreKey`, -`ZarrV2AttributesStoreKey`, ...) are exported from `zarr_metadata.model` -alongside their constants, and each `to_key_value` return type is keyed by -them, so the set of store keys a model can emit is visible in its signature. -`from_key_value` deliberately keeps `Mapping[str, bytes]` input: it accepts -any string-keyed store mapping and ignores unrelated keys. - -`to_json` returns a document that shares no mutable state with the model: -every value that can hold a mutable container (attributes, configurations, -extra fields, v2 codec configurations, fill values, consolidated entries) is -deep-copied on the way out, so editing a serialized document can never -silently mutate the frozen model that produced it. diff --git a/packages/zarr-metadata/changes/4119.removal.md b/packages/zarr-metadata/changes/4119.removal.md deleted file mode 100644 index 2a9a6f84c4..0000000000 --- a/packages/zarr-metadata/changes/4119.removal.md +++ /dev/null @@ -1,41 +0,0 @@ -The document (TypedDict) types are renamed to put the format version at the -front of the name and to mark the JSON-document form with a `JSON` suffix, -so a format version can never be misread as a class revision and the bare -entity names are reserved for the `zarr_metadata.model` dataclasses: - -- `ArrayMetadataV2` → `ZarrV2ArrayMetadataJSON` (and `...Partial` accordingly) -- `ArrayMetadataV3` → `ZarrV3ArrayMetadataJSON` (and `...Partial` accordingly) -- `GroupMetadataV2` → `ZarrV2GroupMetadataJSON` (and `...Partial` accordingly) -- `GroupMetadataV3` → `ZarrV3GroupMetadataJSON` (and `...Partial` accordingly) -- `ConsolidatedMetadataV2` → `ZarrV2ConsolidatedMetadataJSON` -- `ConsolidatedMetadataV3` → `ZarrV3ConsolidatedMetadataJSON` -- `NamedConfigV3` → `ZarrV3NamedConfigJSON` -- `MetadataV3` → `ZarrV3MetadataFieldJSON` (the union of the bare-name and - named-configuration spellings of one metadata field) -- `ExtensionFieldV3` → `ZarrV3ExtensionField` -- `CodecMetadataV2` → `ZarrV2CodecMetadata` -- `DataTypeMetadataV2` → `ZarrV2DataTypeMetadata` -- `ArrayOrderV2` → `ZarrV2ArrayOrder` -- `ArrayDimensionSeparatorV2` → `ZarrV2ArrayDimensionSeparator` -- `ZArrayMetadata` → `ZarrV2ZArrayJSON` (the strict on-disk `.zarray` document) -- `ZGroupMetadata` → `ZarrV2ZGroupJSON` (the strict on-disk `.zgroup` document) -- `ZAttrsMetadata` → `ZarrV2ZAttrsJSON` (the `.zattrs` document) - -The old names are removed, not aliased. The `zarr_metadata.pydantic` field -types take the bare entity names (`ZarrV3ArrayMetadata`, ...), matching the -model classes they validate into. - -The conventions, stated once for future additions: CamelCase type names put -the format version first (`ZarrV2ArrayMetadataJSON`, -`ZarrV3ArrayMetadataStoreKey`), while SCREAMING_SNAKE constants and -snake_case functions put it last (`ARRAY_METADATA_STORE_KEY_V2`, -`validate_array_metadata_v3`). The `JSON` suffix marks a raw-document type -whose bare name is taken by (or reserved for) a `zarr_metadata.model` -dataclass; raw field-level types the models hold verbatim -(`ZarrV2CodecMetadata`, `ZarrV3ExtensionField`) keep their bare names. -Extension-entity types put the registered entity name first and end in -exactly one role suffix (`BloscCodecMetadata`, `Uint8DataTypeName`) — the -`V2` in `V2ChunkKeyEncodingMetadata` is that encoding's entity name, not a -format version, which is always spelled `ZarrV2`/`ZarrV3`. Every public -type name is checked against this grammar by -`tests/test_public_api.py::test_public_type_names_comply_with_naming_grammar`.