feat: add Array.with_shape and Array.with_chunk_grid - #116
Open
kylebarron wants to merge 9 commits into
Open
Conversation
Wraps zarrs Array::set_shape. Returns a new array rather than mutating in place, so each handle stays internally consistent for its lifetime. Performs no I/O; store_metadata() persists the new shape. Refs #111 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shared pymethods macro already emits with_shape onto AsyncArray; this adds the stub and an async round-trip test. The method is sync on AsyncArray because it performs no I/O. Refs #111 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kylebarron
commented
Jul 30, 2026
Comment on lines
+168
to
+172
| /// Return a new array with `shape`, leaving this one unchanged. | ||
| fn with_shape( | ||
| &self, | ||
| shape: $crate::array::PyArrayShape, | ||
| ) -> $crate::error::ZarristaResult<Self> { |
Member
Author
There was a problem hiding this comment.
with_shape is immutable and creates a new array reference. The old array reference has the old shape
kylebarron
commented
Jul 30, 2026
Member
Author
There was a problem hiding this comment.
@d-v-b you can ignore reading the specs; they're verbose history only for future agent context
kylebarron
commented
Jul 30, 2026
Comment on lines
+232
to
+235
| Growing an array leaves the new region reading as the fill value. Shrinking | ||
| leaves any chunks outside the new bounds in the store, where they are no | ||
| longer addressable through this array; reclaiming that space will be a | ||
| separate call. |
Member
Author
There was a problem hiding this comment.
Perhaps we should have a vacuum call that removes any now-unaccessible chunks?
Wraps the unsafe zarrs Array::set_shape_and_chunk_grid. Takes a single ChunkGrid rather than the upstream (shape, grid_metadata) pair, since zarrista's ChunkGrid already carries the array shape; passing both would let them contradict each other. Existing chunks are not migrated. The stub documents the hazard, following the convention store_encoded_chunk already uses for a caller-beware API. Refs #111 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shared pymethods macro already emits with_chunk_grid onto AsyncArray; this adds the stub and an async round-trip test. The method is sync on AsyncArray because it performs no I/O. Refs #111 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies review feedback on #116: an Array names stored data rather than holding it, so with_shape/with_chunk_grid return a new reference. Wording applied to both methods on both classes, and to the Rust /// summaries so help() matches the stubs. Refs #111 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kylebarron
commented
Jul 30, 2026
| `erase_metadata`, ...) raises at runtime. | ||
| """ | ||
| def with_chunk_grid(self, chunk_grid: ChunkGrid) -> Array: | ||
| """Return a new array reference with `chunk_grid`, leaving this one unchanged. |
Member
Author
There was a problem hiding this comment.
Suggested change
| """Return a new array reference with `chunk_grid`, leaving this one unchanged. | |
| """Return a new array reference with `chunk_grid`. | |
| This does not mutate the existing `array`. |
kylebarron
commented
Jul 30, 2026
Member
Author
There was a problem hiding this comment.
Ok this PR LGTM now.
I'll wait till tomorrow to merge so @d-v-b can take a look if he wants
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two metadata-only reshaping methods to
ArrayandAsyncArray, wrappingzarrs'
Array::set_shapeandArray::set_shape_and_chunk_grid.Closes #111.
API
Both return a new array reference and leave the receiver untouched. Neither
performs any I/O;
store_metadata()persists. OnAsyncArrayboth are syncmethods, since there is nothing to await.
Why a new reference rather than in-place mutation
zarr-python's
Array.resize()mutates in place, so this diverges deliberately.An
Arrayis a cached snapshot ofzarr.jsonplus the objects derived from it,and rebinding (
array = array.with_shape(...)) leaves no stale referencebehind. Mutating in place would instead require interior mutability
(
RwLock<Arc<Array<…>>>) across all 57self.innersites, forceshape(),chunk_grid_shape(), anddimension_names()to return owned values rather thanborrowed, and open a window where a Python-level sequence such as
(array.shape, array.chunk_grid_shape)could straddle a mutation and observe amix of before and after. An immutable reference is internally consistent for its
whole lifetime.
It also matches
read_only(), which already derives a newArrayfrom anexisting one.
with_chunk_gridtakes one argument, not twozarrs' signature is
set_shape_and_chunk_grid(array_shape, chunk_grid_metadata),because its chunk-grid metadata is a shape-free chunking scheme. zarrista's
ChunkGridis not: every constructor takesarray_shapefirst(
ChunkGrid.regular(array_shape, chunk_shape)). A literal port would state theshape twice and let the two contradict each other:
Taking the
ChunkGridalone and reading the shape off it makes thatunrepresentable — and removes an error class, since the grid and shape now agree
by construction.
Safety
set_shape_and_chunk_gridisunsafeupstream: it does not check that chunksalready in the store match the new grid. Changing the chunk shape leaves existing
chunks at keys that no longer describe the same region, so later reads may fail
to decode or return wrong data.
Python has no
unsafe, so this is documented on the stub rather than encoded inthe name — following the convention
store_encoded_chunkalready uses for acaller-beware API. The docs point at
with_shapewhen only the shape ischanging, since that preserves the chunking and keeps existing chunks valid.
Implementation
Both methods perform no I/O and are byte-identical across the sync and async
classes, so they live in the
shared_array_methods!macro rather than insync.rs/async.rs. zarrs'Arrayis notClone, sowith_storage(self.inner.storage())supplies the copy while keeping the storagetype parameter unchanged.
Shape and grid validation is left entirely to zarrs; because
?runs beforeSelf::new, a rejected input never yields an object.Not included
Shrinking, or regridding, leaves chunks outside the new bounds in the store.
zarr-python folds that cleanup into
resize(); here it cannot live in thesemethods, which do no I/O at all, so it will be a separate vacuum call built
on zarrs'
erase_chunks.test_shrink_leaves_out_of_bounds_chunksandtest_existing_chunks_are_not_migratedpin the current behavior so that changehas something to flip.
Two further follow-ups came out of the design discussion, each worth its own
issue: a pull sync point (
reopen()) for when the store has moved ahead of alocal reference, and optimistic concurrency for
store_metadatavia conditionalPUT.
Testing
15 new tests across
tests/array/test_with_shape.py(8) andtests/array/test_with_chunk_grid.py(7): the new-reference/original split,that nothing is written before
store_metadata, grow/shrink/regrid round-tripsthrough reopen, data surviving a grow, wrong dimensionality raising, rectilinear
grids, orphaned chunks after a shrink, chunks left unmigrated after a regrid, and
async round-trips for both methods.
Full suite: 125 passed.
cargo fmt,cargo clippy --all-targets --all-features -D warnings, andruff check/format --checkovertests/andpython/allclean.
🤖 Generated with Claude Code