Skip to content

feat: add Array.with_shape and Array.with_chunk_grid - #116

Open
kylebarron wants to merge 9 commits into
mainfrom
feat/array-with-shape
Open

feat: add Array.with_shape and Array.with_chunk_grid#116
kylebarron wants to merge 9 commits into
mainfrom
feat/array-with-shape

Conversation

@kylebarron

@kylebarron kylebarron commented Jul 30, 2026

Copy link
Copy Markdown
Member

Adds two metadata-only reshaping methods to Array and AsyncArray, wrapping
zarrs' Array::set_shape and Array::set_shape_and_chunk_grid.

Closes #111.

API

# change the shape, keep the chunking
array = array.with_shape([8, 8])
array.store_metadata()

# change the chunking (and, optionally, the shape with it)
array = array.with_chunk_grid(ChunkGrid.regular([8, 8], [4, 4]))
array.store_metadata()

Both return a new array reference and leave the receiver untouched. Neither
performs any I/O; store_metadata() persists. On AsyncArray both are sync
methods, since there is nothing to await.

Why a new reference rather than in-place mutation

zarr-python's Array.resize() mutates in place, so this diverges deliberately.

An Array is a cached snapshot of zarr.json plus the objects derived from it,
and rebinding (array = array.with_shape(...)) leaves no stale reference
behind. Mutating in place would instead require interior mutability
(RwLock<Arc<Array<…>>>) across all 57 self.inner sites, force shape(),
chunk_grid_shape(), and dimension_names() to return owned values rather than
borrowed, and open a window where a Python-level sequence such as
(array.shape, array.chunk_grid_shape) could straddle a mutation and observe a
mix of before and after. An immutable reference is internally consistent for its
whole lifetime.

It also matches read_only(), which already derives a new Array from an
existing one.

with_chunk_grid takes one argument, not two

zarrs' signature is set_shape_and_chunk_grid(array_shape, chunk_grid_metadata),
because its chunk-grid metadata is a shape-free chunking scheme. zarrista's
ChunkGrid is not: every constructor takes array_shape first
(ChunkGrid.regular(array_shape, chunk_shape)). A literal port would state the
shape twice and let the two contradict each other:

array.with_shape_and_chunk_grid([8, 8], ChunkGrid.regular([4, 4], [2, 2]))
#                                ^^^^^^                   ^^^^^^

Taking the ChunkGrid alone and reading the shape off it makes that
unrepresentable — and removes an error class, since the grid and shape now agree
by construction.

Safety

set_shape_and_chunk_grid is unsafe upstream: it does not check that chunks
already in the store match the new grid. Changing the chunk shape leaves existing
chunks at keys that no longer describe the same region, so later reads may fail
to decode or return wrong data.

Python has no unsafe, so this is documented on the stub rather than encoded in
the name — following the convention store_encoded_chunk already uses for a
caller-beware API. The docs point at with_shape when only the shape is
changing, since that preserves the chunking and keeps existing chunks valid.

Implementation

Both methods perform no I/O and are byte-identical across the sync and async
classes, so they live in the shared_array_methods! macro rather than in
sync.rs / async.rs. zarrs' Array is not Clone, so
with_storage(self.inner.storage()) supplies the copy while keeping the storage
type parameter unchanged.

Shape and grid validation is left entirely to zarrs; because ? runs before
Self::new, a rejected input never yields an object.

Not included

Shrinking, or regridding, leaves chunks outside the new bounds in the store.
zarr-python folds that cleanup into resize(); here it cannot live in these
methods, which do no I/O at all, so it will be a separate vacuum call built
on zarrs' erase_chunks. test_shrink_leaves_out_of_bounds_chunks and
test_existing_chunks_are_not_migrated pin the current behavior so that change
has something to flip.

Two further follow-ups came out of the design discussion, each worth its own
issue: a pull sync point (reopen()) for when the store has moved ahead of a
local reference, and optimistic concurrency for store_metadata via conditional
PUT.

Testing

15 new tests across tests/array/test_with_shape.py (8) and
tests/array/test_with_chunk_grid.py (7): the new-reference/original split,
that nothing is written before store_metadata, grow/shrink/regrid round-trips
through reopen, data surviving a grow, wrong dimensionality raising, rectilinear
grids, orphaned chunks after a shrink, chunks left unmigrated after a regrid, and
async round-trips for both methods.

Full suite: 125 passed. cargo fmt, cargo clippy --all-targets --all-features -D warnings, and ruff check / format --check over tests/ and python/ all
clean.

🤖 Generated with Claude Code

kylebarron and others added 3 commits July 30, 2026 12:05
Wraps zarrs Array::set_shape. Returns a new array rather than mutating
in place, so each handle stays internally consistent for its lifetime.
Performs no I/O; store_metadata() persists the new shape.

Refs #111

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shared pymethods macro already emits with_shape onto AsyncArray;
this adds the stub and an async round-trip test. The method is sync
on AsyncArray because it performs no I/O.

Refs #111

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kylebarron
kylebarron requested a review from d-v-b July 30, 2026 16:31
Comment thread src/array/shared.rs Outdated
Comment on lines +168 to +172
/// Return a new array with `shape`, leaving this one unchanged.
fn with_shape(
&self,
shape: $crate::array::PyArrayShape,
) -> $crate::error::ZarristaResult<Self> {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with_shape is immutable and creates a new array reference. The old array reference has the old shape

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@d-v-b you can ignore reading the specs; they're verbose history only for future agent context

Comment thread python/zarrista/_array.pyi Outdated
Comment thread python/zarrista/_array.pyi Outdated
Comment thread python/zarrista/_array.pyi Outdated
Comment on lines +232 to +235
Growing an array leaves the new region reading as the fill value. Shrinking
leaves any chunks outside the new bounds in the store, where they are no
longer addressable through this array; reclaiming that space will be a
separate call.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should have a vacuum call that removes any now-unaccessible chunks?

kylebarron and others added 3 commits July 30, 2026 12:55
Wraps the unsafe zarrs Array::set_shape_and_chunk_grid. Takes a single
ChunkGrid rather than the upstream (shape, grid_metadata) pair, since
zarrista's ChunkGrid already carries the array shape; passing both would
let them contradict each other.

Existing chunks are not migrated. The stub documents the hazard, following
the convention store_encoded_chunk already uses for a caller-beware API.

Refs #111

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The shared pymethods macro already emits with_chunk_grid onto AsyncArray;
this adds the stub and an async round-trip test. The method is sync on
AsyncArray because it performs no I/O.

Refs #111

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies review feedback on #116: an Array names stored data rather than
holding it, so with_shape/with_chunk_grid return a new reference. Wording
applied to both methods on both classes, and to the Rust /// summaries so
help() matches the stubs.

Refs #111

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kylebarron kylebarron changed the title feat: add Array.with_shape for resizing arrays feat: add Array.with_shape and Array.with_chunk_grid Jul 30, 2026
`erase_metadata`, ...) raises at runtime.
"""
def with_chunk_grid(self, chunk_grid: ChunkGrid) -> Array:
"""Return a new array reference with `chunk_grid`, leaving this one unchanged.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Return a new array reference with `chunk_grid`, leaving this one unchanged.
"""Return a new array reference with `chunk_grid`.
This does not mutate the existing `array`.

@kylebarron kylebarron left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this PR LGTM now.

I'll wait till tomorrow to merge so @d-v-b can take a look if he wants

@kylebarron kylebarron added this to the 0.1 milestone Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support resizing existing arrays

1 participant