Zarr-Python: roadmap after 3.1

[jhamman]

I gave this presentation at SciPy last week on the progress Zarr-Python has made over the last year. tldr; we've come a long way!

I also shared a potential feature list that I think could form the beginning of the roadmap beyond version 3.1 (which landed today!).

Targets for Zarr 3.2 and beyond

• Performance tuning (sharding, codec-pipeline, async+multi-threading)
• Additional array-types (e.g. sparse) and data-types (e.g. ml-dtypes)
• New Extensions
  • Variable length chunk grids
  • Sparse arrays
• GPU support
  • On-GPU (de)compression
  • Other hardware (Apple, MLX)

Curious to get the input from others on what else we're looking to work on next.

A nice outcome from this discussion would be an update to the Zarr-Python Roadmap which is now nicely out of date: https://zarr.readthedocs.io/en/stable/developers/roadmap.html

cc @zarr-developers/python-core-devs, @zarr-developers/python-emeritus

[dstansby]

Adding read/write permissions to arrays & groups would be a good one to add, which I have as a work in progress.

[d-v-b]

• a lazy array indexing API.
  I actually think this will unlock some performance improvements in our array IO.
• remove StorePath, give all of its methods to the Store classes
• Do something other than return None for missing keys in the store APIs (e.g., use a Result type, assuming there are no performance penalties)
• Define an awaitable Future object, and have all of our user-facing async APIs return instances of this Future object that wrap the awaitable they currently return. Future objects would also have a sync or result method that just calls sync on the underlying awaitable

[normanrz]

• a lazy array indexing API.
  I actually think this will unlock some performance improvements in our array IO.

Is that the same as array "views"? I would be a fan of that

• remove StorePath, give all of its methods to the Store classes

What is the rationale for that? I think it is quite useful to have a pointer to a file (or even byte range) in a store.

[d-v-b]

Is that the same as array "views"? I would be a fan of that

Yeah, we could think of it like "views" but I think the more basic analogy is with the semantics of slicing generic collections. When you slice into a tuple, you get another tuple, not a numpy array. Zarr should follow the same principle. In concrete terms this would require modelling a zarr array as supported by a collection of(stored object, index) tuples. Indexing the zarr array would create a new array supported by a strict subset of the original collection of (stored object, index) tuples. This would also allow indexing to be reversible, i.e. we could concatenate zarr arrays, or build zarr arrays from stored objects on different storage backends.

What is the rationale for that? I think it is quite useful to have a pointer to a file (or even byte range) in a store.

If it's useful to have this, then it should be part of the Store API. StorePath is literally just a store, a string, and a set of convenience methods that use the store and the string. We can get the exact same functionality by putting all of this logic on the store classes themselves, with the benefit of removing a lot of unnecessary code.

[normanrz]

Is that the same as array "views"? I would be a fan of that

Yeah, we could think of it like "views" but I think the more basic analogy is with the semantics of slicing generic collections. When you slice into a tuple, you get another tuple, not a numpy array. Zarr should follow the same principle. In concrete terms this would require modelling a zarr array as supported by a collection of(stored object, index) tuples. Indexing the zarr array would create a new array supported by a strict subset of the original collection of (stored object, index) tuples. This would also allow indexing to be reversible, i.e. we could concatenate zarr arrays, or build zarr arrays from stored objects on different storage backends.

I thinkt that would be great. It would need careful API design to become usable and not too confusing. For my cases, an iterator that returns read/writable chunk or shard views would suffice.

[d-v-b]

yes my plan here is to start with low-level stuff like https://github.com/d-v-b/zarr-python/blob/0b9916443d555d9e762f5501314383dc828c26bf/src/zarr/core/array.py#L5253-L5286 and start working that into our indexing routines.

[martindurant]

Define an awaitable Future object

This has been done before in dask-distributed (inspired by concurrent.futures). fsspec decided not to follow the model, although it was discussed - of course we have sync() in common. I wonder if there is scope to come up with a spinoff project (e.g., https://docs.rs/futures/latest/futures/ !) for the general public good.

[martindurant]

remove StorePath, give all of its methods to the Store classes

This is not really user-facing, so whatever makes the most sense internally. If you can remove code and complexity, it's probably worth it.