diff --git a/changes/3352.bugfix.md b/changes/3352.bugfix.md deleted file mode 100644 index 7461486776..0000000000 --- a/changes/3352.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -Fix `zarr.api.asynchronous.open_like` so it can create a new array by default when the -target path does not already exist. It now defaults to `mode="a"`; when using a read-only -store to open an existing array, pass `mode="r"` explicitly. diff --git a/changes/4128.feature.md b/changes/4128.feature.md deleted file mode 100644 index c62a615ac2..0000000000 --- a/changes/4128.feature.md +++ /dev/null @@ -1 +0,0 @@ -Added `Group.get_array`, `Group.get_group`, `AsyncGroup.get_array`, and `AsyncGroup.get_group`: type-safe accessors that return the child array or group at a given path, raising `ArrayNotFoundError` / `GroupNotFoundError` if no node exists there, and `ContainsGroupError` / `ContainsArrayError` if the node is not of the requested kind. Unlike `Group.__getitem__`, which returns `Array | Group`, these methods have precise return types. Nested paths like `"subgroup/subarray"` are supported. diff --git a/changes/4157.bugfix.md b/changes/4157.bugfix.md deleted file mode 100644 index 6b0d0fcc67..0000000000 --- a/changes/4157.bugfix.md +++ /dev/null @@ -1,11 +0,0 @@ -`MemoryStore` now copies buffers as they are written, so it never retains the -caller's memory. Previously an uncompressed write handed the store a zero-copy -view of the user's array, and mutating that array afterwards would silently -rewrite chunks already committed to the store. - -Only `MemoryStore` is affected: stores that serialize on write, such as -`LocalStore` and `ZipStore`, never aliased the caller's memory. Uncompressed -writes to a `MemoryStore` are correspondingly slower, since the copy that makes -the stored data independent is now actually performed; compressed writes are -unchanged. Buffers supplied through the `store_dict` argument remain the -caller's responsibility and are stored as-is. diff --git a/changes/4172.misc.md b/changes/4172.misc.md deleted file mode 100644 index 0be7226476..0000000000 --- a/changes/4172.misc.md +++ /dev/null @@ -1,7 +0,0 @@ -Improved `CoordinateIndexer` construction for large, sorted, in-bounds, one-dimensional integer -coordinate selections over regular chunk grids (e.g. `arr.get_coordinate_selection(sorted_idx)`, -`arr.vindex[sorted_idx]`, and the gather behind sparse/CSR row selections). When boundary searching -is estimated to be cheaper than processing every coordinate, per-chunk projections are now built -with `searchsorted`, making index construction ~15x faster for large gathers. Sparse sorted -selections spanning many chunks relative to their coordinate count, as well as unsorted, negative, -multi-dimensional, and irregular-grid selections, continue to use the existing implementation. diff --git a/changes/4179.bugfix.md b/changes/4179.bugfix.md deleted file mode 100644 index e02523114c..0000000000 --- a/changes/4179.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the opt-in `FusedCodecPipeline` for sharded arrays whose inner or index codec chain contains a codec implementing only the async codec interface (no `SupportsSyncCodec`). Such arrays previously raised `TypeError: All codecs must implement SupportsSyncCodec` on both read and write; the pipeline now declines its synchronous fast path for them and falls back to the async path, matching the behavior of the default `BatchedCodecPipeline`. Fully sync-capable codec chains keep the fast path unchanged. diff --git a/changes/4183.bugfix.md b/changes/4183.bugfix.md deleted file mode 100644 index 809708f596..0000000000 --- a/changes/4183.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -Fixed `TypeError: unhashable type: 'writeable void-scalar'` when writing to sharded arrays whose fill value is a `np.void` scalar, e.g. arrays with a structured dtype. - -`ArraySpec` equality and hashing now compare the fill value by its byte representation rather than numeric equality. As a result, two specs with a `NaN` (or `NaT`) fill value now compare equal, while fill values of `-0.0` and `0.0` now compare unequal. This also restores the sharding codec's per-chunk spec cache, which had been disabled because of this bug. diff --git a/changes/4187.feature.md b/changes/4187.feature.md deleted file mode 100644 index 87133e2034..0000000000 --- a/changes/4187.feature.md +++ /dev/null @@ -1,4 +0,0 @@ -`ZipStore` now accepts an open binary file-like object in place of a path, enabling -zip archives on remote storage (e.g. a file opened with `fsspec` or an -`obstore.ReadableFile`). Operations that require a filesystem location -(`clear`, `move`) raise `NotImplementedError` for file-object-backed stores. diff --git a/changes/4194.bugfix.md b/changes/4194.bugfix.md deleted file mode 100644 index 21a4924664..0000000000 --- a/changes/4194.bugfix.md +++ /dev/null @@ -1,10 +0,0 @@ -`FusedCodecPipeline` no longer runs chunk IO and codec compute on the thread -driving zarr's internal event loop. Previously each read/write executed its -synchronous fast path inline on that loop thread, and because every sync-API -call from every user thread is serviced by the same loop, concurrent -operations serialized behind each other's codec work — reported as the fused -pipeline being slower than `BatchedCodecPipeline` for zstd-compressed data -under multi-threaded (e.g. dask) access. The synchronous batch now runs on a -worker thread (one hop per batch, not per chunk), keeping the loop free. -Multi-threaded single-chunk reads of zstd data are ~4.5x faster than before -and now scale with reader threads; single-threaded performance is unchanged. diff --git a/changes/4199.bugfix.md b/changes/4199.bugfix.md deleted file mode 100644 index d0c522cd7e..0000000000 --- a/changes/4199.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -The end-to-end benchmarks no longer invoke `sudo` to drop the OS page cache during a regular `pytest` run. Cache clearing is now opt-in via the `ZARR_BENCHMARK_CLEAR_CACHE` environment variable, which the benchmark CI jobs set. diff --git a/changes/4201.bugfix.md b/changes/4201.bugfix.md deleted file mode 100644 index d837a8a9e2..0000000000 --- a/changes/4201.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the opt-in `FusedCodecPipeline` for serializers that advertise the partial-decode/encode mixins with only the documented async partial methods: the partial dispatch previously asserted on the private `_decode_partial_sync`/`_encode_partial_sync` hooks (an `AssertionError`, or an `AttributeError` mid-IO under `python -O`); such codecs now take the full-chunk sync path. diff --git a/changes/4202.bugfix.md b/changes/4202.bugfix.md deleted file mode 100644 index 6130fc5b33..0000000000 --- a/changes/4202.bugfix.md +++ /dev/null @@ -1,10 +0,0 @@ -Fixed `FusedCodecPipeline` (the opt-in synchronous pipeline) silently skipping -array-array/bytes-bytes codecs placed outside a sharding serializer on its -partial-decode/partial-encode fast paths. With an outer compressor (e.g. -`compressors=[GzipCodec()]` around a `ShardingCodec` serializer), the fused -pipeline wrote non-conforming stored bytes that `BatchedCodecPipeline` (and any -other conforming reader) could not read, and could fail to read data that -`BatchedCodecPipeline` had written. With an outer array-array codec (e.g. -`TransposeCodec`), it silently returned wrong data in both directions with no -error. Only the opt-in `FusedCodecPipeline` was affected; the default -`BatchedCodecPipeline` was never impacted. diff --git a/changes/4203.bugfix.md b/changes/4203.bugfix.md deleted file mode 100644 index 42ca977193..0000000000 --- a/changes/4203.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed silent data corruption in the experimental `FusedCodecPipeline`: reordering or duplicating fancy-index reads (e.g. `arr[perm, :]`, `arr.oindex[[0, 0, 1], :]`) on uncompressed, crc-free sharded arrays could return the shard in natural order because the vectorized whole-shard decode accepted any selection whose output shape matched the shard shape. The bulk decode now fires only for identity full-shard reads, declines structured dtypes (whose byte-order handling it lacks), and requires shard-index offsets to exactly tile the data section, so corrupt indexes with overlapping or out-of-range offsets can no longer be served as array data. diff --git a/changes/4204.bugfix.md b/changes/4204.bugfix.md deleted file mode 100644 index 90101d1059..0000000000 --- a/changes/4204.bugfix.md +++ /dev/null @@ -1,16 +0,0 @@ -`ManagedMemoryStore.get_sync`/`set_sync`/`delete_sync` now apply the store's -`path` prefix, matching the async `get`/`set`/`delete` methods. Previously the -sync methods were inherited unchanged from `MemoryStore` and used the raw key, -so code that takes the sync fast path (e.g. `FusedCodecPipeline`) would read -and write chunks outside the store's `path` prefix, silently returning fill -values when the data was re-read through a fresh handle. `GpuMemoryStore.set_sync` -now converts its value to a `gpu.Buffer`, matching `set`, so writes through the -sync API keep the store's all-values-are-GPU invariant. Also fixed -`ManagedMemoryStore.get_partial_values` applying its `path` prefix twice -whenever `path` is non-empty, which made it always return `None` for every -requested key. - -The shared store test suite (`zarr.testing.store.StoreTests`) gained -sync/async parity checks — comparing sync and async observations of the same -key on the same store instance, including with a `byte_range` — so every -store subclass now exercises this invariant. diff --git a/changes/4205.bugfix.md b/changes/4205.bugfix.md deleted file mode 100644 index 0492febb7d..0000000000 --- a/changes/4205.bugfix.md +++ /dev/null @@ -1,9 +0,0 @@ -Fixed several small correctness issues from the codec-pipeline performance work: construction-time -codec warnings (e.g. sharding's "disables partial reads" warning) no longer fire twice per array -open — including for `FusedCodecPipeline`, which previously re-warned via its own codec-chain -reconstruction and, on the async fallback path, on every decode/encode call; `concurrent_iter` now -schedules its tasks eagerly, matching its documented contract; an invalid -`codec_pipeline.max_workers` config/environment value now warns and falls back to the default -instead of raising mid-read; and `FusedCodecPipeline`'s async fallback helpers now cancel -already-spawned fetch/decode/write tasks instead of abandoning them in the background when one -fails. diff --git a/changes/4206.bugfix.md b/changes/4206.bugfix.md deleted file mode 100644 index a01c969449..0000000000 --- a/changes/4206.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `FusedCodecPipeline`'s gating of its synchronous fast paths: stores exposing only part of the sync surface (e.g. `set_sync` without `get_sync`) now fall back cleanly to the async path instead of failing mid-write, and `WrapperStore` now forwards `get_sync`/`set_sync`/`delete_sync` to the wrapped store so wrapped sync-capable stores keep the fast path. The capability decision uses a private, interim convention (`zarr.abc.store._store_supports_sync_io`) rather than new public API, pending a formal sync/async store architecture. Also fixed `LatencyStore`: synchronous reads and writes now pay the configured latency, `get_ranges`/`get_partial_values` no longer bypass latency injection, and derived stores (e.g. from `with_read_only`) keep a stochastic `(loc, scale)` latency configuration instead of freezing a single sample. diff --git a/changes/4219.bugfix.md b/changes/4219.bugfix.md deleted file mode 100644 index 728e8a7e3a..0000000000 --- a/changes/4219.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -`DefaultChunkKeyEncoding.decode_chunk_key` now validates that a chunk key -starts with the configured `c` prefix and raises `ValueError` for -malformed keys, instead of silently decoding them incorrectly. diff --git a/docs/blog/.authors.yml b/docs/blog/.authors.yml new file mode 100644 index 0000000000..10ce423cfc --- /dev/null +++ b/docs/blog/.authors.yml @@ -0,0 +1,6 @@ +authors: + d-v-b: + name: Davis Bennett + description: Core developer + avatar: https://github.com/d-v-b.png + url: https://github.com/d-v-b diff --git a/docs/blog/index.md b/docs/blog/index.md new file mode 100644 index 0000000000..fca29e2578 --- /dev/null +++ b/docs/blog/index.md @@ -0,0 +1,3 @@ +# Blog + +News, release highlights, and design notes from the Zarr-Python developers. diff --git a/docs/blog/posts/3.3.0-release.md b/docs/blog/posts/3.3.0-release.md new file mode 100644 index 0000000000..13368848ae --- /dev/null +++ b/docs/blog/posts/3.3.0-release.md @@ -0,0 +1,169 @@ +--- +date: 2026-07-30 +authors: + - d-v-b +categories: + - Release +--- + +# Zarr-Python 3.3.0 + +We're happy to announce the release of version 3.3.0 of Zarr-Python. It's been a while since our last release ([3.2.1](https://github.com/zarr-developers/zarr-python/releases/tag/v3.2.1) dropped in May of this year), +and we're bringing some exciting additions to the latest version. For the full release notes, see the [3.3.0 release notes](../../release-notes.md), otherwise stick around for an overview of two performance-centric highlights of this release. + + + +## Faster low-latency storage + +Relevant issues and pull requests: + +- [#3524](https://github.com/zarr-developers/zarr-python/issues/3524) -- the performance report that started this work +- [#3885](https://github.com/zarr-developers/zarr-python/pull/3885) -- synchronous codec APIs and the `FusedCodecPipeline` + +### The cost of async overhead + +Zarr-Python 3.x uses async routines for fetching data and decoding chunks. In terms of code, this means our store (data fetching) and codec (chunk decoding) APIs are both async. This makes +I/O against high-latency storage backends like cloud object storage efficient. But for *low-latency* storage, like in-process memory or the file system, async routines add measurable overhead and offer no benefit. Async only adds value when there's work to be done while waiting for I/O to complete, but when I/O latency is low, it completes too quickly to run anything while waiting, and we are left paying the performance bill for obligatory async task scheduling that offered no value. + +This performance problem became acute when Zarr-Python users reported that in-memory array indexing workloads ran *slower* in Zarr-Python 3.1.3 relative to Zarr-Python 2.18.7 ([#3524](https://github.com/zarr-developers/zarr-python/issues/3524)). Fortunately this performance regression had a straightforward fix (I don't say "easy" because it was a lot of work). + +### Synchronous execution restores performance + +If async overhead makes low-latency storage slow, does *removing* that overhead restore performance? Yes, it does! + +In [#3885](https://github.com/zarr-developers/zarr-python/pull/3885) we defined synchronous versions of our storage and codec APIs -- the `SyncByteGetter` and `SyncByteSetter` protocols, plus a `get_ranges_sync` method on the `Store` ABC -- and then combined them in a new codec orchestration class called `FusedCodecPipeline`. The `FusedCodecPipeline` is an opt-in alternative to the default (the `BatchedCodecPipeline`) that gives large speedups for low-latency storage. It is currently marked [experimental](../../user-guide/experimental.md), so we may change it as we learn more; the default pipeline is untouched, and existing code keeps working unless you opt in. + +The win here is *not* a faster compressor. It is the removal of async scheduling overhead (including some [nasty `asyncio.to_thread` overhead](https://github.com/python/cpython/issues/136084)), plus a few vectorized fast paths for dense, uncompressed shards. And we only expect this new pipeline to accelerate workloads targeting a subset of storage backends, namely any store with methods that advertise low latency. + +On this author's 10-core Apple M4 laptop, the `FusedCodecPipeline` delivers the following results against memory-backed arrays: + +- uncompressed writes are *~4 times faster* +- uncompressed reads are *~5 times faster* +- compressed writes are *~2 times faster* +- compressed reads are *~2 times faster* + +These numbers came from a [runnable example](../../user-guide/examples/codec_pipeline_performance.md) that ships with the documentation. Run it yourself to get a sense of how the `FusedCodecPipeline` behaves on your system -- when and how you use it depends on your hardware, your array layout, and how your chunks are compressed. What's certain is that for in-memory arrays, and arrays saved to the local file system, the `FusedCodecPipeline` is worth a try. + +Getting good numbers requires choosing the right level of thread-based parallelism for your workload, which is part of the configuration of the `FusedCodecPipeline`. For uncompressed chunks there's no CPU-bound work to do after fetching a chunk and so +thread-based parallelism is worse than useless and slows things down. But for compressed chunks, threading offers a substantial payoff. + +### How to use it + +Select the pipeline through the [runtime configuration](../../user-guide/config.md) by setting `codec_pipeline.path`. Set it globally to affect every array created or opened afterwards: + +```python exec="true" session="blog-330" source="above" +import zarr + +zarr.config.set( + {"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"} +) +``` + +Or scope it to a block of code by using `zarr.config.set` as a context manager, which is the safer choice if you only want the new pipeline for part of your program: + +```python exec="true" session="blog-330" source="above" result="ansi" +import numpy as np +import zarr +from zarr.storage import MemoryStore + +with zarr.config.set( + {"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"} +): + arr = zarr.create_array( + store=MemoryStore(), + shape=(1000, 1000), + chunks=(100, 100), + shards=(1000, 1000), + dtype="float32", + ) + arr[:] = np.random.random((1000, 1000)).astype("float32") + result = arr[:] + +print(result.shape) +``` + +Thread-based parallelism is configured separately, via `codec_pipeline.max_workers`. It defaults to `None`, meaning a pool sized to `os.cpu_count()`. Note that this setting is read *only* by the `FusedCodecPipeline` -- the default `BatchedCodecPipeline` ignores it, so tuning it without opting in above does nothing. + +As noted, memory-backed and uncompressed workloads often do better with a single worker, which runs everything inline on the calling thread: + +```python exec="true" session="blog-330" source="above" +import zarr + +# No thread pool: run codec compute inline. Often best for uncompressed, +# memory-backed arrays, where there's no CPU-bound work to overlap. +zarr.config.set({"codec_pipeline.max_workers": 1}) + +# A fixed-size thread pool, which pays off once compression is in play. +zarr.config.set({"codec_pipeline.max_workers": 8}) + +# Or back to the default, sized to the number of CPUs. +zarr.config.set({"codec_pipeline.max_workers": None}) +``` + +To return to the default pipeline, set `codec_pipeline.path` back to the batched implementation: + +```python exec="true" session="blog-330" source="above" +import zarr + +zarr.config.set( + {"codec_pipeline.path": "zarr.core.codec_pipeline.BatchedCodecPipeline"} +) +``` + +## Faster sharded reads + +Relevant issues and pull requests: + +- [#3004](https://github.com/zarr-developers/zarr-python/pull/3004) -- optimize partial shard reads +- [#3925](https://github.com/zarr-developers/zarr-python/pull/3925) -- `Store.get_ranges` for concurrent, coalesced multi-range reads +- [#3987](https://github.com/zarr-developers/zarr-python/pull/3987) -- control coalescing through `ArrayConfig` and the runtime config + +### How sharding works + +Chunks encoded with the `sharding_indexed` codec contain a secondary level of chunking, called subchunks. For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. + +A reader can satisfy a request for all four inner chunks by issuing four separate byte-range requests, or by making a *single* request for a byte range that spans all four inner chunks. The latter option is nice because it cuts down on the number of requests we need. Historically Zarr-Python used this optimization when reading entire outer chunks; in 3.3.0, we use this optimization in more cases, resulting in more efficient I/O patterns for sharded reads. + +### Interval equivalence + +Byte ranges, being intervals, obey some combination rules: the values in two half-open intervals `[a, b), [b, c)` can be captured by the single interval `[a, c)`. That means a reader can get multiple inner chunks with *one* byte-range request by requesting a range of bytes starting with the first byte of the first subchunk and ending with the last byte of the last subchunk. When individual requests are expensive, this kind of optimization is worth a lot. + +The requested inner chunks are not necessarily contiguous -- there might be a byte range gap between them. As long as that gap is not too big, its often efficient to fetch the entire byte range, gap included, and pick out the inner chunk byte ranges after I/O is done. + +### Byte range coalescing + +We call this procedure -- merging adjacent byte ranges -- "byte range coalescing", and it's a new performance optimization shipping in Zarr-Python 3.3.0. Unlike the `FusedCodecPipeline`, this one is on by default with base settings we think are good, so most users won't need to tune anything. + +Two knobs control it, both documented in the [runtime configuration guide](../../user-guide/config.md). Nearby byte ranges in the same shard are merged into a single request when the gap between them is no larger than `array.sharding_coalesce_max_gap_bytes` (default 1 MiB) and the merged read stays within `array.sharding_coalesce_max_bytes` (default 16 MiB). The gap threshold is what trades wasted bytes against saved requests: raising it reads more data you didn't ask for, in exchange for fewer requests. + +For a runnable demonstration -- counting the store requests saved and timing them against a store with simulated latency -- see the [sharded read coalescing example](../../user-guide/examples/sharding_coalescing.md). + +You can set them globally, or per array by passing `config={...}` to [`zarr.create_array`][]: + +```python exec="true" session="blog-330" source="above" result="ansi" +import zarr +from zarr.storage import MemoryStore + +arr = zarr.create_array( + store=MemoryStore(), + shape=(1000, 1000), + chunks=(100, 100), + shards=(1000, 1000), + dtype="float32", + config={ + "sharding_coalesce_max_gap_bytes": 4 * 1024**2, # 4 MiB + "sharding_coalesce_max_bytes": 64 * 1024**2, # 64 MiB + }, +) +print(arr.shape) +``` + +## Tell us what you think + +We hope these new features are helpful, and we would appreciate any feedback that helps us improve them, or any other aspect of Zarr-Python. + +## Going faster + +The updates in this release are just the first step of a larger performance-oriented direction for Zarr-Python. Landing these two enhancements taught us a *lot* about the performance-sensitive areas of the library. We can and will invest more time in performance tuning, e.g. by adding or changing abstractions, writing code for special cases, etc. + +We plan to consider including compiled code that should enable significant performance improvements. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned! diff --git a/docs/release-notes.md b/docs/release-notes.md index 7b147a30bd..3b54ea993a 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -4,7 +4,7 @@ -## 3.3.0 (2026-07-15) +## 3.3.0 (2026-07-30) ### Features @@ -14,13 +14,19 @@ concurrently. ([#3004](https://github.com/zarr-developers/zarr-python/pull/3004)) - Added a `subchunk_write_order` option to `ShardingCodec` to control the physical order of subchunks within a shard. Supported values are `morton`, `unordered`, `lexicographic`, and `colexicographic`. `unordered` makes no guarantee about subchunk layout. This setting affects only on-disk layout, not the data read back, and is not persisted in array metadata: it applies per codec instance and is not recovered when reopening a sharded array. ([#3826](https://github.com/zarr-developers/zarr-python/pull/3826)) - Added `SyncByteGetter` and `SyncByteSetter` runtime-checkable protocols and a `get_ranges_sync` method on the `Store` ABC. These let custom byte getters/setters opt into the synchronous codec pipeline's fast path for in-memory IO, which the sharding codec uses for its inner chunks. ([#3885](https://github.com/zarr-developers/zarr-python/pull/3885)) -- Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays (up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression) and no regressions on compute-bound workloads. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. ([#3885](https://github.com/zarr-developers/zarr-python/pull/3885)) +- Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. ([#3885](https://github.com/zarr-developers/zarr-python/pull/3885)) - Add `zarr.abc.store.Store.get_ranges` for concurrent, coalesced multi-range reads from a single key. The method is defined on the `Store` ABC with a default implementation built on `Store.get`, so every store inherits a working version; stores with native multi-range backends (e.g. `FsspecStore`) can override for efficiency. Coalescing knobs (`max_concurrency`, `max_gap_bytes`, `max_coalesced_bytes`) are passed as keyword arguments to `get_ranges`. Failures from underlying fetches surface as a `BaseExceptionGroup` (PEP 654); callers should use `except*` to filter for specific exception types such as `FileNotFoundError`. ([#3925](https://github.com/zarr-developers/zarr-python/pull/3925)) - Two new fields on `ArrayConfig` control how the sharding codec coalesces partial-shard reads: `sharding_coalesce_max_gap_bytes` (default 1 MiB) and `sharding_coalesce_max_bytes` (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than `sharding_coalesce_max_gap_bytes` and the merged read stays within `sharding_coalesce_max_bytes`. Defaults are seeded from the matching `array.sharding_coalesce_max_gap_bytes` / `array.sharding_coalesce_max_bytes` keys in [`zarr.config`][] at array-creation time, and can be overridden per array by passing `config={...}` to [`zarr.create_array`][]. ([#3987](https://github.com/zarr-developers/zarr-python/pull/3987)) +- Added `Group.get_array`, `Group.get_group`, `AsyncGroup.get_array`, and `AsyncGroup.get_group`: type-safe accessors that return the child array or group at a given path, raising `ArrayNotFoundError` / `GroupNotFoundError` if no node exists there, and `ContainsGroupError` / `ContainsArrayError` if the node is not of the requested kind. Unlike `Group.__getitem__`, which returns `Array | Group`, these methods have precise return types. Nested paths like `"subgroup/subarray"` are supported. ([#4128](https://github.com/zarr-developers/zarr-python/pull/4128)) +- `ZipStore` now accepts an open binary file-like object in place of a path, enabling + zip archives on remote storage (e.g. a file opened with `fsspec` or an + `obstore.ReadableFile`). Operations that require a filesystem location + (`clear`, `move`) raise `NotImplementedError` for file-object-backed stores. ([#4187](https://github.com/zarr-developers/zarr-python/pull/4187)) + ### Bugfixes -- Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. ([#202](https://github.com/zarr-developers/zarr-python/issues/202)) +- Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. ([#4100](https://github.com/zarr-developers/zarr-python/pull/4100)) - Fix equality comparison of `ArrayV2Metadata` and `ArrayV3Metadata` objects with a `NaN` fill value. Such objects are now compared by their JSON-serialized form, so two otherwise-identical metadata objects with a `NaN` (or infinite) fill value compare equal. ([#2929](https://github.com/zarr-developers/zarr-python/issues/2929)) @@ -46,25 +52,11 @@ - Fixed writing to 0-dimensional arrays that use the sharding codec. Previously assigning to a 0-dimensional sharded array raised an error. ([#3966](https://github.com/zarr-developers/zarr-python/pull/3966)) - Fix flaky stateful test bookkeeping when `delete_dir` matches string prefixes instead of true directory descendants. Previously a path such as `6/faNT…` could be incorrectly removed when deleting `6/f`. (See [issue #3977](https://github.com/zarr-developers/zarr-python/issues/3977).) ([#3977](https://github.com/zarr-developers/zarr-python/issues/3977)) -- `FsspecStore.from_url()` and `from_mapper()` now close the async filesystem - they create when `store.close()` is called. Previously the underlying aiohttp - `ClientSession` was left open until garbage collection, producing - `"Unclosed client session"` `ResourceWarning`s from aiohttp. - - The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when - `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` - when a sync→async conversion was performed). When `_owns_fs` is ``True``, - `store.close()` calls the new `_close_fs()` helper, which invokes - `fs.set_session()` and closes the returned client. Callers who supply their own - filesystem instance to `FsspecStore()` directly remain responsible for its - lifecycle; `_owns_fs` is ``False`` for those stores. - - **Scope note**: This fix closes the S3 client session that is active at the time - `store.close()` is called. Some S3-backed filesystem implementations (e.g. - s3fs with ``cache_regions=True``) may internally refresh and replace their - client during I/O operations, abandoning prior sessions before ``store.close()`` - is invoked. Those intermediate sessions are outside the scope of this fix and - are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/pull/4003)) +- `FsspecStore.close()` no longer closes the underlying fsspec filesystem or its + network session. fsspec caches and shares filesystem instances across callers, + so the store cannot know whether it is the only user, and closing a shared + session would break other stores; the filesystem's lifecycle belongs to + whoever created it. ([#4165](https://github.com/zarr-developers/zarr-python/pull/4165)) - Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) - Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) @@ -82,6 +74,82 @@ - Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/pull/4116)) - Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) +- Fix `zarr.api.asynchronous.open_like` so it can create a new array by default when the + target path does not already exist. It now defaults to `mode="a"`; when using a read-only + store to open an existing array, pass `mode="r"` explicitly. ([#3352](https://github.com/zarr-developers/zarr-python/pull/3352)) +- `MemoryStore` now copies buffers as they are written, so it never retains the + caller's memory. Previously an uncompressed write handed the store a zero-copy + view of the user's array, and mutating that array afterwards would silently + rewrite chunks already committed to the store. + + Only `MemoryStore` is affected: stores that serialize on write, such as + `LocalStore` and `ZipStore`, never aliased the caller's memory. Uncompressed + writes to a `MemoryStore` are correspondingly slower, since the copy that makes + the stored data independent is now actually performed; compressed writes are + unchanged. Buffers supplied through the `store_dict` argument remain the + caller's responsibility and are stored as-is. ([#4157](https://github.com/zarr-developers/zarr-python/pull/4157)) + +- Fixed the opt-in `FusedCodecPipeline` for sharded arrays whose inner or index codec chain contains a codec implementing only the async codec interface (no `SupportsSyncCodec`). Such arrays previously raised `TypeError: All codecs must implement SupportsSyncCodec` on both read and write; the pipeline now declines its synchronous fast path for them and falls back to the async path, matching the behavior of the default `BatchedCodecPipeline`. Fully sync-capable codec chains keep the fast path unchanged. ([#4179](https://github.com/zarr-developers/zarr-python/pull/4179)) +- Fixed `TypeError: unhashable type: 'writeable void-scalar'` when writing to sharded arrays whose fill value is a `np.void` scalar, e.g. arrays with a structured dtype. + + `ArraySpec` equality and hashing now compare the fill value by its byte representation rather than numeric equality. As a result, two specs with a `NaN` (or `NaT`) fill value now compare equal, while fill values of `-0.0` and `0.0` now compare unequal. This also restores the sharding codec's per-chunk spec cache, which had been disabled because of this bug. ([#4183](https://github.com/zarr-developers/zarr-python/pull/4183)) + +- `FusedCodecPipeline` no longer runs chunk IO and codec compute on the thread + driving zarr's internal event loop. Previously each read/write executed its + synchronous fast path inline on that loop thread, and because every sync-API + call from every user thread is serviced by the same loop, concurrent + operations serialized behind each other's codec work — reported as the fused + pipeline being slower than `BatchedCodecPipeline` for zstd-compressed data + under multi-threaded (e.g. dask) access. The synchronous batch now runs on a + worker thread (one hop per batch, not per chunk), keeping the loop free. + Multi-threaded single-chunk reads of compressed data now scale with reader + threads; single-threaded performance is unchanged. ([#4194](https://github.com/zarr-developers/zarr-python/pull/4194)) +- The end-to-end benchmarks no longer invoke `sudo` to drop the OS page cache during a regular `pytest` run. Cache clearing is now opt-in via the `ZARR_BENCHMARK_CLEAR_CACHE` environment variable, which the benchmark CI jobs set. ([#4199](https://github.com/zarr-developers/zarr-python/pull/4199)) +- Fixed the opt-in `FusedCodecPipeline` for serializers that advertise the partial-decode/encode mixins with only the documented async partial methods: the partial dispatch previously asserted on the private `_decode_partial_sync`/`_encode_partial_sync` hooks (an `AssertionError`, or an `AttributeError` mid-IO under `python -O`); such codecs now take the full-chunk sync path. ([#4201](https://github.com/zarr-developers/zarr-python/pull/4201)) +- Fixed `FusedCodecPipeline` (the opt-in synchronous pipeline) silently skipping + array-array/bytes-bytes codecs placed outside a sharding serializer on its + partial-decode/partial-encode fast paths. With an outer compressor (e.g. + `compressors=[GzipCodec()]` around a `ShardingCodec` serializer), the fused + pipeline wrote non-conforming stored bytes that `BatchedCodecPipeline` (and any + other conforming reader) could not read, and could fail to read data that + `BatchedCodecPipeline` had written. With an outer array-array codec (e.g. + `TransposeCodec`), it silently returned wrong data in both directions with no + error. Only the opt-in `FusedCodecPipeline` was affected; the default + `BatchedCodecPipeline` was never impacted. ([#4202](https://github.com/zarr-developers/zarr-python/pull/4202)) +- Fixed silent data corruption in the experimental `FusedCodecPipeline`: reordering or duplicating fancy-index reads (e.g. `arr[perm, :]`, `arr.oindex[[0, 0, 1], :]`) on uncompressed, crc-free sharded arrays could return the shard in natural order because the vectorized whole-shard decode accepted any selection whose output shape matched the shard shape. The bulk decode now fires only for identity full-shard reads, declines structured dtypes (whose byte-order handling it lacks), and requires shard-index offsets to exactly tile the data section, so corrupt indexes with overlapping or out-of-range offsets can no longer be served as array data. ([#4203](https://github.com/zarr-developers/zarr-python/pull/4203)) +- `ManagedMemoryStore.get_sync`/`set_sync`/`delete_sync` now apply the store's + `path` prefix, matching the async `get`/`set`/`delete` methods. Previously the + sync methods were inherited unchanged from `MemoryStore` and used the raw key, + so code that takes the sync fast path (e.g. `FusedCodecPipeline`) would read + and write chunks outside the store's `path` prefix, silently returning fill + values when the data was re-read through a fresh handle. `GpuMemoryStore.set_sync` + now converts its value to a `gpu.Buffer`, matching `set`, so writes through the + sync API keep the store's all-values-are-GPU invariant. Also fixed + `ManagedMemoryStore.get_partial_values` applying its `path` prefix twice + whenever `path` is non-empty, which made it always return `None` for every + requested key. + + The shared store test suite (`zarr.testing.store.StoreTests`) gained + sync/async parity checks — comparing sync and async observations of the same + key on the same store instance, including with a `byte_range` — so every + store subclass now exercises this invariant. The suite's former + `test_get_bytes`/`test_get_json` methods (and their `_sync` variants) were + folded into these parity tests and no longer exist as separate methods. ([#4204](https://github.com/zarr-developers/zarr-python/pull/4204)) + +- Fixed several small correctness issues from the codec-pipeline performance work: construction-time + codec warnings (e.g. sharding's "disables partial reads" warning) no longer fire twice per array + open — including for `FusedCodecPipeline`, which previously re-warned via its own codec-chain + reconstruction and, on the async fallback path, on every decode/encode call; `concurrent_iter` now + schedules its tasks eagerly, matching its documented contract; an invalid + `codec_pipeline.max_workers` config/environment value now warns and falls back to the default + instead of raising mid-read; and `FusedCodecPipeline`'s async fallback helpers now cancel + already-spawned fetch/decode/write tasks instead of abandoning them in the background when one + fails. ([#4205](https://github.com/zarr-developers/zarr-python/pull/4205)) +- Fixed `FusedCodecPipeline`'s gating of its synchronous fast paths: stores exposing only part of the sync surface (e.g. `set_sync` without `get_sync`) now fall back cleanly to the async path instead of failing mid-write, and `WrapperStore` now forwards `get_sync`/`set_sync`/`delete_sync` to the wrapped store so wrapped sync-capable stores keep the fast path. The capability decision uses a private, interim convention (`zarr.abc.store._store_supports_sync_io`) rather than new public API, pending a formal sync/async store architecture. Also fixed `LatencyStore`: synchronous reads and writes now pay the configured latency, `get_ranges`/`get_partial_values` no longer bypass latency injection, and derived stores (e.g. from `with_read_only`) keep a stochastic `(loc, scale)` latency configuration instead of freezing a single sample. ([#4206](https://github.com/zarr-developers/zarr-python/pull/4206)) +- `DefaultChunkKeyEncoding.decode_chunk_key` now validates that a chunk key + starts with the configured `c` prefix and raises `ValueError` for + malformed keys, instead of silently decoding them incorrectly. ([#4219](https://github.com/zarr-developers/zarr-python/pull/4219)) + ### Improved Documentation - Document the changes to `zarr.errors` in the 3.0 migration guide, including the removal of v2 exception classes and the introduction of `NodeNotFoundError`. ([#3009](https://github.com/zarr-developers/zarr-python/issues/3009)) @@ -120,13 +188,30 @@ enumeration, bulk attribute updates, and the `use_consolidated` keyword. ([#4132](https://github.com/zarr-developers/zarr-python/pull/4132)) - Fixed the documented default of ``max_age_seconds`` in the ``CacheStore`` docstring: the default is ``"infinity"`` (no expiration), not ``None``, which is rejected. Also noted that ``cache_store`` must support deletes. ([#4133](https://github.com/zarr-developers/zarr-python/pull/4133)) +- Added a blog section to the documentation, with a post covering two performance + highlights of the 3.3.0 release: the opt-in `FusedCodecPipeline` and byte-range + coalescing for partial reads of sharded arrays. + + Added two runnable examples that accompany the post: + `examples/codec_pipeline_performance` compares the `BatchedCodecPipeline` and + `FusedCodecPipeline` on a sharded array across two stores and two codec + regimes, showing when the fused pipeline's thread pool helps and when it does + not, and `examples/sharding_coalescing` demonstrates how read coalescing + reduces the number of store requests when reading subregions of a sharded + array. + + Also removed the hardware-specific speedup figures from the `FusedCodecPipeline` + release note, since they depend on the array layout, codec, and machine. ([#4191](https://github.com/zarr-developers/zarr-python/pull/4191)) + ### Deprecations and Removals - The ``BloscShuffle`` and ``BloscCname`` enums (``zarr.codecs.BloscShuffle``, ``zarr.codecs.BloscCname``) are now deprecated. Pass the equivalent literal string (e.g. ``"zstd"``, ``"bitshuffle"``) when constructing a ``BloscCodec``. The enum classes remain importable but emit ``DeprecationWarning`` on member - access, and will be removed in a future release. ``BloscCodec.cname`` and + access, and will be removed in a future release. They are no longer ``Enum`` + subclasses: constructor calls (e.g. ``BloscCname("zstd")``), iteration, and + ``.value`` access no longer work. ``BloscCodec.cname`` and ``BloscCodec.shuffle`` are now plain strings rather than enum members. Additional renames in ``zarr.codecs.blosc`` from the same change: the type @@ -162,8 +247,9 @@ ### Misc -- [#214](https://github.com/zarr-developers/zarr-python/issues/214), [#215](https://github.com/zarr-developers/zarr-python/pull/215), [#3908](https://github.com/zarr-developers/zarr-python/pull/3908), [#3972](https://github.com/zarr-developers/zarr-python/pull/3972), [#3975](https://github.com/zarr-developers/zarr-python/pull/3975), [#3979](https://github.com/zarr-developers/zarr-python/pull/3979), [#3990](https://github.com/zarr-developers/zarr-python/pull/3990), [#3998](https://github.com/zarr-developers/zarr-python/pull/3998), [#4000](https://github.com/zarr-developers/zarr-python/pull/4000), [#4001](https://github.com/zarr-developers/zarr-python/pull/4001), [#4046](https://github.com/zarr-developers/zarr-python/pull/4046), [#4054](https://github.com/zarr-developers/zarr-python/pull/4054), [#4073](https://github.com/zarr-developers/zarr-python/issues/4073), [#4086](https://github.com/zarr-developers/zarr-python/issues/4086), [#4138](https://github.com/zarr-developers/zarr-python/pull/4138) +- [#4139](https://github.com/zarr-developers/zarr-python/pull/4139), [#4140](https://github.com/zarr-developers/zarr-python/pull/4140), [#3908](https://github.com/zarr-developers/zarr-python/pull/3908), [#3972](https://github.com/zarr-developers/zarr-python/pull/3972), [#3975](https://github.com/zarr-developers/zarr-python/pull/3975), [#3979](https://github.com/zarr-developers/zarr-python/pull/3979), [#3990](https://github.com/zarr-developers/zarr-python/pull/3990), [#3998](https://github.com/zarr-developers/zarr-python/pull/3998), [#4000](https://github.com/zarr-developers/zarr-python/pull/4000), [#4001](https://github.com/zarr-developers/zarr-python/pull/4001), [#4012](https://github.com/zarr-developers/zarr-python/pull/4012), [#4046](https://github.com/zarr-developers/zarr-python/pull/4046), [#4054](https://github.com/zarr-developers/zarr-python/pull/4054), [#4073](https://github.com/zarr-developers/zarr-python/issues/4073), [#4086](https://github.com/zarr-developers/zarr-python/issues/4086), [#4138](https://github.com/zarr-developers/zarr-python/pull/4138) +- [#4172](https://github.com/zarr-developers/zarr-python/pull/4172) ## 3.2.1 (2026-05-05) diff --git a/docs/user-guide/examples/codec_pipeline_performance.md b/docs/user-guide/examples/codec_pipeline_performance.md new file mode 100644 index 0000000000..f21e31636e --- /dev/null +++ b/docs/user-guide/examples/codec_pipeline_performance.md @@ -0,0 +1,7 @@ +--8<-- "examples/codec_pipeline_performance/README.md" + +## Source Code + +```python exec="false" reason="pymdownx snippet include directive, not python source" +--8<-- "examples/codec_pipeline_performance/codec_pipeline_performance.py" +``` diff --git a/docs/user-guide/examples/sharding_coalescing.md b/docs/user-guide/examples/sharding_coalescing.md new file mode 100644 index 0000000000..8b2e054af5 --- /dev/null +++ b/docs/user-guide/examples/sharding_coalescing.md @@ -0,0 +1,7 @@ +--8<-- "examples/sharding_coalescing/README.md" + +## Source Code + +```python exec="false" reason="pymdownx snippet include directive, not python source" +--8<-- "examples/sharding_coalescing/sharding_coalescing.py" +``` diff --git a/examples/codec_pipeline_performance/README.md b/examples/codec_pipeline_performance/README.md new file mode 100644 index 0000000000..5d85412c29 --- /dev/null +++ b/examples/codec_pipeline_performance/README.md @@ -0,0 +1,59 @@ +# Codec Pipeline Performance + +This example compares the default `BatchedCodecPipeline` against the opt-in +`FusedCodecPipeline` on a sharded array, across two stores (memory and local) +and two codec regimes (uncompressed and gzip), at one worker and at `cpu_count`. + +A *codec pipeline* turns chunks of array data into stored bytes and back, running +the configured codecs and performing the storage IO. The default +`BatchedCodecPipeline` schedules both asynchronously -- roughly one coroutine per +chunk operation. That model pays off for high-latency stores, where there is +useful work to do while waiting on IO. For low-latency stores (in-process memory, +the local filesystem) the IO completes too quickly for the overlap to be worth +its cost, and the async scheduling becomes pure overhead. + +`FusedCodecPipeline` runs codec compute and synchronous IO synchronously, +removing that overhead. It is +[experimental](https://zarr.readthedocs.io/en/stable/user-guide/experimental/) +and opt-in; the default pipeline is unchanged. + +## What it shows + +- How to select a pipeline with `zarr.config.set`, and why the array must be + *created* inside the config block: the pipeline class is resolved at array + construction time and then travels with the array. +- That the benefit depends strongly on layout and on whether compression is in + play. Some configurations are slower under the fused pipeline -- the script + reports speedups below 1.00x rather than hiding them. +- That `codec_pipeline.max_workers` is read only by `FusedCodecPipeline`; the + default pipeline ignores it entirely. + +## Running + +```bash +uv run codec_pipeline_performance.py +``` + +The script has no arguments and writes only to an in-memory store. + +## Interpreting the output + +The numbers are specific to your CPU, your Python build, and the workload chosen +here. They are a measurement of your machine, not a published benchmark -- treat +a single run as indicative and re-measure against your own data and store before +switching pipelines in production. + +Two effects are worth watching for: + +- **The two codec regimes tell opposite stories about `max_workers`.** + Uncompressed IO is dominated by per-chunk *scheduling*, so `Fused (1 worker)` + is already fastest and a thread pool only adds overhead. gzip is genuinely + CPU-bound: a single worker compresses chunk after chunk sequentially and can + be *slower than the default*, while a thread pool spreads that compression + over cores and reclaims the win. That flip is why the fused pipeline is + threaded by default, and why pinning `max_workers=1` is worth it for + memory-backed uncompressed data. +- **Chunk size decides whether threading can help at all.** The 64×64 inner + chunks here are small enough that per-chunk scheduling dominates uncompressed + IO, yet large enough that per-chunk gzip is real work to parallelize. Much + coarser chunks leave the pool with too few items to spread. diff --git a/examples/codec_pipeline_performance/codec_pipeline_performance.py b/examples/codec_pipeline_performance/codec_pipeline_performance.py new file mode 100644 index 0000000000..b821f3e6a7 --- /dev/null +++ b/examples/codec_pipeline_performance/codec_pipeline_performance.py @@ -0,0 +1,214 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main", +# "numpy", +# ] +# /// + +""" +Compare the `BatchedCodecPipeline` and the `FusedCodecPipeline`. + +The default `BatchedCodecPipeline` schedules storage IO and codec compute +asynchronously -- roughly one coroutine per chunk operation. For a *sharded* +array that means one coroutine per inner chunk inside every shard. That is the +right model for high-latency stores, where there is useful work to do while +waiting for IO. For low-latency stores (in-process memory, the local +filesystem) the IO completes too quickly for the overlap to pay for itself, and +the scheduling becomes pure overhead. + +The `FusedCodecPipeline` runs codec compute and synchronous IO synchronously, +removing that overhead. Whether it wins, and whether its thread pool helps, +depends on which resource is actually scarce: + + * Uncompressed IO is dominated by per-chunk *scheduling*, not compute. There + is nothing for a thread pool to parallelize, so a single worker is already + fastest and extra workers only add overhead. + * gzip is genuinely CPU-bound. A single worker compresses every chunk + sequentially and can be *slower than the default*, while a thread pool + spreads that compression across cores and reclaims the win. This is when + `max_workers > 1` earns its keep. + +Run it with: + + uv run codec_pipeline_performance.py + +Numbers are hardware-, layout-, and codec-dependent. Treat the output as a +measurement of *your* machine, not as a published benchmark. +""" + +from __future__ import annotations + +import operator +import os +import statistics +import tempfile +import timeit +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np + +import zarr +from zarr.storage import LocalStore, MemoryStore + +if TYPE_CHECKING: + from collections.abc import Callable + + from zarr.abc.store import Store + +BATCHED = "zarr.core.codec_pipeline.BatchedCodecPipeline" +FUSED = "zarr.core.codec_pipeline.FusedCodecPipeline" + +# gzip is CPU-bound to encode, which is exactly the regime where the thread +# pool matters. Level 6 is gzip's own default. +GZIP = {"name": "gzip", "configuration": {"level": 6}} + +# 4096x4096 int32 = 64 MiB, split into 16 shards of 1024x1024, each holding +# 16x16 = 256 inner chunks of 64x64 -> 4096 inner chunks in total. The chunks +# are small enough that per-chunk coroutine scheduling dominates uncompressed +# IO, yet large enough that per-chunk gzip is real work to spread over cores. +SHAPE = (4096, 4096) +SHARDS = (1024, 1024) +CHUNKS = (64, 64) +DTYPE = "int32" + +CONFIGS: tuple[tuple[str, dict[str, object]], ...] = ( + ("Batched (default)", {"codec_pipeline.path": BATCHED}), + ("Fused (1 worker)", {"codec_pipeline.path": FUSED, "codec_pipeline.max_workers": 1}), + ("Fused (cpu_count)", {"codec_pipeline.path": FUSED, "codec_pipeline.max_workers": None}), +) + + +def time_call(fn: Callable[[], object], repeat: int = 3) -> float: + """Median wall-clock seconds for one call to `fn`. + + `timeit.Timer` supplies `perf_counter` and disables the cyclic garbage + collector during each run, so a collection triggered by earlier work cannot + land inside a measurement. `number=1` because a single call here already + moves 64 MiB -- the per-call overhead `timeit` amortizes is irrelevant at + this scale. + """ + return statistics.median(timeit.Timer(fn).repeat(repeat=repeat, number=1)) + + +def measure( + settings: dict[str, object], + store: Store, + data: np.ndarray, + compressors: object, +) -> tuple[float, float]: + """Time one full write and one full read of `data` under `settings`. + + The whole operation runs inside `zarr.config.set`, not just the array + construction. The pipeline class is resolved when the array is built, but + `codec_pipeline.max_workers` is read *per operation*, so a timed call made + outside the config block would silently use whatever worker count was + globally in effect -- which makes every configuration look identical. + """ + everything = slice(None) + + def write_once() -> None: + with zarr.config.set(settings): + array = zarr.create_array( + store=store, + shape=SHAPE, + chunks=CHUNKS, + shards=SHARDS, + dtype=DTYPE, + compressors=compressors, + fill_value=0, + overwrite=True, + ) + operator.setitem(array, everything, data) + + write = time_call(write_once) + + # The bytes on disk are identical whichever pipeline wrote them, so reading + # back what we just wrote isolates read performance on the same data. + def read_once() -> object: + with zarr.config.set(settings): + return zarr.open_array(store=store, mode="r")[everything] + + read = time_call(read_once) + + if not np.array_equal(read_once(), data): + raise AssertionError("round trip mismatch") + return write, read + + +def make_store(kind: str, tmp: Path) -> Store: + if kind == "memory": + return MemoryStore() + return LocalStore(tmp / f"demo_{kind}_{os.getpid()}.zarr") + + +def main() -> None: + n_cpu = os.cpu_count() or 1 + + # Each regime gets the data that actually exercises it. `arange` is + # trivially compressible, which is fine when nothing compresses it, but it + # would make gzip finish almost instantly and hide the CPU-bound behavior + # this example is about. The noisy array keeps gzip genuinely busy. + n = int(np.prod(SHAPE)) + plain_data = np.arange(n, dtype=DTYPE).reshape(SHAPE) + noisy_data = np.random.default_rng(0).integers(0, 2**24, size=SHAPE, dtype=DTYPE) + + n_shards = int(np.prod([s // c for s, c in zip(SHAPE, SHARDS, strict=True)])) + per_shard = int(np.prod([s // c for s, c in zip(SHARDS, CHUNKS, strict=True)])) + print(f"zarr {zarr.__version__} | {n_cpu} CPUs") + print( + f"array {SHAPE} {DTYPE} = {plain_data.nbytes / 2**20:.0f} MiB | " + f"{n_shards} shards x {per_shard} inner chunks = {n_shards * per_shard} chunks\n" + ) + + with tempfile.TemporaryDirectory() as tmp: + for store_kind in ("memory", "local"): + for codec_label, compressors, data in ( + ("uncompressed", None, plain_data), + ("gzip-6 (CPU-bound)", GZIP, noisy_data), + ): + print(f"=== {store_kind} store / {codec_label} ===") + print( + f"{'pipeline':<22}{'write (s)':>11}{'vs base':>10}" + f"{'read (s)':>12}{'vs base':>10}" + ) + results: dict[str, tuple[float, float]] = {} + for label, settings in CONFIGS: + store = make_store(store_kind, Path(tmp)) + results[label] = measure(settings, store, data, compressors) + + base_write, base_read = results[CONFIGS[0][0]] + for label, (write, read) in results.items(): + print( + f"{label:<22}{write:>10.3f}{base_write / write:>9.1f}x" + f"{read:>11.3f}{base_read / read:>9.1f}x" + ) + + # The headline comparison: does the thread pool earn its keep? + single_write, single_read = results["Fused (1 worker)"] + pool_write, pool_read = results["Fused (cpu_count)"] + print( + f" workers (cpu_count vs 1 worker): " + f"write {single_write / pool_write:.1f}x " + f"read {single_read / pool_read:.1f}x" + ) + print() + + print( + "Reading it:\n" + " * Uncompressed IO is scheduling-bound, so Fused (1 worker) is already\n" + " fastest -- a thread pool has nothing to parallelize and only adds\n" + " overhead.\n" + " * gzip is CPU-bound, so Fused (1 worker) can be *slower* than the\n" + " default, while Fused (cpu_count) spreads compression across cores\n" + " and reclaims the win. That flip is why the fused pipeline is\n" + " threaded by default, and why pinning max_workers=1 is worth it for\n" + " memory-backed uncompressed data.\n" + " * `codec_pipeline.max_workers` is read only by the FusedCodecPipeline;\n" + " the default BatchedCodecPipeline ignores it." + ) + + +if __name__ == "__main__": + main() diff --git a/examples/sharding_coalescing/README.md b/examples/sharding_coalescing/README.md new file mode 100644 index 0000000000..29ba08c9ce --- /dev/null +++ b/examples/sharding_coalescing/README.md @@ -0,0 +1,63 @@ +# Sharded Read Coalescing + +This example demonstrates byte-range coalescing for partial reads of sharded +arrays, a performance optimization added in Zarr-Python 3.3.0 and enabled by +default. + +A shard is one stored object containing many inner chunks, each occupying its own +byte range. Reading N inner chunks could mean N separate byte-range requests. +Because byte ranges are intervals, nearby ranges can be merged: `[a, b)` and +`[b, c)` together cover `[a, c)`, so a single request can serve both. Merging +trades reading some bytes you did not ask for against issuing fewer requests -- +worthwhile whenever a request is expensive, as with object storage. + +## What it shows + +- Reading scattered inner chunks from one shard with coalescing **off** issues + one store request per inner chunk; with the **default** settings the same read + collapses to a single request. +- The resulting wall-clock difference against a store with simulated latency. +- That coalescing changes only *how* data is fetched, never *what* is returned -- + the script asserts both configurations produce identical arrays. +- A case where coalescing changes nothing: a contiguous selection already has + adjacent byte ranges, so it merges under any setting. + +## Running + +```bash +uv run sharding_coalescing.py +``` + +## How the comparison is set up + +Two details make the effect observable, and both are worth understanding if you +adapt this script: + +- **The selection must have gaps.** A contiguous read produces adjacent byte + ranges that merge regardless of configuration. The strided selections skip + inner chunks, creating the gaps that the `sharding_coalesce_max_gap_bytes` + budget decides whether to bridge. +- **Latency must be charged per merged fetch.** The example defines a small + `WrapperStore` subclass that sleeps in `get`. It deliberately does *not* keep + `WrapperStore.get_ranges`, which forwards straight to the wrapped store and + would bypass the latency entirely; inheriting the `Store` ABC's `get_ranges` + instead runs the coalescer over its own `get`, so each merged fetch pays once. + + `zarr.testing.store` ships a ready-made `LatencyStore`, but importing it pulls + in `pytest`. Defining the wrapper inline keeps the example runnable with only + `zarr` and `numpy` installed. + +## Configuration + +Two settings control the behavior, both settable globally via `zarr.config` or +per array via `config=` on `zarr.create_array` / `Array.with_config`: + +| Setting | Default | Meaning | +| --- | --- | --- | +| `sharding_coalesce_max_gap_bytes` | 1 MiB | Merge two ranges only if the gap between them is no larger than this | +| `sharding_coalesce_max_bytes` | 16 MiB | Never let a merged read exceed this size | + +Setting the gap to `0` merges only exactly-adjacent ranges, which approximates +the pre-3.3.0 behavior; that is how the example emulates the old path. Raising +the gap reads more unwanted bytes in exchange for fewer round trips -- the right +value depends on how expensive a request is against how fast your link is. diff --git a/examples/sharding_coalescing/sharding_coalescing.py b/examples/sharding_coalescing/sharding_coalescing.py new file mode 100644 index 0000000000..5da62ed806 --- /dev/null +++ b/examples/sharding_coalescing/sharding_coalescing.py @@ -0,0 +1,226 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = [ +# "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main", +# "numpy", +# ] +# /// + +""" +Demonstrate byte-range coalescing for partial reads of sharded arrays. + +A shard is a single stored object holding many inner chunks, each occupying +its own byte range. Reading N inner chunks could mean issuing N separate +byte-range requests to the store. Because byte ranges are intervals, a reader +can instead merge nearby ranges: `[a, b)` and `[b, c)` together cover +`[a, c)`, so one request can serve both. Merging trades reading some bytes you +did not ask for against issuing fewer requests -- a good trade whenever a +request is expensive, which is the normal case for object storage. + +Zarr-Python 3.3.0 does this automatically. Two settings control it: + + * `sharding_coalesce_max_gap_bytes` (default 1 MiB) -- merge two ranges only + if the gap between them is no larger than this. + * `sharding_coalesce_max_bytes` (default 16 MiB) -- never let a merged read + exceed this size. + +Setting the gap to 0 disables merging of non-adjacent ranges, which +approximates the pre-3.3.0 behavior. This script compares the two, counting +store requests and measuring wall-clock time against a store with simulated +latency. + +Run it with: + + uv run sharding_coalescing.py +""" + +from __future__ import annotations + +import asyncio +import operator +import statistics +import timeit +from contextlib import contextmanager +from functools import partial +from typing import TYPE_CHECKING + +import numpy as np + +import zarr +import zarr.core._coalesce as coalesce_module +from zarr.abc.store import ByteRequest, RangeByteRequest, Store +from zarr.storage import MemoryStore, WrapperStore + +if TYPE_CHECKING: + from collections.abc import Callable, Iterator, Sequence + + from zarr.core.buffer import Buffer, BufferPrototype + +# Emulates the pre-3.3.0 behavior: with a zero gap budget, only ranges that are +# exactly adjacent get merged, so scattered inner chunks are fetched one by one. +NO_COALESCING = {"sharding_coalesce_max_gap_bytes": 0} + +# The shipped defaults. Spelled out here so the comparison is explicit rather +# than relying on whatever the global config happens to be. +DEFAULT_COALESCING = { + "sharding_coalesce_max_gap_bytes": 1 << 20, # 1 MiB + "sharding_coalesce_max_bytes": 16 << 20, # 16 MiB +} + +GET_LATENCY_S = 0.005 # 5 ms per request, a modest stand-in for object storage + + +class PerRequestLatencyStore(WrapperStore[Store]): + """Wraps a store, charging a fixed latency per byte-range fetch. + + `zarr.testing.store` ships a `LatencyStore`, but importing it pulls in + `pytest`; defining the wrapper here keeps this example runnable with only + zarr and numpy installed. + + Two details matter for the measurement: + + * The latency is applied in `get`, which is what an individual fetch costs. + * `get_ranges` is explicitly *not* overridden to forward to the wrapped + store. `WrapperStore.get_ranges` does forward, which would skip this + class's `get` entirely and make every configuration look identical. + Inheriting the `Store` ABC's implementation instead runs the coalescer + over `self.get`, so each *merged* fetch pays the latency once -- which is + exactly the cost coalescing exists to reduce. + """ + + get_ranges = Store.get_ranges + + def __init__(self, store: Store, *, get_latency: float) -> None: + super().__init__(store) + self.get_latency = get_latency + + def _with_store(self, store: Store) -> PerRequestLatencyStore: + # `WrapperStore` rebuilds the wrapper when opening read-only, so the + # latency setting has to be carried across. + return type(self)(store, get_latency=self.get_latency) + + async def get( + self, + key: str, + prototype: BufferPrototype, + byte_range: ByteRequest | None = None, + ) -> Buffer | None: + await asyncio.sleep(self.get_latency) + return await self._store.get(key, prototype, byte_range) + + +@contextmanager +def counting_requests() -> Iterator[Callable[[], int]]: + """Count the store fetches issued inside the block. + + Wraps the coalescing planner rather than the store: every merged group it + returns, plus every range it declined to merge, becomes exactly one fetch. + Counting here rather than at the store means the number reported is the + planner's decision, which is precisely what the settings control. + """ + original = coalesce_module.coalesce_ranges + total = 0 + + def counting_coalesce_ranges( + byte_ranges: Sequence[ByteRequest | None], + *, + max_gap_bytes: int, + max_coalesced_bytes: int, + ) -> tuple[ + list[list[tuple[int, RangeByteRequest]]], + list[tuple[int, ByteRequest | None]], + ]: + nonlocal total + groups, uncoalescable = original( + byte_ranges, + max_gap_bytes=max_gap_bytes, + max_coalesced_bytes=max_coalesced_bytes, + ) + total += len(groups) + len(uncoalescable) + return groups, uncoalescable + + coalesce_module.coalesce_ranges = counting_coalesce_ranges + try: + yield lambda: total + finally: + coalesce_module.coalesce_ranges = original + + +def measure_read(array: zarr.Array, selection: slice) -> tuple[int, float]: + """Return (store fetches, median seconds) for reading `selection`.""" + read = partial(operator.getitem, array, selection) + + with counting_requests() as fetches: + result = read() + requests = fetches() + + # `timeit.Timer` supplies the loop, `perf_counter`, and GC handling. The + # median of several runs keeps one unlucky run from dominating. + elapsed = statistics.median(timeit.Timer(read).repeat(repeat=5, number=1)) + + assert result.size > 0 # a read that returned nothing would time as "fast" + return requests, elapsed + + +def main() -> None: + n = 8192 + chunk = 64 + inner_chunks = n // chunk + + base = MemoryStore() + source = (np.arange(n, dtype="uint64") % 251).astype("uint8") + + # One shard holding every inner chunk, uncompressed so inner-chunk byte + # offsets stay predictable and the demonstration is easy to reason about. + writable = zarr.create_array( + store=base, shape=(n,), chunks=(chunk,), shards=(n,), dtype="uint8", compressors=None + ) + writable[:] = source + + store = PerRequestLatencyStore(base, get_latency=GET_LATENCY_S) + + print(f"zarr {zarr.__version__}") + print(f"array: {n} uint8 values, {inner_chunks} inner chunks of {chunk} in a single shard") + print(f"store: MemoryStore wrapped with {GET_LATENCY_S * 1000:.0f} ms of latency per request\n") + + # A strided selection touches inner chunks with unread chunks in between, + # so there are real gaps for the coalescer to bridge. A contiguous + # selection would merge under any setting, since its ranges are adjacent. + selections = { + "every 2nd inner chunk": slice(None, None, chunk * 2), + "every 4th inner chunk": slice(None, None, chunk * 4), + "contiguous quarter": slice(0, n // 4), + } + + header = f"{'selection':<24} {'coalescing':<12} {'requests':>9} {'time':>10}" + print(header) + print("-" * len(header)) + + for label, selection in selections.items(): + results = {} + for mode, config in (("off", NO_COALESCING), ("default", DEFAULT_COALESCING)): + array = zarr.open_array(store=store, mode="r").with_config(config) + requests, elapsed = measure_read(array, selection) + results[mode] = (requests, elapsed) + print(f"{label:<24} {mode:<12} {requests:>9} {elapsed * 1000:>9.1f}ms") + + off_requests, off_time = results["off"] + on_requests, on_time = results["default"] + if on_requests < off_requests: + print( + f"{'':<24} {'->':<12} " + f"{off_requests // on_requests:>8}x fewer {off_time / on_time:>9.1f}x faster" + ) + else: + print(f"{'':<24} {'->':<12} {'no change (ranges already adjacent)':>30}") + print() + + # Correctness is the point: coalescing must not change what you read back. + for mode, config in (("off", NO_COALESCING), ("default", DEFAULT_COALESCING)): + array = zarr.open_array(store=store, mode="r").with_config(config) + assert np.array_equal(array[::128], source[::128]), mode + print("Both configurations return identical data; coalescing only changes how it is fetched.") + + +if __name__ == "__main__": + main() diff --git a/mkdocs.yml b/mkdocs.yml index 87aaf23430..b414c73196 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,6 +31,8 @@ nav: - Examples: - user-guide/examples/custom_dtype.md - user-guide/examples/rectilinear_chunks.md + - user-guide/examples/codec_pipeline_performance.md + - user-guide/examples/sharding_coalescing.md - API Reference: - api/zarr/index.md - ' zarr.abc': @@ -95,6 +97,8 @@ nav: - 'zarr-metadata ↪': https://zarr-metadata.readthedocs.io/ - release-notes.md - contributing.md + - Blog: + - blog/index.md hooks: - mkdocs_hooks.py @@ -153,6 +157,14 @@ extra_css: plugins: - autorefs + - blog: + blog_dir: blog + post_dir: "{blog}/posts" + post_url_format: "{slug}" + # The blog is a simple reverse-chronological list of posts; the archive + # and category indexes add navigation we don't have the volume to justify. + archive: false + categories: false - search - markdown-exec - mkdocstrings: