Conversation
kylebarron
approved these changes
Jul 28, 2026
|
|
||
| ### How sharding works | ||
|
|
||
| Chunks encoded with the `sharding_indexed` codec contain subchunks, or "inner chunks". For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. |
There was a problem hiding this comment.
Suggested change
| Chunks encoded with the `sharding_indexed` codec contain subchunks, or "inner chunks". For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. | |
| Chunks encoded with the `sharding_indexed` codec contain a secondary level of chunking, called subchunks. For example, if the `chunk_grid` field of the array metadata declares an "outer chunk" size of, say `(10, 10)`, a `sharding_indexed` codec in the `codecs` field could declare an "inner chunk" size of `(5, 5)`. Readers accessing such a chunk will observe a stored object (a stream of bytes) that decodes to an array with size `(10, 10)` (the "outer chunk"), which is comprised of four separate, contiguous byte ranges that each decode to a `(5, 5)` inner chunk. Each inner chunk occupies its own byte range in the outer chunk. |
Comment on lines
+154
to
+155
| "sharding_coalesce_max_gap_bytes": 4 << 20, # 4 MiB | ||
| "sharding_coalesce_max_bytes": 64 << 20, # 64 MiB |
There was a problem hiding this comment.
Nit: I think << is complex for most people. In my examples I usually prefer 4 * 1024 ** 3. I think bytes -> KB -> MB is easier to follow
|
|
||
| The work released in this PR is just the first step of a larger performance-oriented direction for Zarr-Python. Landing these two enhancements taught us a *lot* about the performance-sensitive areas of the library. We can and will invest more time in performance tuning, e.g. by adding or changing abstractions, writing code for special cases, etc. | ||
|
|
||
| But we should also look to wrapping lower-level libraries that simply have an easier path to performance by virtue of being written in a compiled language. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned! |
There was a problem hiding this comment.
Suggested change
| But we should also look to wrapping lower-level libraries that simply have an easier path to performance by virtue of being written in a compiled language. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned! | |
| We plan to consider including compiled code that should enable significant performance improvements. The [`zarrs`](https://zarrs.dev/) project is an ecosystem of Zarr tools written in Rust, with [extremely high performance](https://book.zarrs.dev/#-zarrs-is-fast-). Is there a `zarrs` binding in Zarr-Python's future? I hope so! We are keenly observing development of [`zarrista`](https://developmentseed.org/zarrista/latest/) as a proof-of-concept for what a Python-`zarrs` binding layer might look like. Stay tuned! |
d-v-b
added a commit
that referenced
this pull request
Jul 28, 2026
Two suggestions from the review on #246 were not yet applied: - Replace the `4 << 20` bit-shift literals with `4 * 1024**2`, which reads more clearly as bytes -> KiB -> MiB. The suggestion wrote `1024 ** 3`, but these values are MiB, so the exponent is 2. - Reword the closing paragraph to state the compiled-code plan directly instead of framing it as something we "should look to". The rest of the review (subchunks phrasing, dropping the `N`/`M` notation, the "first step of a larger direction" opener, and the author description) is already in the branch. Assisted-by: ClaudeCode:claude-opus-5
…zarr-developers#4205) - Codec construction warnings (e.g. sharding's "disables partial reads") fired twice per array open, and on every decode/encode through the fused pipeline's async fallback. Re-constructions of an already-validated codec chain now go through codecs_from_list_unchecked, which validates structure without repeating first-construction advisory warnings; each warning fires exactly once per open under both pipelines. - concurrent_iter returned a lazy generator while its docstring promised eagerly scheduled tasks; it now materializes the task list so awaiting one at a time cannot serialize the batch. - A garbage codec_pipeline.max_workers value (e.g. from the environment) raised ValueError mid-read; it now warns and falls back to the default, consistent with tolerant handling of config input. - The as-completed pipeline helpers abandoned in-flight tasks when one failed, leaving stray background writes and "Task exception was never retrieved" warnings; failures now cancel and drain outstanding tasks. - Benchmarks: seed the data generator for reproducibility; fix a copy-pasted docstring. - Remove dead commented-out test blocks referencing the removed set_range API. Assisted-by: ClaudeCode:claude-sonnet-5
…th 11 updates (zarr-developers#4216) * chore(deps): bump the python-dependencies group across 1 directory with 11 updates Bumps the python-dependencies group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [numpy](https://github.com/numpy/numpy) | `2.5.0` | `2.5.1` | | [typer](https://github.com/fastapi/typer) | `0.26.8` | `0.27.0` | | [coverage](https://github.com/coveragepy/coveragepy) | `7.14.3` | `7.15.2` | | [hypothesis](https://github.com/HypothesisWorks/hypothesis) | `6.155.7` | `6.160.0` | | [tomlkit](https://github.com/python-poetry/tomlkit) | `0.15.0` | `0.15.1` | | [uv](https://github.com/astral-sh/uv) | `0.11.26` | `0.11.31` | | [mkdocs-material[imaging]](https://github.com/squidfunk/mkdocs-material) | `9.7.6` | `9.7.7` | | [mkdocstrings](https://github.com/mkdocstrings/mkdocstrings) | `1.0.4` | `1.0.6` | | [markdown-exec[ansi]](https://github.com/pawamoy/markdown-exec) | `1.12.1` | `1.12.3` | | [ruff](https://github.com/astral-sh/ruff) | `0.15.20` | `0.15.22` | | [mypy](https://github.com/python/mypy) | `2.1.0` | `2.3.0` | Updates `numpy` from 2.5.0 to 2.5.1 - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.5.0...v2.5.1) Updates `typer` from 0.26.8 to 0.27.0 - [Release notes](https://github.com/fastapi/typer/releases) - [Changelog](https://github.com/fastapi/typer/blob/master/docs/release-notes.md) - [Commits](fastapi/typer@0.26.8...0.27.0) Updates `coverage` from 7.14.3 to 7.15.2 - [Release notes](https://github.com/coveragepy/coveragepy/releases) - [Changelog](https://github.com/coveragepy/coveragepy/blob/main/CHANGES.rst) - [Commits](coveragepy/coveragepy@7.14.3...7.15.2) Updates `hypothesis` from 6.155.7 to 6.160.0 - [Release notes](https://github.com/HypothesisWorks/hypothesis/releases) - [Commits](HypothesisWorks/hypothesis@v6.155.7...v6.160.0) Updates `tomlkit` from 0.15.0 to 0.15.1 - [Release notes](https://github.com/python-poetry/tomlkit/releases) - [Changelog](https://github.com/python-poetry/tomlkit/blob/master/CHANGELOG.md) - [Commits](python-poetry/tomlkit@0.15.0...0.15.1) Updates `uv` from 0.11.26 to 0.11.31 - [Release notes](https://github.com/astral-sh/uv/releases) - [Changelog](https://github.com/astral-sh/uv/blob/0.11.31/CHANGELOG.md) - [Commits](astral-sh/uv@0.11.26...0.11.31) Updates `mkdocs-material[imaging]` from 9.7.6 to 9.7.7 - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](squidfunk/mkdocs-material@9.7.6...9.7.7) Updates `mkdocstrings` from 1.0.4 to 1.0.6 - [Release notes](https://github.com/mkdocstrings/mkdocstrings/releases) - [Changelog](https://github.com/mkdocstrings/mkdocstrings/blob/main/CHANGELOG.md) - [Commits](mkdocstrings/mkdocstrings@1.0.4...1.0.6) Updates `markdown-exec[ansi]` from 1.12.1 to 1.12.3 - [Release notes](https://github.com/pawamoy/markdown-exec/releases) - [Changelog](https://github.com/pawamoy/markdown-exec/blob/main/CHANGELOG.md) - [Commits](pawamoy/markdown-exec@1.12.1...1.12.3) Updates `ruff` from 0.15.20 to 0.15.22 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/0.15.22/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.20...0.15.22) Updates `mypy` from 2.1.0 to 2.3.0 - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v2.1.0...v2.3.0) --- updated-dependencies: - dependency-name: coverage dependency-version: 7.15.2 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: hypothesis dependency-version: 6.160.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: markdown-exec[ansi] dependency-version: 1.12.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocs-material[imaging] dependency-version: 9.7.7 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mkdocstrings dependency-version: 1.0.6 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: mypy dependency-version: 2.3.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: numpy dependency-version: 2.5.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: ruff dependency-version: 0.15.22 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: tomlkit dependency-version: 0.15.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies - dependency-name: typer dependency-version: 0.27.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: python-dependencies - dependency-name: uv dependency-version: 0.11.31 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: python-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> * fix: satisfy numpy 2.5.1 type stubs in indexing selection normalization numpy 2.5.1 stubs infer np.asarray(<list>) as a float64 array, so mypy now rejects the untyped asarray calls in replace_lists and CoordinateIndexer. Pass dtype=np.intp where the integer dtype is guaranteed, and cast to ArrayOfIntOrBool where the list contents are only known at runtime. Assisted-by: ClaudeCode:claude-fable-5 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
…elopers#4219) * fix: reject malformed chunk keys in DefaultChunkKeyEncoding decode_chunk_key stripped a single leading character and split the rest, so any key at all decoded to something. "0/1" silently became (1,) -- the "0" was eaten as if it were the "c" prefix -- and a key written with one separator decoded wrongly under an encoding configured with the other. Validate the "c<separator>" prefix and raise ValueError when it is absent, so a key that is not a chunk key for this encoding is reported rather than silently misread. Adds the tests this method never had, covering the round trip for both separators and each way a key can fail to carry the prefix. Assisted-by: ClaudeCode:claude-fable-5 * Rename 250.bugfix.md to 4219.bugfix.md
…rs forward it (zarr-developers#4206) The fused write gate checked only SupportsSetSync, but write_sync also needs get_sync (partial-chunk read-modify-write) and delete_sync (all-fill chunk cleanup): a set-sync-only store passed the gate, wrote some chunks, then died mid-batch with TypeError. And WrapperStore forwarded no *_sync method, so every wrapped store (e.g. LatencyStore) silently lost the sync fast path — latency benchmarks measured the async fallback while claiming to measure the fused sync path. Both gates now consult _store_supports_sync_io: structural membership in SupportsSyncStore (the full get/set/delete sync surface) combined with a per-instance _supports_sync_io opt-out (absent means capable). This is a private, interim convention pending a formal sync/async store architecture — the store-side twin of the codec-side _sync_capable convention from zarr-developers#4179 — deliberately not new public API. WrapperStore delegates the three sync methods and forwards the wrapped store's capability, so wrapping a sync store keeps the fast path and wrapping an async-only store falls back cleanly; LoggingStore logs the delegated sync calls. LatencyStore fixes: sync reads/writes now sleep the configured latency on the worker thread; get_ranges/get_partial_values route through the latency-injecting get instead of bypassing the wrapper; _with_store passes the raw (loc, scale) latency config instead of a single sampled float, so derived stores keep the distribution. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
dev-blog
branch
2 times, most recently
from
July 30, 2026 16:11
6bd5449 to
3dac9b3
Compare
Adds a development blog to the documentation site with a 3.3.0 release post covering the FusedCodecPipeline and sharded partial-read coalescing, plus two runnable examples referenced by the post (examples/codec_pipeline_performance, examples/sharding_coalescing). Compiles every pending changelog fragment into a single 3.3.0 release notes section dated 2026-07-30, merging it with the section compiled earlier in zarr-developers#4148, and empties changes/ — making this commit suitable to tag as v3.3.0. Assisted-by: ClaudeCode:claude-fable-5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
adds a blog