fix(fused codec pipeline): bulk shard decode must only engage for identity full reads - #4203
Merged
d-v-b merged 1 commit intoJul 29, 2026
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4203 +/- ##
=======================================
Coverage 93.90% 93.91%
=======================================
Files 91 91
Lines 12672 12680 +8
=======================================
+ Hits 11900 11908 +8
Misses 772 772
🚀 New features to boost your workflow:
|
…x density check The FusedCodecPipeline's vectorized whole-shard decode accepted any indexer without `sel_shape` whose output shape matched the shard shape. An OrthogonalIndexer from `arr[perm, :]` / `arr.oindex[...]` satisfies that, so reordering or duplicating fancy-index reads on uncompressed, crc-free sharded arrays silently returned the shard in natural order. The bulk path now requires an identity full read: one whole-dimension, step-1 slice per dimension (structural, not a BasicIndexer type check, since `arr[:]` arrives as an OrthogonalIndexer). Also: - decline structured dtypes in the bulk path, which lacks the Struct byte-order branch of BytesCodec._decode_sync (latent until zarr-developers#3054 is fixed); - `_ShardIndex.is_dense` now requires offsets to exactly tile the data section instead of merely being unique, so corrupt indexes with overlapping or out-of-range offsets (e.g. pointing into an index_location='start' index region) cannot be served as array data. Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
fix/bulk-shard-decode-identity-gate
branch
from
July 29, 2026 14:32
2287a0b to
42241ba
Compare
d-v-b
marked this pull request as ready for review
July 29, 2026 14:37
Contributor
Author
|
self-merging as this is scoped to the fused codec pipeline and it's a data corruption risk |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR ensures that we more correctly enter a bulk-decode fast path, avoiding bulk-decodes where inappropriate, which led to data corruption.
based on a claude-authored PR here: d-v-b#256
Summary
[Describe what this PR changes and why, in your own words.]
For reviewers
[What would you most value a second look at? What are you already confident in? For a refactor, say whether behavior is meant to be unchanged.]
Author attestation
TODO
docs/user-guide/*.mdchanges/