fix: FusedCodecPipeline falls back to async path for sharded arrays with async-only inner codecs - #237
Open
d-v-b wants to merge 1 commit into
Open
fix: FusedCodecPipeline falls back to async path for sharded arrays with async-only inner codecs#237d-v-b wants to merge 1 commit into
d-v-b wants to merge 1 commit into
Conversation
7 tasks
…ith async-only inner codecs ShardingCodec structurally satisfies SupportsSyncCodec, but its sync methods delegate to the configured inner and index codec chains — so a shard whose inner or index chain contains a codec implementing only the async codec interface passed the fused pipeline's sync gate and then raised TypeError mid-IO in ChunkTransform construction. Sync capability is now answered by _codec_supports_sync, which combines the structural protocol check with a per-instance _sync_capable opt-out (absent means capable). ShardingCodec reports False when any codec in its inner or index chain is not sync-capable (recursively, so a nested shard propagates its opt-out outward), which makes ChunkTransform construction raise at pipeline evolve time and the pipeline decline the sync fast path — such arrays route through the async paths, matching BatchedCodecPipeline. Fully sync-capable chains keep the fast path. Closes zarr-developers#4178 Assisted-by: ClaudeCode:claude-fable-5
d-v-b
force-pushed
the
fix/fused-pipeline-async-inner-codecs
branch
from
July 29, 2026 13:17
fb15e2e to
8c0162d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Problem
Under the opt-in
FusedCodecPipeline(zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})), a sharded array whose inner codec chain contains a third-party codec implementing only the async codec interface (_encode_single/_decode_single, noSupportsSyncCodec) raised on both write and read:The pipeline's existing guard handles async-only codecs at the top level of the chain (
evolve_from_array_speccatches theTypeErrorand falls back to the async path), butShardingCodecstructurally satisfiesSupportsSyncCodec, so the pipeline committed to the sync fast path and crashed inside the sharding codec's innerChunkTransform. The defaultBatchedCodecPipelinehandles the same configuration correctly. Found in the 3.3.0 pre-release audit.Fix
Sync capability is now a dynamic query,
_codec_supports_sync(src/zarr/abc/codec.py): structural protocol membership plus an optional_sync_capableopt-out.ShardingCodec._sync_capablereportsFalsewhen its inner or index codec chain is not fully sync-capable (recursive, so nested sharding propagates).ChunkTransform.__post_init__consults the query, so construction raises for such chains and the pipeline's existing top-level guard setssync_transform=None— declining the sync fast path and routing reads through the async partial shard decode (byte-range coalesced, matchingBatchedCodecPipeline) and writes through the async fallback.All-sync chains still build the sync transform and keep the fast path, byte-identical.
BatchedCodecPipelinebehavior is unchanged. No towncrier fragment:FusedCodecPipelineis new in unreleased 3.3.0.Tests
FusedCodecPipeline, asserts the codec actually ran (invocation counters), and re-reads the store under the default pipeline to prove the bytes are valid cross-pipeline.tests/test_fused_pipeline.py tests/test_pipeline_parity.py tests/test_fastpath_equivalence.py tests/test_codecs/test_sharding.py tests/test_codec_pipeline.py: 504 passed, 6 skipped (pre-existing skips).ShardingCodec._decode_partial_single(no whole-shard over-read); all-sync chain →read_syncfast path still fires.🤖 Generated with Claude Code