Skip to content

fix: FusedCodecPipeline falls back to async path for sharded arrays with async-only inner codecs - #237

Open
d-v-b wants to merge 1 commit into
mainfrom
fix/fused-pipeline-async-inner-codecs
Open

fix: FusedCodecPipeline falls back to async path for sharded arrays with async-only inner codecs#237
d-v-b wants to merge 1 commit into
mainfrom
fix/fused-pipeline-async-inner-codecs

Conversation

@d-v-b

@d-v-b d-v-b commented Jul 22, 2026

Copy link
Copy Markdown
Owner

🤖 AI text below 🤖

Problem

Under the opt-in FusedCodecPipeline (zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})), a sharded array whose inner codec chain contains a third-party codec implementing only the async codec interface (_encode_single/_decode_single, no SupportsSyncCodec) raised on both write and read:

TypeError: All codecs must implement SupportsSyncCodec. The following do not: <name>

The pipeline's existing guard handles async-only codecs at the top level of the chain (evolve_from_array_spec catches the TypeError and falls back to the async path), but ShardingCodec structurally satisfies SupportsSyncCodec, so the pipeline committed to the sync fast path and crashed inside the sharding codec's inner ChunkTransform. The default BatchedCodecPipeline handles the same configuration correctly. Found in the 3.3.0 pre-release audit.

Fix

Sync capability is now a dynamic query, _codec_supports_sync (src/zarr/abc/codec.py): structural protocol membership plus an optional _sync_capable opt-out.

  • ShardingCodec._sync_capable reports False when its inner or index codec chain is not fully sync-capable (recursive, so nested sharding propagates).
  • ChunkTransform.__post_init__ consults the query, so construction raises for such chains and the pipeline's existing top-level guard sets sync_transform=None — declining the sync fast path and routing reads through the async partial shard decode (byte-range coalesced, matching BatchedCodecPipeline) and writes through the async fallback.

All-sync chains still build the sync transform and keep the fast path, byte-identical. BatchedCodecPipeline behavior is unchanged. No towncrier fragment: FusedCodecPipeline is new in unreleased 3.3.0.

Tests

  • New regression test: sharded array with an async-only inner codec round-trips (full write, partial write, full read, partial read) under FusedCodecPipeline, asserts the codec actually ran (invocation counters), and re-reads the store under the default pipeline to prove the bytes are valid cross-pipeline.
  • tests/test_fused_pipeline.py tests/test_pipeline_parity.py tests/test_fastpath_equivalence.py tests/test_codecs/test_sharding.py tests/test_codec_pipeline.py: 504 passed, 6 skipped (pre-existing skips).
  • Instrumented check: async-only inner chain → partial reads go through ShardingCodec._decode_partial_single (no whole-shard over-read); all-sync chain → read_sync fast path still fires.
  • mypy and ruff clean on all touched files.

🤖 Generated with Claude Code

…ith async-only inner codecs

ShardingCodec structurally satisfies SupportsSyncCodec, but its sync
methods delegate to the configured inner and index codec chains — so a
shard whose inner or index chain contains a codec implementing only the
async codec interface passed the fused pipeline's sync gate and then
raised TypeError mid-IO in ChunkTransform construction.

Sync capability is now answered by _codec_supports_sync, which combines
the structural protocol check with a per-instance _sync_capable opt-out
(absent means capable). ShardingCodec reports False when any codec in
its inner or index chain is not sync-capable (recursively, so a nested
shard propagates its opt-out outward), which makes ChunkTransform
construction raise at pipeline evolve time and the pipeline decline the
sync fast path — such arrays route through the async paths, matching
BatchedCodecPipeline. Fully sync-capable chains keep the fast path.

Closes zarr-developers#4178

Assisted-by: ClaudeCode:claude-fable-5
@d-v-b
d-v-b force-pushed the fix/fused-pipeline-async-inner-codecs branch from fb15e2e to 8c0162d Compare July 29, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant