feat(sharding): manage inner chunks as a semi-regular grid (sharding_indexed v1.1)

[d-v-b]

🤖 AI text below 🤖

Implements zarr-specs #370
(sharding_indexed v1.1, proposed by @LDeakin, xref
zarr-extensions #34):
the inner chunk_shape no longer needs to evenly divide the shard shape.

What changed

The sharding codec now models its inner chunk layout as a semi-regular grid
managed by the existing ChunkGrid machinery: inner chunks are spaced at
chunk_shape intervals and clipped by the shard shape.

• Chunks per shard is the ceiling division of the shard shape by
  chunk_shape (previously floor division, guarded by a divisibility check in
  ShardingCodec.validate).
• Clipped chunks are stored at their clipped shape, per the spec ("inner
  chunks are clipped by the shard shape"), saving the bytes that fill-padding
  would cost. E.g. a shard of shape [12] with chunk_shape = [5] stores three
  inner chunks of shapes [5], [5], [2].
• All encode/decode paths (sync + async, full + partial) now use per-chunk
  ArraySpecs derived from the inner grid, instead of one uniform spec. The
  indexing layer already produced correct clipped selections and
  is_complete_chunk flags for semi-regular grids, so no indexing changes were
  needed.
• The uncompressed whole-shard bulk-decode fast path explicitly gates on an
  evenly divided grid (clipped shards have variable payload sizes) and falls
  back to the per-chunk path.
• The scalar-broadcast write memoization only reuses encoded bytes across inner
  chunks of the nominal shape, since clipped chunks encode to different bytes.
• Array.read_chunk_sizes, Array.cdata_shape, and nchunks_initialized now
  account for the inner grid restarting at every shard boundary (for
  non-divisible shapes the global inner grid is itself semi-regular, e.g. shard
  12 / chunks 5 over a 24-element array gives sizes (5, 5, 2, 5, 5, 2),
  not (5, 5, 5, 5, 4)).

Because the geometry is fully grid-derived, the generalization composes with
everything the codec already supports: rectilinear outer chunk grids (each
shard clips its own inner grid — the original motivation in the proposal) and
nested sharding (clipping composes through levels, e.g. 12 → 5 → 2).

Compatibility

• Metadata representation is unchanged; arrays with divisible chunk_shape
  produce byte-identical output to before.
• Arrays written with a non-divisible chunk_shape require a v1.1-aware
  implementation to read; v1.0 implementations MUST reject them (per the spec
  PR), and older zarr-python versions do reject them at validate time.

Review process

The second commit addresses findings from an extensive adversarial multi-agent
review (six independent review lenses over the diff, each finding then attacked
by a verifier that had to reproduce it):

• nchunks_initialized undercounted on clipped grids (floor division;
  could report 0 when chunk_shape exceeds the shard shape) — fixed with
  ceiling division plus a regression test.
• The scalar-broadcast write memo was dead — it gated on ArraySpec
  object identity, which never holds because _get_chunk_spec is uncached
  (sharding codec use of lru caching fails with numpy void scalars zarr-developers/zarr-python#3054); it now gates on shape equality, with a regression test that counts
  encodes.
• The "rectilinear" clipping test wasn't rectilinear — shard edges
  [[6, 4]] normalize to a regular grid; it now uses [[4, 6]] under the
  rectilinear config flag and asserts the metadata grid type.

Verified-refuted (no change needed): async/ZipStore paths and all four
subchunk_write_order variants behave correctly on clipped grids;
_unique_edge_lengths in chunk_grids.py is dead code but was dead before
this change (cleanup left for a separate PR).

Tests

• Parametrized roundtrip test over array/shard/chunk shape combinations
  (trailing clipped chunks, chunk larger than shard, array not divisible by
  shard, single fully-clipped shard, and an evenly divisible control), each with
  full and partial reads/writes and both index locations.
• A byte-layout test asserting the spec example: shard [12], chunks [5] →
  three index entries with payload lengths 5, 5, 2 and a 12-byte data section.
• Rectilinear outer grid, nested sharding, read_chunk_sizes/cdata_shape,
  nchunks_initialized, and scalar-write-memo tests.
• Full test suite passes under both BatchedCodecPipeline and
  FusedCodecPipeline.

🤖 Generated with Claude Code