fully separate sync and async internal APIs

[d-v-b]

current state

our Store API uses async methods for doing IO. In Python today, this is ergonomic for high-latency storage (stuff stored on a server far away), and unergonomic for low-latency storage (stuff stored in memory).

similarly, our Codec API uses async methods for encoding and decoding chunks. This is ergonomic for codecs that need to do IO (but only because we made the store api async), and unergonomic for... all other codecs, because they are compute-bound, which is a bad fit for python asyncio.

The FusedCodecPipeline work in #3885 demonstrated that avoiding asyncio overhead can make Zarr much more efficient. But the code changes required for realizing that performance improvement are janky: we added new sync logic on top of the existing obligatory-async Store / Codec implementations. Going forward this will be a big source of bugs and confusion, so IMO we need to quickly come up with an API evolution of the Store and Codec APIs that isn't janky. this issue is me outlining my current vision for that evolution. this is not final, and I am open to any feedback, but i probably won't find it easy to understand the POV that the current state is OK.

desired state

note: the Bikeshed suffix here conveys "this is not the final name we will use"

base classes / protocols

• SyncStoreBikeshed is a base class / protocol for stores that target storage backends which are most efficiently accessed via synchronous code.
• AsyncStoreBikeshed is a separate base class / protocol for stores that target storage backends whicher are most efficiently accessed via async.
• SyncAsyncStoreAdapter is a base class / protocol that defines how async store operations should be invoked in a sync context.
• SyncCodecBikeshed is a base class / protocol for codecs that most efficiently executed using sync code
• AsyncCodecBikeshed is a base class / protocol for codecs that most efficiently executed using async code
• SyncAsyncCodecAdapterBikeshed is a base class / protocol that defines how async codec operations should be invoked in a sync context.

The SyncAsycAdapters take configuration that gives the caller control over the event loop in which coroutines are run. this allows zarr-python to use its own event loop, but also another library to disregard that event loop if they want to plug in their own. We want to avoid locking downstream libraries in to running their coroutines on our event loop.

implementations

• MemoryStore is sync
• LocalStore is sync
• AsyncLocalStore is async and uses uvloop (if that's actually helpful, I don't know).
• fsspec and obstore-backed stores are async.
• Generic store wrappers like LatencyStore come in sync / async variants. That's a first draft idea. I would also like to explore other ways of composing functionality on top of stores, e.g. allowing users to provide per-method adapters instead of subclassing. speculative, tbd.
• All codecs are sync
• The sharding codec and any other codec that invokes store methods has an additional async form, which will need to handle inner codecs being sync and / or async
• The user-facing Array class is sync, and its job is to provide an array-like API layer over an ArrayEngine, which comes in two varieties: ArrayEngine and AsyncArrayEngine. An array engine implements all the stuff the array needs for reading / writing chunks and metadata. The array engine does not need to act like a numpy array -- that's a job for the Array class. The array engine is also a target for bindings to another language's zarr implementation, like zarrs via zarrista.

desired current state -> desired state transition

• we define the Store and Codec APIs in separate subpackages, zarr-codec and zarr-storage. we implement codecs and stores there. Unless we are very clever, zarr-codec depends on zarr-storage because the sharding codec does IO. zarr-codec is free to use whatever build system and dependencies are useful, so numcodecs can be fully replaced by zarr-codec. version 0.1.0 of zarr-codec and zarr-storage implement zarr-python's current APIs exactly, so zarr-python can depend on 0.1.0 of both without any changes besides replacing python code with re-exports.
• we define new codec / store APIs in 0.2.0 of both packages. zarr-python releases a new version with bridge compatibility for both 0.1.0 AND new 0.2.0 apis. we slowly deprecate the 0.1.0 APIs over a series of subsequent zarr-python releases. I think we can generalize this pattern to other APIs we want to evolve.
• zarr-python defines ArrayEngine and AsyncArrayEngine APIs and rebuilds the Array API on top of them. see [do not merge] zarrs bindings  #4064 for an example of this.
• zarr-python's metadata classes slim down. instead of containing codec implementations, that's delegated to the ArrayEngine / AsyncArrayEngine, and array metadata is instead a simple data class that does little more than normalize JSON. See refactor: use lower-level metadata models in zarr-python #4129 for a prototype of this.

This is a lot of effort but tbh I think the hardest part will be agreeing on the vision. I would like to formalize whatever direction we agree on in #4149. If we can all get behind something, I think the payoff is high, and I'm absolutely willing to put effort in to getting there.