Fix/improve coverage

.github/workflows/ci.yml

@@ -89,9 +89,9 @@ jobs:
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: data/
-          key: "${{ runner.os }}-data-${{ hashFiles('tests/conftest.py') }}"
+          key: "${{ runner.os }}-data-${{ hashFiles('src/ngio/utils/_datasets.py') }}"
           restore-keys: |
-            "${{ runner.os }}-data-"
+            ${{ runner.os }}-data-
 
       - name: Restore/save shared data cache (trusted)
         if: github.event_name != 'pull_request'
@@ -102,12 +102,12 @@ jobs:
         uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: data/
-          key: "${{ runner.os }}-data-${{ hashFiles('tests/conftest.py') }}"
+          key: "${{ runner.os }}-data-${{ hashFiles('src/ngio/utils/_datasets.py') }}"
           restore-keys: |
-            "${{ runner.os }}-data-"
+            ${{ runner.os }}-data-
 
       - name: 🧪 Run Tests
-        run: pixi run -e ${{ matrix.environment }} pytest ${{ matrix.platform != 'ubuntu-latest' && '--no-cov' || '' }}
+        run: pixi run -e ${{ matrix.environment }} pytest -n 4 --durations=10 ${{ matrix.platform == 'ubuntu-latest' && matrix.environment == 'test11' && '--cov=ngio --cov-report=xml' || '' }}
 
       # If something goes wrong with scheduled tests, open an issue in the repo
       - name: 📝 Report Failures
@@ -124,7 +124,7 @@ jobs:
           update_existing: true
 
       - name: Coverage
-        if: success() && matrix.platform == 'ubuntu-latest'
+        if: success() && matrix.platform == 'ubuntu-latest' && matrix.environment == 'test11'
         uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
         with:
           token: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/ci_upstream.yml

@@ -56,12 +56,12 @@ jobs:
         uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: data/
-          key: "${{ runner.os }}-data-${{ hashFiles('tests/conftest.py') }}"
+          key: "${{ runner.os }}-data-${{ hashFiles('src/ngio/utils/_datasets.py') }}"
           restore-keys: |
-            "${{ runner.os }}-data-"
+            ${{ runner.os }}-data-
 
       - name: 🧪 Run Tests
-        run: pytest --no-cov
+        run: pytest -n 4 --durations=10
 
       - name: 📝 Report Failures
         if: failure() && github.event_name == 'schedule'
@@ -109,12 +109,12 @@ jobs:
         uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: data/
-          key: "${{ runner.os }}-data-${{ hashFiles('tests/conftest.py') }}"
+          key: "${{ runner.os }}-data-${{ hashFiles('src/ngio/utils/_datasets.py') }}"
           restore-keys: |
-            "${{ runner.os }}-data-"
+            ${{ runner.os }}-data-
 
       - name: 🧪 Run Tests
-        run: pytest --no-cov
+        run: pytest -n 4 --durations=10
 
       - name: 📝 Report Failures
         if: failure()

CHANGELOG.md

@@ -2,6 +2,39 @@
 
 ## [Unreleased]
 
+### Fix
+- Fix the deprecated `yx_scaling_factor` shim assembling scaling factors in `(z, x, y)` order instead of the canonical `(z, y, x)`: an asymmetric tuple like `yx_scaling_factor=(2, 4)` silently produced swapped per-level y/x scales. The shim also now trims leading factors for shapes with fewer than 3 axes (a 2D image previously raised a length-mismatch error).
+- Fix `concatenate_image_tables` (and variants) index construction: the concatenated table's index name was never set due to an inverted `None` check, and `mode="lazy"` built the index from the extras columns only — producing duplicated index values — while `mode="eager"` also hashed the original table index. Both modes now produce the same unique per-row index. The async variant also forwards `mode` to the per-image prefetch, so lazy prefetching no longer eagerly loads dataframes.
+- Fix `Roi.union`/`Roi.intersection` dropping ROI name `""` and label `0` (the joins used truthiness instead of `None` checks; label `0` is a legal label). The joined ROI's slices now also keep a deterministic axis order (the receiver's axes first, then axes only present in the other ROI) instead of an arbitrary set order.
+- `Roi.from_values` now validates its inputs (it used `model_construct`, which skips pydantic validation entirely), so invalid values such as a negative `label` or fewer than 2 axes raise instead of being silently accepted.
+- Fix `PixelSize.__eq__` raising `TypeError` when compared with a non-`PixelSize` object (it now returns `NotImplemented`, so `ps == "x"` is `False` and `ps in [...]` works) and make the `time_unit` comparison symmetric: previously `a == b` and `b == a` could disagree when only one side had a `time_unit`. **Behavior change**: pixel sizes with different `time_unit`s now always compare unequal.
+- Fix `NgioWellMeta.add_image`/`remove_image` and `NgioPlateMeta.add_well`/`add_acquisition`/`remove_well` mutating the receiver in place: the "updated copy" they returned shared its lists with the original object, so the original was silently modified too. All five now leave the receiver untouched.
+- Fix `AxesSetup.from_ordered_list` silently dropping a non-canonical axis name when a canonical name appeared to its left (e.g. `["z", "custom", "y", "x"]`): the canonical name reclaimed the slot already assigned to the custom axis. Canonical names now always claim their own slot and non-canonical names fill the remaining slots right-aligned.
+- Fix `ngio.experimental.iterators` grid helpers: `grid()` gave every generated ROI the same name (they now get unique per-tile names, e.g. `t0_z0_y32_x64`, prefixed with `base_name` when given), and `by_chunks` with an overlap equal to or larger than the chunk size now raises a clear `NgioValueError` instead of crashing with `range() arg 3 must not be zero`.
+- The global config singleton is now loaded lazily on first `get_config()` call instead of at import time, so setting `NGIO_CONFIG_PATH` after importing ngio (but before first use) is honored.
+- Fix the v0.5 metadata decoder passing a non-string axis `unit` through unchanged: the stringification computed for non-string JSON values was discarded (`unit=v05_axis.unit` instead of the normalized value).
+- Fix accessing labels on a read-only image without a `labels` group: `labels_container` (and everything routed through it) raised `NgioValueError: Cannot create a group in read only mode` instead of degrading gracefully like the tables path (`list_labels()` → `[]`, `labels_container` → `NgioValidationError`).
+- Fix an empty ROI table with no backend being unusable: `RoiTable().rois()` / `.add(roi)` raised `NgioValueError` instead of treating the table as empty (the `set_table_data` path already handled this case).
+- Fix duplicate ROI names not surviving a write/read roundtrip: the dedup in `RoiDictWrapper` renamed only the internal dict key, so the serialized table still contained duplicate index values. The renamed ROI now carries the deduplicated name.
+- Fix `OmeZarrPlate.get_well` never returning the instance it cached (it stored one `OmeZarrWell` and returned a second, freshly built one; repeated calls now return the cached instance, matching `get_image`).
+- A negative index inside a slicing sequence (e.g. `get_array(y=[-1, 0])`) now raises `NgioValueError` instead of a bare `AssertionError` (which would also vanish under `python -O`).
+
+### Tests
+- Enable `test_derive_from_legacy_images` (`tests/unit/images/test_create.py`), which was never collected due to a missing `test_` prefix; it now runs for both NGFF versions with the level pairing and channel-axis handling corrected. Rename the misnamed `test_fail_derive_singleton` to `test_pyramid_clamps_singleton_dimensions`.
+- Deduplicate the 18-item test-image parametrization (six verbatim copies) into a shared `zarr_name` fixture in `tests/conftest.py`.
+- Register a `network` pytest marker and apply it to the two tests hitting raw.githubusercontent.com, so `-m "not network"` runs offline. The Zenodo dataset downloads moved from conftest import time into session fixtures — test collection no longer blocks on the network.
+- Remove `--cov` from the default pytest `addopts` (local runs no longer pay ~40% coverage overhead); CI passes `--cov=ngio --cov-report=xml` explicitly on the codecov leg and `ci_upstream.yml` drops the now-unneeded `--no-cov`.
+- Merge the four per-backend round-trip tests in `test_backends.py` into one parametrized test; drop the redundant middle-layer copy of the #207 backend-preservation test (still covered at the `write_table` and container layers).
+- Anchor test-data paths to `__file__` instead of the current working directory (`tests/conftest.py`, `tests/create_test_data.py`, `test_unit_v04_utils.py`), so pytest can run from any directory. Reduce `test_multiprocessing_safety` from 1000 to 100 tasks and delete an unreferenced meta JSON fixture.
+- Raise overall coverage from 91% to 95%: new unit tests for the v0.5 metadata codec (`test_unit_v05_utils.py`, mirroring the v0.4 file), container/image error paths and deprecated shims, ROI-table and plate edge cases, `NgioCache`, the `FeatureExtractorIterator` dask path, and io_pipes error branches. `hcs/_plate.py`, `images/_image.py`, `utils/_cache.py`, `experimental/iterators/_feature.py`, `io_pipes/_match_shape.py`, `_zoom_transform.py`, and `_ops_slices.py` are now at 100%.
+- Speed up fixtures: the `moto_s3_server` fixture is now session-scoped (tests isolate via `random_zarr_path()`, and starting the server cost ~4s per S3 test); read-only consumers of `images_all_versions` and the Zenodo cardiomyocyte datasets now share session-scoped copies (`images_all_versions_readonly`, `cardiomyocyte_tiny_path_readonly`, `cardiomyocyte_small_mip_path_readonly`) instead of re-copying up to 126 MB per test, while mutating tests keep fresh per-test copies; `test_real_mask` copies only the single image it writes to; `test_table_ops.py` builds its two sample containers once per module.
+
+### Chores
+- Speed up CI: run the test suite with `pytest-xdist` (`-n 4`, new `test` extra dependency); collect coverage and upload to codecov on the ubuntu/`test11` leg only (one report per run, unchanged content); add `--durations=10` to CI runs for runtime observability. The Zenodo download in `tests/conftest.py` is now guarded by a `FileLock` (xdist workers on a cold cache) and no longer re-extracts an already-unzipped dataset (`re_unzip=False`).
+- Shrink the store-matrix test payload (`tests/stores/utils.py`) from a `(3, 5, 64, 64)`/3-level image to `(3, 2, 32, 32)`/2 levels — same monitored behaviors (multi-channel z-stack pyramid, label, all four table backends, derive across the 4×4 store matrix), far fewer object round-trips through the mocked S3/HTTP stores (~75s → considerably less on CI).
+- Slim the iterator tests: the two mutating tests (`test_segmentation_iterator`, `test_masked_segmentation_iterator`) now copy only the single image they write to (new `single_image_copy` fixture, replacing per-test copies of all 18 test images; the unused `images_all_versions` fixture is removed) and run the full 9-axes matrix on v0.5 plus a v0.4 smoke subset (`yx`, `tczyx`) instead of both versions × 9 — the iterators are version-agnostic and the v0.4 write path stays monitored by the creation/store tests. The two read-only iterator tests keep the full 18-image matrix.
+- Fix the CI data-cache `restore-keys` fallback never matching: the fallback key was written inside a YAML block scalar with double quotes (`"Linux-data-"`), which are literal there, so a cache-key rotation caused every PR job to re-download the ~160 MB Zenodo datasets during the test step (~80s/job). The cache key now also hashes `src/ngio/utils/_datasets.py` (the dataset registry) instead of `tests/conftest.py`, so test-suite refactors no longer rotate the key (`ci.yml`, `ci_upstream.yml`).
+
 ### Feature
 - Add a configurable IO retry policy: `NgioConfig.io_retry` (`RetryConfig`) with `max_retries` (default `0`, never retry), a backoff strategy (`ConstantBackoff`, `LinearBackoff`, or `ExponentialBackoff`, each with `delay_s`, `max_delay_s`, `jitter`), and error matching via `retry_on` substrings (matched against `"ExceptionName: message"`) or the discouraged blanket `retry_all_errors` flag, which emits an `NgioUserWarning` and is mutually exclusive with `retry_on`. Ngio's own `NgioError`s are never retried. Retry helpers live in `ngio.utils._retry`; the public `ngio.utils.retry_io` decorator reads the global config at call time.
 - Add `ngio.utils.NgioStore`, a picklable zarr `WrapperStore` that applies the `io_retry` policy to every store IO call (the policy snapshot travels with the store into dask task graphs) and centralizes store-type dispatch behind uniform services (`store_type`, `full_url`, `sync_fs_and_path`, `get_mapper`, `local_root`, `memory_dict`, `list_dir_collected`, `NgioStore.from_any`/`ensure`). Every group ngio opens (`open_group_wrapper` / `ZarrGroupHandler`) is now backed by an `NgioStore`, so the `io_retry` policy applies to all zarr IO — metadata, pixel data, and lazy dask reads/writes executed on workers (the policy snapshot pickles with the store into the task graph). User-provided `zarr.Group`s are transparently reopened on a wrapped store; `ZipStore` is now an explicitly supported store type (it previously warned).