diff --git a/.github/workflows/html-proofer.yml b/.github/workflows/html-proofer.yml new file mode 100644 index 00000000..8bedcf03 --- /dev/null +++ b/.github/workflows/html-proofer.yml @@ -0,0 +1,62 @@ +name: "Build and Lint Site" + +on: + workflow_dispatch: + pull_request: + branches: ["main"] + paths: + - "specifications/**" + - "rfc/**" + - "resources/**" + - "community/**" + - "contributing/**" + - "help-desk/**" + - "_html_extra/**" + - "_static/**" + - "_ext/**" + - "images/**" + - "**/*.md" + - "conf.py" + - "requirements.txt" + - "readthedocs.yml" + - ".github/workflows/build-lint.yml" + +concurrency: + group: build-lint-${{ github.ref }} + cancel-in-progress: true + +jobs: + build-site: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 1 + submodules: recursive + + # dependencies + - uses: ruby/setup-ruby@v1 + with: + ruby-version: "3.3" + - uses: actions/setup-python@v5 + with: + python-version: "3.13" + - uses: actions/setup-node@v4 + with: + node-version: "22" + + - name: Install dependencies + run: | + pip install -r requirements.txt + pip install specifications/dev + gem install html-proofer + + - name: Build the site + run: sphinx-build -b html . _build/html + + - name: Validate all links, enforce alt text + run: htmlproofer --ignore-files "/.*\/specifications\/(0\.[0-9]+|dev)\/.*/,/.*webpack-macros\.html/,/.*\/rfc\/4\/markdown\/.*/,/.*\/rfc\/5\/.*/,/.*\/rfc\/1\/.*/,/.*\/0\.[0-9]+(\.html|\/.*)/" --disable-external ./_build/html + + - name: Ensure no unexpected encoded HTML in output + run: | + ! fgrep -R 'lt;blockquote' _build/html \ No newline at end of file diff --git a/README.md b/README.md index 4821dff2..cf8f3600 100644 --- a/README.md +++ b/README.md @@ -14,16 +14,9 @@ See also [CONTRIBUTING.md](./contributing/index.md) Specifications have been moved to [ome/ngff-spec](https://github.com/ome/ngff-spec). -### Editing specifications - -Specifications are written in markdown, or technically -[bikeshed](https://github.com/tabatkins/bikeshed) -- a markdown document, with -special extensions understood by the bikeshed tool. The bikeshed tool is run -during the Sphinx build step (see conf.py). - # RFCs -Requests for comments (RFCs) are used to discuss and capture high-level decisions within the NGFF community. +Requests for comments (RFCs) are used to discuss and capture high-level decisions within the NGFF community. RFCs are contained under the `rfc` directory at the moment but may be moved out into a separate repo in the future. @@ -37,5 +30,3 @@ RFCs are contained under the `rfc` directory at the moment but may be moved out - - diff --git a/boilerplate/header.include b/boilerplate/header.include index 79b2fd56..7bd9e500 100644 --- a/boilerplate/header.include +++ b/boilerplate/header.include @@ -10,7 +10,7 @@
- OME logo (6 circles in a hexagon) + OME logo (6 circles in a hexagon)

[TITLE]

[LONGSTATUS], diff --git a/contributing/specification/index.md b/contributing/specification/index.md index ca262d33..7090703d 100644 --- a/contributing/specification/index.md +++ b/contributing/specification/index.md @@ -6,14 +6,14 @@ request for comment (RFC) documents. In the future we will flesh out this page with a guide to RFCs, but in the meantime the RFC process is outlined in the -[Implementation section of RFC 1](../rfc/1/index.md#implementation). +[Implementation section of RFC 1](rfc1-implementation). ## Comment on a Request For Comment (RFC) If you want to leave a suggestion or comment on an RFC that is under review, please leave a comment in a new page under the "comments/" directory for the relevant RFC. A template is also available for formatting your comment: -[template](../rfc/1/templates/review_template). +[template](rfc1-review-template). ## How to change the specification @@ -143,7 +143,7 @@ Other useful admonitions and directives (e.g., `hint`, `note`) can be found [her ## Building *only* the spec document -The spec document under the [ngff-spec](github.com/ome/ngff-spec) repository can be built as a standalone document to make writing and rendering a smoother experience. +The spec document under the [ngff-spec](https://github.com/ome/ngff-spec) repository can be built as a standalone document to make writing and rendering a smoother experience. To build the spec document, you first need to install the necessary dependencies: After cloning the ngff-spec repo, navigate into the repository on your machine and install the dependencies using pip: diff --git a/help-desk/index.md b/help-desk/index.md index cadbd2ac..c3567770 100644 --- a/help-desk/index.md +++ b/help-desk/index.md @@ -1,8 +1,8 @@ # Help Desk -* [Glossary](#glossary) -* [FAQs](#faqs) -* [Where to seek for help](#where-to-seek-for-help) +- [Glossary](#glossary) +- [FAQs](#faqs) +- [Where to seek for help](#where-to-seek-for-help) ## Glossary @@ -29,11 +29,23 @@ Jump to: [Back to top](#glossary) +### D + +[Back to top](#glossary) + +### E + +[Back to top](#glossary) + +### F + +[Back to top](#glossary) + ### G [Back to top](#glossary) -### H +### H **HCS** - Dedicated layout of OME-Zarr files representing the used wells and plates for High Content Screening (HCS). The layout allows a user to efficiently address and analyze specific data from specific wells. @@ -41,12 +53,24 @@ Jump to: [Back to top](#glossary) +### I + +[Back to top](#glossary) + ### J **JSON file** - A simple standardized human-readable data format for sharing data using nested key-value pairs. [Back to top](#glossary) +### K + +[Back to top](#glossary) + +### L + +[Back to top](#glossary) + ### M **Multiscale** - Multiscale means storing the same image at multiple resolutions (full size, half size, quarter size, etc.). This makes it easy for tools to zoom and navigate very large images quickly—just like how map apps load lower-resolution versions first, then sharpen the view as you zoom in. @@ -54,6 +78,7 @@ Jump to: [Back to top](#glossary) ### N + **NGFF** - Next Generation File Format (NGFF) is a community for solving problems related to producing open-source software for microscopy data. [Back to top](#glossary) @@ -66,7 +91,7 @@ Jump to: **OME-NGFF** - OME-NGFF refers to the resulting efforts of the OME & NGFF communities. -**OME-Zarr** - An OME-Zarr is a standardized file format for microscopy images. More details on the [main page](#../). +**OME-Zarr** - An OME-Zarr is a standardized file format for microscopy images. More details on the [main page](#main-page). [Back to top](#glossary) @@ -76,6 +101,10 @@ Jump to: [Back to top](#glossary) +### Q + +[Back to top](#glossary) + ### R **RFC** - Request For Comments (RFC) is a process for proposing a change to the standardized specification for OME-Zarr files. @@ -84,7 +113,7 @@ Jump to: ### S -**Shard/ing** - Like [Chunks](#c), sharding is a way of breaking up pixel data for more efficient access. Shards are groups of chunks with headers in front of them for describing the chunks inside. +**Shard/ing** - Like [Chunks](#c), sharding is a way of breaking up pixel data for more efficient access. Shards are groups of chunks in a single object with headers describing the chunks inside. [Back to top](#glossary) @@ -96,6 +125,9 @@ Jump to: ### U +[Back to top](#glossary) + +### V [Back to top](#glossary) @@ -105,9 +137,17 @@ Jump to: [Back to top](#glossary) +### X + +[Back to top](#glossary) + +### Y + +[Back to top](#glossary) + ### Z -**Zarr** -- A file format, see the [main page](#../) for more information about what a Zarr is and the benefits of using Zarr. +**Zarr** -- A file format, see the [main page](#main-page) for more information about what a Zarr is and the benefits of using Zarr. **Zarr Group** - A Zarr group is like a folder inside a dataset. It can hold arrays, metadata, or other groups, helping organize complex data into a clear hierarchy. Think of it as a way to keep all related images and information neatly together. @@ -172,15 +212,14 @@ That is often the case. OME-Zarr include multiresolution data (i.e. pyramids), w Not yet fully. Some workarounds are possible (1D arrays, mesh formats), but this is an active area of development. - - +(where-to-seek-for-help)= ## Where to look for help + If you have questions or need help with OME-Zarr, you can reach out through the following channels: -| Where | Description | When to use | +| Where | Description | When to use | |-------|-------------| --------------| -| [Image.sc Forum](https://forum.image.sc) | A community forum for image analysis and bioimaging. | To know when to use the tags [ngff](https://forum.image.sc/tag/ngff), [ome-ngff](https://forum.image.sc/tag/ome-ngff), and [ome-zarr](https://forum.image.sc/tag/ome-zarr) in the forum please read [Landing Page](https://ngff.openmicroscopy.org/index.html), Glossary and FAQs in this page. | +| [Image.sc Forum](https://forum.image.sc) | A community forum for image analysis and bioimaging. | To know when to use the tags [ngff](https://forum.image.sc/tag/ngff), [ome-ngff](https://forum.image.sc/tag/ome-ngff), and [ome-zarr](https://forum.image.sc/tag/ome-zarr) in the forum please read [Landing Page](https://ngff.openmicroscopy.org/index.html), Glossary and FAQs in this page. | | [ome/ngff GitHub Issues](https://github.com/ome/ngff/issues) | The official repository for OME-NGFF specifications and related discussions. | For reporting bugs, suggesting features, or discussing technical aspects of the OME-NGFF specifications. | | Office Hours | Regularly scheduled virtual meetings where you can ask questions and get help from the NGFF community | Office hours rotate between APAC/AU/EU- and AMER/AU/EU-friendly times. Join them when you want to chat about OME-Zarr. Read more in the [Community](../community/index.md) page. | | OME-Zarr libraries GitHub Issues| Various repositories for OME-Zarr libraries. | For issues specific to a particular OME-Zarr library, use the respective GitHub repository's issue. | | In person events | Conferences, workshops, and meetups where you can connect with the NGFF community. | To network, learn, and discuss OME-Zarr in person. Check the [Community](../community/index.md) page for upcoming events. | - diff --git a/index.md b/index.md index 2c700d20..c54d30c9 100644 --- a/index.md +++ b/index.md @@ -1,3 +1,4 @@ +(main-page)= # Next-Generation File Formats (NGFF) + OME-Zarr Welcome to the Next-Generation File Formats (NGFF) main page! This site is dedicated to providing resources for the NGFF community and those that are interested in getting started with OME-Zarr. @@ -15,33 +16,44 @@ In summary, OME-Zarr allows you to store, share and stream large images. You can An OME-Zarr is a file format optimized for storing, viewing, & sharing large images. There are two parts to an OME-Zarr: -* **The "Zarr" part describes how the pixel data for the images are laid out** -* The "OME", which stands for [Open Microscopy Environment](https://www.openmicroscopy.org/), part describes metadata about the pixel data. This includes metadata such as: - * spatial relationships - * high content screening data - * well data - * [and more](./specifications/index)! +- **The "Zarr" part describes how the pixel data for the images are laid out**. [Zarr](https://zarr.dev) is a next-generation data format used for scientific datasets in multiple domains. +- **The "OME" part describes metadata about the pixel data.** OME stands for [Open Microscopy Environment](https://www.openmicroscopy.org/). This includes metadata such as: + - spatial relationships + - high content screening data + - well data + - [and more](./specifications/index)! ## Why would I use OME-Zarr? +In general, OME-Zarr is growing as a default [FAIR](https://en.wikipedia.org/wiki/FAIR_data) choice for storing and sharing microscopy images. + OME-Zarr files have two major benefits: -* Chunking is inherent to "Zarr" files. This means "Zarr" files are stored in independently-accessible blocks. - * **Storage**: Microscopy images can be quite large and can therefore reach Cloud system storage limits for individual files; the chunked nature of a Zarr can alleviate this issue. Some storage systems may also duplicate byte-equivalent files, so a chunked file like Zarr may save storage space. - * **Viewing**: Viewers can target specific chunks to load based upon the current view, reducing lag, & enabling massive images to be viewed within browsers. - * **Cost**: When viewing or reading data, the total cost of accessing a Zarr file on the cloud may be less than a more monolithic file format due to the more efficient data access patterns. Ex. A viewer can just access the chunks of the image it needs to display rather than the entire image. -* "OME-Zarr" is a "Zarr" with embedded standardized metadata in the Open Microscopy Environment (OME) format. - * **Sharing**: A standardized imaging metadata format can ease cross-organization file sharing and can therefore aid organizational collaboration and data sharing. - * **Interoperability**: Standardized metadata can also enable the interoperability of tools. +- **Standardization:** "OME-Zarr" is a "Zarr" with embedded standardized metadata in the Open Microscopy Environment (OME) format. + - **_Sharing_**: It eases cross-organization file sharing, aiding organizational collaboration and data sharing. Repositories like the [Image Data Resource](https://idr.openmicroscopy.org/) and [BioImage Archive](https://www.ebi.ac.uk/bioimage-archive/) are currently migrating to having OME-Zarr as a standard format for _all_ their data. + - **_Interoperability_**: Standardized metadata enables the ability to "mix and match" tools from different organizations, benefiting from the strengths of multiple tools as needed. + +- **Parallel access**: Chunking is inherent to "Zarr" files. This means "Zarr" files are stored in independently-accessible blocks. + - **_Storage_**: Microscopy images can be quite large and can therefore reach Cloud system storage limits for individual files; the chunked nature of a Zarr can alleviate this issue. Some storage systems may also duplicate byte-equivalent files, so a chunked file like Zarr may save storage space. + - **_Viewing_**: Viewers can target specific chunks to load based upon the current view, reducing lag, & enabling massive images to be viewed within browsers. + - **_Cost_**: When viewing or reading data, the total cost of accessing a Zarr file on the cloud may be less than a more monolithic file format due to the more efficient data access patterns. Ex. A viewer can just access the chunks of the image it needs to display rather than the entire image. + +Of note, both benefits contribute to **_AI-readiness_**: the standardized metadata & access patterns provide a common layer for machine-learning workflows, reducing the friction for developers to build and test models. + +The [tools](#resources-tools), [data](#resources-data), and [ecosystem](#resources-ecosystem) may provide a better sense of the range of scientific use cases that may benefit from OME-Zarr. The [publications](#resources-publications) page provides a list of publications that have used OME-Zarr in their work. ## When would I not use OME-Zarr? -* If your file isn't very big and you're working with local data, the current specification of OME-Zarr can be less convenient than a single-file format and the benefits are limited. Planned expansions to the OME-Zarr specification will make it more convenient to work with it in these scenarios (e.g. single-file Zarrs) and add features that might make it beneficial to use OME-Zarr even in these scenarios (e.g. transformations). -* If your original file is lossy compressed, you will see a large increase in file size as the images are decompressed into OME-Zarr. There is not yet support for transferring lossy compressed image tiles to OME-Zarr. This currently impacts most whole slide image (WSI) formats such as SVS, CZI, and NDPI, which are lossy JPEG compressed by default and see about a 10x size increase into OME-Zarr. +While the format matures, it may be frustrating to use OME-Zarr in some cases, for example: + +- If you are working with small images, not planning to share them and your current tools already work well, then using OME-Zarr may not be necessary. Planned expansions to the specification (such as single-file Zarrs) will make it more convenient in these scenarios. + +- If you need specific conditions for which OME-Zarr support is not mature, you may need to use a different file format. + - Particularly, if your original file is lossy compressed, you will see a large increase in file size (about an order of magnitude) as the images are decompressed into OME-Zarr, since transferring lossy compressed tiles is not yet supported. This currently impacts most whole slide image (WSI) formats such as SVS, CZI, and NDPI, which are lossy JPEG compressed by default. ## Who is using OME-Zarr? -These are *some* of the organizations (and their dataset pages) that are using OME-Zarr for their data. +These are _some_ of the organizations (and their dataset pages) that are using OME-Zarr for their data. - [Allen Institute](https://bff.allencell.org/datasets) - biohub @@ -49,16 +61,18 @@ These are *some* of the organizations (and their dataset pages) that are using O - [EMBL - Image Data Resource (IDR)](https://idr.openmicroscopy.org/) - [Howard Hughes Medical Institute, Janelia (HHMI)](https://openorganelle.janelia.org/) - [Jackson Laboratory (JAX)](https://images.jax.org/) -- ... [and more](./data/index) +- ... [and more](#resources-data) ## How do I use OME-Zarr? -* Already have a Zarr? - * Check out the [tools section](./tools/index)! -* Want to create a Zarr? - * Check out the [tools section](./tools/index)! -* Want to see or download a Zarr? - * Check out the [data section](./data/index)! +- Already have a Zarr? + - Check out the [tools section](#resources-tools)! +- Want to create a Zarr? + - Check out the [tools section](#resources-tools)! +- Want to see or download a Zarr? + - Check out the [data section](#resources-data)! +- Want to cite OME-Zarr/NGFF in your work? + - Check out the [publications section](#resources-publications)! ## Have other questions? diff --git a/resources/data/index.md b/resources/data/index.md index aef32357..e1c875dc 100644 --- a/resources/data/index.md +++ b/resources/data/index.md @@ -1,3 +1,4 @@ +(resources-data)= # Data Resources Looking for test data? [IDR Samples](https://idr.github.io/ome-ngff-samples/) aims at covering the different corners of the specification. @@ -16,7 +17,7 @@ Resources curating OME-Zarr sample data, for demonstration and testing purposes. | BIA Samples | [BIA Samples logo](https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/omengff.html) | Sample OME-Zarr datasets from the BioImage Archive for testing | | Sanger Institute Samples | [Sanger Institute Samples logo](https://www.sanger.ac.uk/project/ome-zarr/) | Datasets from the Sanger Institute that have been converted to OME-Zarr to test and encourage the file format | | SSBD samples | [SSBD samples logo](https://ssbd.riken.jp/ssbd-ome-ngff-samples) | Sample OME-Zarr datasets from the Systems Science of Biological Dynamics database (SSBD) for testing and demonstration purposes | -| OME 2024 NGFF challenge | [OME 2024 NGFF challenge logo](https://ome.github.io/) | Close to 500 TB of data in the OME-Zarr 0.5 format | +| OME 2024 NGFF challenge | [OME 2024 NGFF challenge logo](https://ome.github.io/) | Close to 500 TB of data in the OME-Zarr 0.5 format | ## Data portals diff --git a/resources/ecosystem/index.md b/resources/ecosystem/index.md index 4b9dbf4f..10c92ea1 100644 --- a/resources/ecosystem/index.md +++ b/resources/ecosystem/index.md @@ -1,8 +1,9 @@ +(resources-ecosystem)= # Ecosystem Workflows and data formats that adopt OME-Zarr and build upon it for extended capabilities and interoperabilitieswith additional systems. -It extends the [tools](../tools/index.md) section by including projects that are not necessarily tools for viewing, writing, reading, converting, validating or processing OME-Zarr data, but that are still part of a ecosystem of projects with first-class, dedicated support for OME-Zarr. +It extends the [tools](#resources-tools) section by including projects that are not necessarily tools for viewing, writing, reading, converting, validating or processing OME-Zarr data, but that are still part of a ecosystem of projects with first-class, dedicated support for OME-Zarr. It fits microscope acquisition software, analysis pipelines, standards and extensions, gallery-like displays, integrative workflows, and the like. diff --git a/resources/publications/index.md b/resources/publications/index.md index 3ab917c1..8c8dbc34 100644 --- a/resources/publications/index.md +++ b/resources/publications/index.md @@ -1,6 +1,138 @@ -Publications -============ +(resources-publications)= -* [OME-NGFF: a next-generation file format for expanding bioimaging data-access strategies](https://www.nature.com/articles/s41592-021-01326-w) 29th November 2021 -* [OME-Zarr: a cloud-optimized bioimaging file format with international community support](https://link.springer.com/article/10.1007/s00418-023-02209-1) 10th July 2023 -* [2024 OME-NGFF workflows hackathon Preprint](https://osf.io/preprints/biohackrxiv/5uhwz_v2) 13 March 2025 +# Publications + +Core publications on NGFF and OME-Zarr. If you plan to cite OME-Zarr in your work, please use one or more of the following references, as appropriate. + +- [OME-NGFF: a next-generation file format for expanding bioimaging data-access strategies](https://www.nature.com/articles/s41592-021-01326-w) 29th November 2021 +- [OME-Zarr: a cloud-optimized bioimaging file format with international community support](https://link.springer.com/article/10.1007/s00418-023-02209-1) 10th July 2023 +- [2024 OME-NGFF workflows hackathon Preprint](https://osf.io/preprints/biohackrxiv/5uhwz_v2) 13 March 2025 + +## Recent publications mentioning NGFF or OME-Zarr + +This feed is generated live from Europe PMC and will show the most recent publications that mention NGFF or OME-Zarr in the title, abstract, or keywords. + +```{raw} html + + +
+

Loading recent publications…

+ +
+ + +``` diff --git a/resources/rfc-status-codes/index.md b/resources/rfc-status-codes/index.md index 651c74b4..7f487f04 100644 --- a/resources/rfc-status-codes/index.md +++ b/resources/rfc-status-codes/index.md @@ -2,7 +2,7 @@ RFC status codes ============ -[RFC 1](../../rfc/1) defined a Request for Comments process for the NGFF community, which is used to drive changes. The text listed a set of codes used to define how a change progresses from an idea to actual adoption. +[RFC 1](#rfcs:rfc1) defined a Request for Comments process for the NGFF community, which is used to drive changes. The text listed a set of codes used to define how a change progresses from an idea to actual adoption. ![State diagram of the RFC process](../../rfc/1/diagram.png) diff --git a/resources/tools/index.md b/resources/tools/index.md index 7537acd0..fdceb8bd 100644 --- a/resources/tools/index.md +++ b/resources/tools/index.md @@ -1,3 +1,4 @@ +(resources-tools)= # Tools A list of tools and libraries with OME-Zarr support. These are developed by various members of the NGFF community. If you think your tool/library should be listed here, please [open a pull request](https://github.com/ome/ngff). @@ -44,7 +45,7 @@ Want to view a Zarr? Use one of these. | Odon | [Odon logo](https://github.com/alexcoulton/odon) | A spatial proteomics OME-Zarr viewer built in Rust.| | QuPath | [QuPath logo](https://github.com/qupath/qupath) | Open source software for digital pathology image analysis | | syGlass | [syGlass logo](https://www.syglass.io/) | A VR desktop application for visualizing and segmenting 3D image data, with OME-Zarr streaming support. | -| Viv (Avivator, Vizarr, Vitessce, ...) | [Viv logo](https://github.com/hms-dbmi/viv) | A toolkit for interactive visualization of high-resolution, multiplexed bioimaging datasets. The viv toolkit is used by the [Avivator](https://avivator.gehlenborglab.org), [Vizarr](https://github.com/hms-dbmi/vizarr) and [Vitessce](http://vitessce.io) image viewers, among others | +| Viv (Avivator, Vizarr, Vitessce, ...) | [Viv logo](https://github.com/hms-dbmi/viv) | A toolkit for interactive visualization of high-resolution, multiplexed bioimaging datasets. The viv toolkit is used by the [Avivator](https://avivator.gehlenborglab.org), [Vizarr](https://github.com/hms-dbmi/vizarr) and [Vitessce](https://vitessce.io) image viewers, among others | | Vol-E | [Vol-E logo](https://vole.allencell.org/) | A browser-based volume viewer | | WEBKNOSSOS | [WEBKNOSSOS logo](https://home.webknossos.org/) | An open-source tool for annotating and exploring large 3D image datasets | @@ -107,7 +108,7 @@ Want to validate a Zarr? Use one of these. | Name | Link | Description | | -------- | ------- | ------- | -| ome-ngff-validator | [OME NGFF Validator logo](https://ome.github.io/ome-ngff-validator/) | Web page for validating OME-Zarr files. | +| ome-ngff-validator | [OME NGFF Validator logo](https://ome.github.io/ome-ngff-validator/) | Web page for validating OME-Zarr files. | | ome-zarr-models | [![image](https://github.githubassets.com/favicons/favicon.svg)](https://github.com/ome-zarr-models/ome-zarr-models-py) | Python package and command line interface that can validate OME-Zarr files | | yaozarrs | [![image](https://github.githubassets.com/favicons/favicon.svg)](https://github.com/imaging-formats/yaozarrs) | Bottom-up python library with models and CLI for creating & validating OME-Zarr groups and documents with optional extras for array-backend-agnostic I/O | diff --git a/rfc/1/index.md b/rfc/1/index.md index ee7fba5f..a4a931b6 100644 --- a/rfc/1/index.md +++ b/rfc/1/index.md @@ -1,6 +1,8 @@ RFC-1: RFC Process ================== +(rfcs:rfc1)= + Definition of the NGFF “Request for Comment” (RFC) process ```{toctree} @@ -238,6 +240,7 @@ right direction. **Reviewers** should strive to provide feedback which informs * **Commenters** are other members of the community who, though not contacted as **Reviewers**, have provided feedback that they would like added to the official record of the RFC. +(rfc1-implementation)= ## Implementation The RFC process can be represented as a state diagram with the various stakeholders responsible for forward motion. @@ -322,7 +325,8 @@ to the **Editors**, either via a public PR adding the review in markdown to the RFC's subdirectory or by emailing the **Editors** directly. (This latter course should only be used when necessary.) -(rfc-recommendations)= Possible recommendations from **Reviewers** in ascending order of support are: +(rfc-recommendations)= +Possible recommendations from **Reviewers** in ascending order of support are: * “Reject” suggests that a **Reviewer** considers there to be no merit to an RFC. This should be a last recourse. Instead, suggestions in a “Major @@ -367,7 +371,8 @@ contact **Reviewers** to see if their recommendations have changed. > 🕑 Authors responses to Reviewers should be returned to the Editors in less than two weeks. -(anchor-rebuttal-r6)= This brings a critical, and possibly iterative, decision point (R6). If all **Reviewers** `approve` and there are no further changes needed, the RFC can progress to S1 as soon as there are two in-progress implementations. If the **Reviewers** do _not_ approve, then the **Editors** will make one of three decisions (R7): +(anchor-rebuttal-r6)= +This brings a critical, and possibly iterative, decision point (R6). If all **Reviewers** `approve` and there are no further changes needed, the RFC can progress to S1 as soon as there are two in-progress implementations. If the **Reviewers** do _not_ approve, then the **Editors** will make one of three decisions (R7): * The **Editors** MAY provide **Authors** a list of necessary changes. These will be based on the **Reviewers** suggestions but possibly modified, e.g., to remove contradictions. * The **Editors** MAY decide that the RFC is to be closed (R9). This is the decision that SHOULD be chosen if there is a unanimous `Reject` recommendation. The **Authors** MAY then decide to re-draft a new RFC (D2). diff --git a/rfc/1/templates/review_template.md b/rfc/1/templates/review_template.md index 10597d42..f1d37f7c 100644 --- a/rfc/1/templates/review_template.md +++ b/rfc/1/templates/review_template.md @@ -1,3 +1,4 @@ +(rfc1-review-template)= # Review Template Replace the title above of this file with “RFC-NUM: Review NUM” diff --git a/rfc/3/comments/3/index.md b/rfc/3/comments/3/index.md index 289e8be4..02e3e405 100644 --- a/rfc/3/comments/3/index.md +++ b/rfc/3/comments/3/index.md @@ -1,5 +1,7 @@ # RFC-3: Comment 3 +(rfcs:rfc3:comment3)= + | **Role** | Name | GitHub Handle | Institution | |----------|------|---------------|-------------| | **Author** | [Cornelia Wetzker](https://orcid.org/0000-0002-8367-5163) | [cwetzker](https://github.com/cwetzker) | TU Dresden | diff --git a/rfc/3/versions/2026-07-17/index.md b/rfc/3/versions/2026-07-17/index.md index 954d514f..78178c27 100644 --- a/rfc/3/versions/2026-07-17/index.md +++ b/rfc/3/versions/2026-07-17/index.md @@ -86,19 +86,19 @@ This RFC is currently in RFC state `R1` (send for review). - [btbest](https://github.com/btbest) - - 2026-02-02 - - [Comment](./comments/1/index) + - [Comment](#rfcs:rfc3:comment1) * - Commenter - Chris Barnes - [clbarnes](https://github.com/clbarnes) - German BioImaging - 2026-02-05 - - [Comment](./comments/2/index) + - [Comment](#rfcs:rfc3:comment2) * - Commenter - Cornelia Wetzker - [cwetzker](https://github.com/cwetzker) - Technische Universität Dresden - 2026-03-19 - - [Comment](./comments/3/index) + - [Comment](#rfcs:rfc3:comment3) ``` ## Overview @@ -109,7 +109,7 @@ conversion of proprietary datasets, usage by microscope vendors[^1], and usage by novel microscopy modalities[^2]. This RFC removes these restrictions, opening NGFF to many more users within its -target domain (and beyond). Because it *only* removes restrictions, existing +target domain (and beyond). Because it _only_ removes restrictions, existing valid OME-Zarr datasets will remain valid after implementation of this proposal. @@ -144,7 +144,6 @@ and, > (`yx`) and images are stacked along the other (anisotropic) axis (`z`), the > spatial axes SHOULD be ordered as `zyx`. - and, > - Every Zarr array referred to by a `path` MUST have the same number of @@ -158,9 +157,9 @@ hamper the adoption of OME-Zarr as an acquisition-time format due to performance concerns: many acquisitions happen in TZCYX order (all channels are acquired together for each z-slice), which violates the "axes must be ordered by type" requirement. In such cases, scientists must first acquire their data -and *then* transpose it — an expensive proposition for large datasets. (Note: +and _then_ transpose it — an expensive proposition for large datasets. (Note: Admittedly, Zarr transpose codecs, as well as the mapAxis transformation from -RFC-5, already offer solutions to this problem. However, the *simplest* +RFC-5, already offer solutions to this problem. However, the _simplest_ solution of flexible array ordering with default codecs and only scale and translation transforms is only open after this RFC.) @@ -195,7 +194,7 @@ fluorophore decay lifetime can be used to get more information about the sample, for example by separating fluorophores not only according to their emission spectra but also their lifetime. The decay measurement constitutes an additional time axis called `u` or `µ` and can result in a 6-dimensional -dataset with axes `CTUZYX`. (See [comment 3](./comments/3/index), and [this +dataset with axes `CTUZYX`. (See [comment 3](#rfcs:rfc3:comment3), and [this paper][flim-paper].) FLIM data currently cannot be represented in OME-Zarr, causing users to use @@ -219,7 +218,7 @@ important part of data quality checks. Currently, this kind of microscopy data cannot be stored in OME-Zarr. Within the same workflow, the crystal orientation is usually encoded as three -*Euler angles* or four *quaternion* components stored at each pixel position. +_Euler angles_ or four _quaternion_ components stored at each pixel position. Although it's possible to use the "custom" axis for this purpose, a dedicated axis with a clear naming convention is more ergonomic for scientists. @@ -241,17 +240,17 @@ to save these intermediate image products for later reprocessing. ### Overlapping labels -OME-Zarr can represent not only raw images, but also *label images*, or +OME-Zarr can represent not only raw images, but also _label images_, or segmentations. Traditionally, segmentations assigned a single integer value to each pixel in the source image. However, newer segmentation methods produce -*coarse-to-fine* segmentations, with semantic meaning. For images spanning many +_coarse-to-fine_ segmentations, with semantic meaning. For images spanning many scales, as produced by modern microscopes, we may want to segment tissues, cells within those tissues, and organelles within those cells. These three levels of segmentations can be stacked along a new "coarseness" axis. -Similarly, new segmentation methods can produce *overlapping binary masks*. Due -to the overlap, they *cannot* be stored as traditional integer masks (one value -per pixel), but are typically instead stored as *n* boolean masks. For this +Similarly, new segmentation methods can produce _overlapping binary masks_. Due +to the overlap, they _cannot_ be stored as traditional integer masks (one value +per pixel), but are typically instead stored as _n_ boolean masks. For this mask, an extra "instance" axis is needed in addition to the [TCZ]YX axes of a source image. @@ -261,7 +260,7 @@ Downstream computational products are necessary for the analysis of microscopy data, and these should ideally be stored in formats compatible with the tooling used to work with the raw data. The format aims to be flexible enough to accommodate these products, such as segmentations (the `label-image` section), -or the outputs of registrations (see [rfc-5](../5/index)). +or the outputs of registrations (see [rfc-5](#rfcs:rfc5:version3)). Some outputs may naturally contain arbitrary dimensions, for example, a grid search of the parameter space for a denoising algorithm may contain an @@ -290,7 +289,8 @@ After this specification change, tools may encounter OME-Zarr files that don't match the earlier expectations of containing a subset of the TCZYX axes. This proposal is agnostic as to what to do in those situations, and indeed the appropriate action depends on the tool, but some suggestions include: -- fail with an informative error message. (i.e. *partial* implementations are + +- fail with an informative error message. (i.e. _partial_ implementations are OK, especially if well-documented.) - prompt the user about which axes to treat as spatial. - arbitrarily choose which axes to treat as spatial. @@ -304,36 +304,31 @@ document, taking as base the current development version: > - The length of `axes` must be between 2 and 5 and MUST be equal to the > dimensionality of the Zarr arrays storing the image data (see > `datasets:path`). -> > - `axes` MUST contain 2 or 3 entries of `type:space` -> > - `axes` MAY contain one additional entry of `type:time` -> > - `axes` MAY contain one additional entry of `type:channel` or a null / > custom type. -> > - `axes` entries MUST be ordered by `type` where the `time` axis must come > first (if present), followed by the `channel` or custom axis (if present) > and the axes of type `space`. -> > - If there are three spatial axes where two correspond to the image plane > (`yx`) and images are stacked along the other (anisotropic) axis (`z`), the > spatial axes SHOULD be ordered as `zyx`. -2. The following lines are *added* to "multiscales metadata": +2. The following lines are _added_ to "multiscales metadata": > 0. The length of the axis names MUST match the number of axes of the array. -> 1. *If* a dataset contains exactly 2 spatial dimensions, those dimensions +> 1. _If_ a dataset contains exactly 2 spatial dimensions, those dimensions > SHOULD be named `y` and `x`, except where rule 4 applies. -> 2. *If* a dataset contains exactly 3 spatial dimensions, those dimensions +> 2. _If_ a dataset contains exactly 3 spatial dimensions, those dimensions > SHOULD be named 'z', 'y', and 'x', except where rule 4 applies. -> 3. *If* a dataset contains exactly 1 time dimension, it should be named `t`. +> 3. _If_ a dataset contains exactly 1 time dimension, it should be named `t`. > 4. When image data axes map straightforwardly to axes with common names in > the relevant field of practice, those axes SHOULD be named according to > such conventions. For example, spatial frequency axes resulting from a > Fourier transformation of `z', 'y', and 'x' SHOULD be named 'w', 'v', and -> `u`, respectively. Similarly, a temporal frequency axis resulting from -> a Fourier transformation of the `t` axis SHOULD be named `w` or `ω`. +`u`, respectively. Similarly, a temporal frequency axis resulting from +a Fourier transformation of the `t`axis SHOULD be named`w`or`ω`. > 5. Axis names MUST NOT be repeated within a dataset, and SHOULD NOT be > different only by upper/lower-case. For example, the same dataset SHOULD > NOT have both an `X` and an `x` axis. @@ -370,7 +365,7 @@ the RFC. Developers of visualization libraries and software are concerned that this PR may result in a "wild west" of OME-Zarr datasets that have little in common, -making it difficult for tools to decide *which* axes to display. More details +making it difficult for tools to decide _which_ axes to display. More details about these concerns are developed below in the "Drawbacks" section. ## Backwards Compatibility @@ -426,21 +421,21 @@ have assuaged most implementation concerns][recap comment]. ## Performance The current OME-Zarr specification ensures arrays are stored in order TCZYX. -With C-order array data, this ensures efficient access for *some* but not *all* +With C-order array data, this ensures efficient access for _some_ but not _all_ access patterns. By removing restrictions on axis orderings, a new class of "mistake" is possible, as someone could save an array in order XYTCZ, which would combine poorly with C-order arrays to view XY planes. However, it is -arguable that Zarr chunking is in fact more important here — XYTCZ *could* be +arguable that Zarr chunking is in fact more important here — XYTCZ _could_ be a perfectly cromulent axis ordering for XY planes if the Zarr chunk size was (1024, 1024, 1, 1, 1). Moreover, imposing a fixed axis ordering can incur performance penalties at -*write* time (where performance is often critical) if the data is not already +_write_ time (where performance is often critical) if the data is not already in the expected order. Therefore, this proposal argues that any performance implications are better addressed through good documentation and good defaults. Indeed, more flexible -dimension ordering could *improve* performance in some scenarios, such as +dimension ordering could _improve_ performance in some scenarios, such as "pixel drilling", that is, extracting the value of a single x/y position over time. @@ -475,7 +470,6 @@ an informative error message (e.g. "The given dataset contains an unknown axis This RFC is placed in the public domain. - [nat methods paper]: https://www.nature.com/articles/s41592-021-01326-w [ome-model]: https://github.com/ome/ngff/pull/239/files#r1609781780 [ngff 0.4]: https://ngff.openmicroscopy.org/specifications/0.4/index.html @@ -502,11 +496,12 @@ This RFC is placed in the public domain. [zulip-rfc-3-thread]: https://imagesc.zulipchat.com/#narrow/channel/328251-NGFF/topic/RFC-3.3A.20remove.20dimension.20restrictions/near/568178521 [^1]: https://github.com/ome/ngff/pull/239#issuecomment-2122809286 + [^2]: https://github.com/ome/ngff/pull/239#issuecomment-2149119404 ## Changelog -| Date | Description | Link | -| ---------- | ---------------------------- | ---------------------------------------------------------------------------- | -| 2024-10-08 | RFC assigned and published | [https://github.com/ome/ngff/pull/239](https://github.com/ome/ngff/pull/239) | -| 2026-07-04 | Updated to address comments, elaborate on use cases, include specific changes to spec doc, and add test data | [https://github.com/ome/ngff/pull/560](https://github.com/ome/ngff/pull/560) | +| Date | Description | Link | +| ---------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | +| 2024-10-08 | RFC assigned and published | [https://github.com/ome/ngff/pull/239](https://github.com/ome/ngff/pull/239) | +| 2026-07-04 | Updated to address comments, elaborate on use cases, include specific changes to spec doc, and add test data | [https://github.com/ome/ngff/pull/560](https://github.com/ome/ngff/pull/560) | diff --git a/specifications/index.md b/specifications/index.md index e84b8ad8..2ff0644e 100644 --- a/specifications/index.md +++ b/specifications/index.md @@ -1,9 +1,8 @@ -Specifications -============== +# Specifications OME-Zarr files have standardized metadata (that is the OME portion of "OME-Zarr") - each new version of OME-Zarr files has its own specification. Those specifications are listed below. -The current released version of the OME-Zarr specification is [0.5](0.5/ngff_spec/index.md). +The current released version of the OME-Zarr specification is [0.5](0.5/index.md). ```{toctree} :maxdepth: 1 @@ -17,4 +16,4 @@ dev/index.md 0.1/index.md dev/version_history -``` \ No newline at end of file +```