Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions .github/workflows/html-proofer.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
name: "Build and Lint Site"

on:
workflow_dispatch:
pull_request:
branches: ["main"]
paths:
- "specifications/**"
- "rfc/**"
- "resources/**"
- "community/**"
- "contributing/**"
- "help-desk/**"
- "_html_extra/**"
- "_static/**"
- "_ext/**"
- "images/**"
- "**/*.md"
- "conf.py"
- "requirements.txt"
- "readthedocs.yml"
- ".github/workflows/build-lint.yml"

concurrency:
group: build-lint-${{ github.ref }}
cancel-in-progress: true

jobs:
build-site:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
submodules: recursive

# dependencies
- uses: ruby/setup-ruby@v1
with:
ruby-version: "3.3"
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- uses: actions/setup-node@v4
with:
node-version: "22"

- name: Install dependencies
run: |
pip install -r requirements.txt
pip install specifications/dev
gem install html-proofer

- name: Build the site
run: sphinx-build -b html . _build/html

- name: Validate all links, enforce alt text
run: htmlproofer --ignore-files "/.*\/specifications\/(0\.[0-9]+|dev)\/.*/,/.*webpack-macros\.html/,/.*\/rfc\/4\/markdown\/.*/,/.*\/rfc\/5\/.*/,/.*\/rfc\/1\/.*/,/.*\/0\.[0-9]+(\.html|\/.*)/" --disable-external ./_build/html

- name: Ensure no unexpected encoded HTML in output
run: |
! fgrep -R 'lt;blockquote' _build/html
11 changes: 1 addition & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,9 @@ See also [CONTRIBUTING.md](./contributing/index.md)

Specifications have been moved to [ome/ngff-spec](https://github.com/ome/ngff-spec).

### Editing specifications

Specifications are written in markdown, or technically
[bikeshed](https://github.com/tabatkins/bikeshed) -- a markdown document, with
special extensions understood by the bikeshed tool. The bikeshed tool is run
during the Sphinx build step (see conf.py).

# RFCs

Requests for comments (RFCs) are used to discuss and capture high-level decisions within the NGFF community.
Requests for comments (RFCs) are used to discuss and capture high-level decisions within the NGFF community.

RFCs are contained under the `rfc` directory at the moment but may be moved out into a separate repo in the future.

Expand All @@ -37,5 +30,3 @@ RFCs are contained under the `rfc` directory at the moment but may be moved out
<!-- prettier-ignore-end -->

<!-- ALL-CONTRIBUTORS-LIST:END -->


2 changes: 1 addition & 1 deletion boilerplate/header.include
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
<body class="h-entry">
<div class="head">

<img src="http://www.openmicroscopy.org/img/logos/ome-logomark.svg" alt="OME logo (6 circles in a hexagon)" style="float:right;width:42px;height:42px;">
<img src="https://www.openmicroscopy.org/img/logos/ome-logomark.svg" alt="OME logo (6 circles in a hexagon)" style="float:right;width:42px;height:42px;">

<h1 id="title" class="p-name no-ref">[TITLE]</h1>
<h2 id="subtitle" class="no-num no-toc no-ref">[LONGSTATUS],
Expand Down
6 changes: 3 additions & 3 deletions contributing/specification/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,14 @@ request for comment (RFC) documents.

In the future we will flesh out this page with a guide to RFCs, but in the
meantime the RFC process is outlined in the
[Implementation section of RFC 1](../rfc/1/index.md#implementation).
[Implementation section of RFC 1](rfc1-implementation).

## Comment on a Request For Comment (RFC)

If you want to leave a suggestion or comment on an RFC that is under review,
please leave a comment in a new page under the "comments/" directory for the
relevant RFC. A template is also available for formatting your comment:
[template](../rfc/1/templates/review_template).
[template](rfc1-review-template).

## How to change the specification

Expand Down Expand Up @@ -143,7 +143,7 @@ Other useful admonitions and directives (e.g., `hint`, `note`) can be found [her

## Building *only* the spec document

The spec document under the [ngff-spec](github.com/ome/ngff-spec) repository can be built as a standalone document to make writing and rendering a smoother experience.
The spec document under the [ngff-spec](https://github.com/ome/ngff-spec) repository can be built as a standalone document to make writing and rendering a smoother experience.
To build the spec document, you first need to install the necessary dependencies:

After cloning the ngff-spec repo, navigate into the repository on your machine and install the dependencies using pip:
Expand Down
63 changes: 51 additions & 12 deletions help-desk/index.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Help Desk

* [Glossary](#glossary)
* [FAQs](#faqs)
* [Where to seek for help](#where-to-seek-for-help)
- [Glossary](#glossary)
- [FAQs](#faqs)
- [Where to seek for help](#where-to-seek-for-help)

## Glossary

Expand All @@ -29,31 +29,56 @@ Jump to:

[Back to top](#glossary)

### D

[Back to top](#glossary)

### E

[Back to top](#glossary)

### F

[Back to top](#glossary)

### G

[Back to top](#glossary)

### H
### H

**HCS** - Dedicated layout of OME-Zarr files representing the used wells and plates for High Content Screening (HCS). The layout allows a user to efficiently address and analyze specific data from specific wells.

**HDF5** - Hierarchical Data Format version 5 (HDF5). It is a data model, file format, and library for organizing, storing, and managing large and complex data.

[Back to top](#glossary)

### I

[Back to top](#glossary)

### J

**JSON file** - A simple standardized human-readable data format for sharing data using nested key-value pairs.

[Back to top](#glossary)

### K

[Back to top](#glossary)

### L

[Back to top](#glossary)

### M

**Multiscale** - Multiscale means storing the same image at multiple resolutions (full size, half size, quarter size, etc.). This makes it easy for tools to zoom and navigate very large images quickly—just like how map apps load lower-resolution versions first, then sharpen the view as you zoom in.

[Back to top](#glossary)

### N

**NGFF** - Next Generation File Format (NGFF) is a community for solving problems related to producing open-source software for microscopy data.

[Back to top](#glossary)
Expand All @@ -66,7 +91,7 @@ Jump to:

**OME-NGFF** - OME-NGFF refers to the resulting efforts of the OME & NGFF communities.

**OME-Zarr** - An OME-Zarr is a standardized file format for microscopy images. More details on the [main page](#../).
**OME-Zarr** - An OME-Zarr is a standardized file format for microscopy images. More details on the [main page](#main-page).

[Back to top](#glossary)

Expand All @@ -76,6 +101,10 @@ Jump to:

[Back to top](#glossary)

### Q

[Back to top](#glossary)

### R

**RFC** - Request For Comments (RFC) is a process for proposing a change to the standardized specification for OME-Zarr files.
Expand All @@ -84,7 +113,7 @@ Jump to:

### S

**Shard/ing** - Like [Chunks](#c), sharding is a way of breaking up pixel data for more efficient access. Shards are groups of chunks with headers in front of them for describing the chunks inside.
**Shard/ing** - Like [Chunks](#c), sharding is a way of breaking up pixel data for more efficient access. Shards are groups of chunks in a single object with headers describing the chunks inside.

[Back to top](#glossary)

Expand All @@ -96,6 +125,9 @@ Jump to:

### U

[Back to top](#glossary)

### V

[Back to top](#glossary)

Expand All @@ -105,9 +137,17 @@ Jump to:

[Back to top](#glossary)

### X

[Back to top](#glossary)

### Y

[Back to top](#glossary)

### Z

**Zarr** -- A file format, see the [main page](#../) for more information about what a Zarr is and the benefits of using Zarr.
**Zarr** -- A file format, see the [main page](#main-page) for more information about what a Zarr is and the benefits of using Zarr.

**Zarr Group** - A Zarr group is like a folder inside a dataset. It can hold arrays, metadata, or other groups, helping organize complex data into a clear hierarchy. Think of it as a way to keep all related images and information neatly together.

Expand Down Expand Up @@ -172,15 +212,14 @@ That is often the case. OME-Zarr include multiresolution data (i.e. pyramids), w

Not yet fully. Some workarounds are possible (1D arrays, mesh formats), but this is an active area of development.



(where-to-seek-for-help)=
## Where to look for help

If you have questions or need help with OME-Zarr, you can reach out through the following channels:
| Where | Description | When to use |
| Where | Description | When to use |
|-------|-------------| --------------|
| [Image.sc Forum](https://forum.image.sc) | A community forum for image analysis and bioimaging. | To know when to use the tags [ngff](https://forum.image.sc/tag/ngff), [ome-ngff](https://forum.image.sc/tag/ome-ngff), and [ome-zarr](https://forum.image.sc/tag/ome-zarr) in the forum please read [Landing Page](https://ngff.openmicroscopy.org/index.html), Glossary and FAQs in this page. |
| [Image.sc Forum](https://forum.image.sc) | A community forum for image analysis and bioimaging. | To know when to use the tags [ngff](https://forum.image.sc/tag/ngff), [ome-ngff](https://forum.image.sc/tag/ome-ngff), and [ome-zarr](https://forum.image.sc/tag/ome-zarr) in the forum please read [Landing Page](https://ngff.openmicroscopy.org/index.html), Glossary and FAQs in this page. |
| [ome/ngff GitHub Issues](https://github.com/ome/ngff/issues) | The official repository for OME-NGFF specifications and related discussions. | For reporting bugs, suggesting features, or discussing technical aspects of the OME-NGFF specifications. |
| Office Hours | Regularly scheduled virtual meetings where you can ask questions and get help from the NGFF community | Office hours rotate between APAC/AU/EU- and AMER/AU/EU-friendly times. Join them when you want to chat about OME-Zarr. Read more in the [Community](../community/index.md) page. |
| OME-Zarr libraries GitHub Issues| Various repositories for OME-Zarr libraries. | For issues specific to a particular OME-Zarr library, use the respective GitHub repository's issue. |
| In person events | Conferences, workshops, and meetups where you can connect with the NGFF community. | To network, learn, and discuss OME-Zarr in person. Check the [Community](../community/index.md) page for upcoming events. |

60 changes: 37 additions & 23 deletions index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(main-page)=
# Next-Generation File Formats (NGFF) + OME-Zarr

Welcome to the Next-Generation File Formats (NGFF) main page! This site is dedicated to providing resources for the NGFF community and those that are interested in getting started with OME-Zarr.
Expand All @@ -15,50 +16,63 @@ In summary, OME-Zarr allows you to store, share and stream large images. You can
An OME-Zarr is a file format optimized for storing, viewing, & sharing large images.
There are two parts to an OME-Zarr:

* **The "Zarr" part describes how the pixel data for the images are laid out**
* The "OME", which stands for [Open Microscopy Environment](https://www.openmicroscopy.org/), part describes metadata about the pixel data. This includes metadata such as:
* spatial relationships
* high content screening data
* well data
* [and more](./specifications/index)!
- **The "Zarr" part describes how the pixel data for the images are laid out**. [Zarr](https://zarr.dev) is a next-generation data format used for scientific datasets in multiple domains.
- **The "OME" part describes metadata about the pixel data.** OME stands for [Open Microscopy Environment](https://www.openmicroscopy.org/). This includes metadata such as:
- spatial relationships
- high content screening data
- well data
- [and more](./specifications/index)!

## Why would I use OME-Zarr?

In general, OME-Zarr is growing as a default [FAIR](https://en.wikipedia.org/wiki/FAIR_data) choice for storing and sharing microscopy images.

OME-Zarr files have two major benefits:

* Chunking is inherent to "Zarr" files. This means "Zarr" files are stored in independently-accessible blocks.
* **Storage**: Microscopy images can be quite large and can therefore reach Cloud system storage limits for individual files; the chunked nature of a Zarr can alleviate this issue. Some storage systems may also duplicate byte-equivalent files, so a chunked file like Zarr may save storage space.
* **Viewing**: Viewers can target specific chunks to load based upon the current view, reducing lag, & enabling massive images to be viewed within browsers.
* **Cost**: When viewing or reading data, the total cost of accessing a Zarr file on the cloud may be less than a more monolithic file format due to the more efficient data access patterns. Ex. A viewer can just access the chunks of the image it needs to display rather than the entire image.
* "OME-Zarr" is a "Zarr" with embedded standardized metadata in the Open Microscopy Environment (OME) format.
* **Sharing**: A standardized imaging metadata format can ease cross-organization file sharing and can therefore aid organizational collaboration and data sharing.
* **Interoperability**: Standardized metadata can also enable the interoperability of tools.
- **Standardization:** "OME-Zarr" is a "Zarr" with embedded standardized metadata in the Open Microscopy Environment (OME) format.
- **_Sharing_**: It eases cross-organization file sharing, aiding organizational collaboration and data sharing. Repositories like the [Image Data Resource](https://idr.openmicroscopy.org/) and [BioImage Archive](https://www.ebi.ac.uk/bioimage-archive/) are currently migrating to having OME-Zarr as a standard format for _all_ their data.
- **_Interoperability_**: Standardized metadata enables the ability to "mix and match" tools from different organizations, benefiting from the strengths of multiple tools as needed.

- **Parallel access**: Chunking is inherent to "Zarr" files. This means "Zarr" files are stored in independently-accessible blocks.
- **_Storage_**: Microscopy images can be quite large and can therefore reach Cloud system storage limits for individual files; the chunked nature of a Zarr can alleviate this issue. Some storage systems may also duplicate byte-equivalent files, so a chunked file like Zarr may save storage space.
- **_Viewing_**: Viewers can target specific chunks to load based upon the current view, reducing lag, & enabling massive images to be viewed within browsers.
- **_Cost_**: When viewing or reading data, the total cost of accessing a Zarr file on the cloud may be less than a more monolithic file format due to the more efficient data access patterns. Ex. A viewer can just access the chunks of the image it needs to display rather than the entire image.

Of note, both benefits contribute to **_AI-readiness_**: the standardized metadata & access patterns provide a common layer for machine-learning workflows, reducing the friction for developers to build and test models.

The [tools](#resources-tools), [data](#resources-data), and [ecosystem](#resources-ecosystem) may provide a better sense of the range of scientific use cases that may benefit from OME-Zarr. The [publications](#resources-publications) page provides a list of publications that have used OME-Zarr in their work.

## When would I not use OME-Zarr?

* If your file isn't very big and you're working with local data, the current specification of OME-Zarr can be less convenient than a single-file format and the benefits are limited. Planned expansions to the OME-Zarr specification will make it more convenient to work with it in these scenarios (e.g. single-file Zarrs) and add features that might make it beneficial to use OME-Zarr even in these scenarios (e.g. transformations).
* If your original file is lossy compressed, you will see a large increase in file size as the images are decompressed into OME-Zarr. There is not yet support for transferring lossy compressed image tiles to OME-Zarr. This currently impacts most whole slide image (WSI) formats such as SVS, CZI, and NDPI, which are lossy JPEG compressed by default and see about a 10x size increase into OME-Zarr.
While the format matures, it may be frustrating to use OME-Zarr in some cases, for example:

- If you are working with small images, not planning to share them and your current tools already work well, then using OME-Zarr may not be necessary. Planned expansions to the specification (such as single-file Zarrs) will make it more convenient in these scenarios.

- If you need specific conditions for which OME-Zarr support is not mature, you may need to use a different file format.
- Particularly, if your original file is lossy compressed, you will see a large increase in file size (about an order of magnitude) as the images are decompressed into OME-Zarr, since transferring lossy compressed tiles is not yet supported. This currently impacts most whole slide image (WSI) formats such as SVS, CZI, and NDPI, which are lossy JPEG compressed by default.

## Who is using OME-Zarr?

These are *some* of the organizations (and their dataset pages) that are using OME-Zarr for their data.
These are _some_ of the organizations (and their dataset pages) that are using OME-Zarr for their data.

- [Allen Institute](https://bff.allencell.org/datasets)
- biohub
- [Broad Institute](https://broadinstitute.github.io/cellpainting-gallery/overview.html)
- [EMBL - Image Data Resource (IDR)](https://idr.openmicroscopy.org/)
- [Howard Hughes Medical Institute, Janelia (HHMI)](https://openorganelle.janelia.org/)
- [Jackson Laboratory (JAX)](https://images.jax.org/)
- ... [and more](./data/index)
- ... [and more](#resources-data)

## How do I use OME-Zarr?

* Already have a Zarr?
* Check out the [tools section](./tools/index)!
* Want to create a Zarr?
* Check out the [tools section](./tools/index)!
* Want to see or download a Zarr?
* Check out the [data section](./data/index)!
- Already have a Zarr?
- Check out the [tools section](#resources-tools)!
- Want to create a Zarr?
- Check out the [tools section](#resources-tools)!
- Want to see or download a Zarr?
- Check out the [data section](#resources-data)!
- Want to cite OME-Zarr/NGFF in your work?
- Check out the [publications section](#resources-publications)!

## Have other questions?

Expand Down
3 changes: 2 additions & 1 deletion resources/data/index.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
(resources-data)=
# Data Resources

Looking for test data? [IDR Samples](https://idr.github.io/ome-ngff-samples/) aims at covering the different corners of the specification.
Expand All @@ -16,7 +17,7 @@ Resources curating OME-Zarr sample data, for demonstration and testing purposes.
| BIA Samples | [<img src="https://www.google.com/s2/favicons?sz=256&domain_url=https://livingobjects.ebi.ac.uk/bioimaging-integrator-data/pages/idr_ngff_data.html" alt="BIA Samples logo" width="30" height="30">](https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/omengff.html) | Sample OME-Zarr datasets from the BioImage Archive for testing |
| Sanger Institute Samples | [<img src="https://www.google.com/s2/favicons?sz=256&domain_url=https://www.sanger.ac.uk/project/ome-zarr/" alt="Sanger Institute Samples logo" width="30" height="30">](https://www.sanger.ac.uk/project/ome-zarr/) | Datasets from the Sanger Institute that have been converted to OME-Zarr to test and encourage the file format |
| SSBD samples | [<img src="https://www.google.com/s2/favicons?sz=256&domain_url=https://ssbd.riken.jp/ssbd-ome-ngff-samples" alt="SSBD samples logo" width="30" height="30">](https://ssbd.riken.jp/ssbd-ome-ngff-samples) | Sample OME-Zarr datasets from the Systems Science of Biological Dynamics database (SSBD) for testing and demonstration purposes |
| OME 2024 NGFF challenge | [<img src="http://www.openmicroscopy.org/img/logos/ome-logomark.svg" alt="OME 2024 NGFF challenge logo" width="30" height="30">](https://ome.github.io/) | Close to 500 TB of data in the OME-Zarr 0.5 format |
| OME 2024 NGFF challenge | [<img src="https://www.openmicroscopy.org/img/logos/ome-logomark.svg" alt="OME 2024 NGFF challenge logo" width="30" height="30">](https://ome.github.io/) | Close to 500 TB of data in the OME-Zarr 0.5 format |

## Data portals

Expand Down
Loading
Loading