Skip to content

format.md

Latest commit

 

History

History
237 lines (190 loc) · 12.2 KB

format.md

File metadata and controls

237 lines (190 loc) · 12.2 KB
Raw

Decant Archive Format — Specification v1

DECANT_FORMAT = 1

This document specifies the on-disk format decant writes, precisely enough that a byte-exact archive can be reconstructed without the decant tool — using only this spec, a Zarr v2 reader, and NumPy. That independent-recoverability is the whole point: an archival format must outlive the program that wrote it.

1. What this is (and isn't)

A decant archive is a profile of OME-Zarr, not a new container format. A byte-exact archive is a valid OME-Zarr group with two additions:

  1. a husk array holding every non-pixel byte of the original file, and
  2. a .decant/manifest.json key describing how to reconstruct the original.

Any OME-Zarr reader opens and displays the pixels (0) and ignores the additions (the multiscales metadata references only 0 and any pyramid levels; Zarr skips the unknown .decant/ key). A decant-aware reader — or any implementation of §5 — uses the additions to regenerate the original file byte-for-byte.

2. Container

An archive is one of:

  • NAME.zarr.zip — a single file: a Zarr v2 ZipStore whose entries are STORED (zip compression method 0, no deflate). Stored matters twice: the chunks are already blosc-compressed, and stored entries stay contiguous / range-addressable. This aligns with the draft NGFF RFC-9 "Zipped OME-Zarr" (.ozx).
  • NAME.zarr/ — the same keys as an unpacked directory.

Both forms normally hold exactly one Zarr v2 group; the exception is the nested-series NAME.zarr/ directory (§3), which holds one sub-bundle per member. A reader accepts either form.

3. Group layout

Byte-exact (mode D), single file:

0/                     pixel array. dtype from the manifest; chunks put ONE plane
                       per chunk: (1,…,1, Y, X) or (1,…,1, Y, X, S). Codec is
                       configurable (default blosc-zstd-byteshuffle, clevel 5) and
                       recorded per-array in .zarray, so a reader decompresses it
                       without consulting the manifest.
1/, 2/, …              OPTIONAL derived multiscale pyramid levels. Viewing only —
                       NEVER read during reconstruction. XY-downsampled per plane.
husk/                  1-D uint8 array: the "husk" (§4). Codec: blosc(zstd, noshuffle).
.zgroup, .zattrs       Zarr v2 group doc + OME-NGFF 0.4 metadata. `.zattrs` also
                       carries a `decant` marker key: {"pixels": "0", …}.
.decant/manifest.json  the reconstruction manifest (§4b). A store key — not a Zarr
                       node. Inside a .zarr.zip it is a zip entry of that name.

Verbatim (mode V): replaces 0 + husk with a single original/ array — a 1-D uint8 chunked array holding the whole original file, blosc-zstd. Byte-exact but not natively browsable (no multiscales). Reconstruction is a plain concatenation of that array.

Series (mode "series"), consolidated: one shared 0 holds every member's planes in global order; each member keeps its own husk husk_0, husk_1, …. The manifest's members[] records each member's page range and husk array.

Series (mode "series"), nested: a directory NAME.zarr/ containing one per-member bundle m0.zarr.zip, m1.zarr.zip, … plus a .decant/manifest.json sidecar file. Each member bundle is itself a mode-D or mode-V archive. (This is the only archive form that is not a single file.)

4. The husk

The husk is the original file with every pixel byte-range removed, concatenated in ascending file-offset order. Equivalently, it is the complement of the union of the pixel regions [offset, offset+count) over [0, original_size). Everything that is not decoded pixel data lives here verbatim: the TIFF header, all IFDs and tag values, inter-strip padding, tile edge-padding, and any gaps. Because edge padding rides in the husk, tiled TIFFs reconstruct byte-exact regardless of padding content.

husk_raw_size = original_size − Σ databytecounts.

4b. Manifest schema (.decant/manifest.json)

A single JSON object.

field type meaning
decant_version string tool version that wrote the archive (informational)
format int schema version = 1. A reader MUST refuse unknown values.
mode string "byte-exact" | "verbatim" | "series"
original_name string basename of the original file (mode series: the archive basename — authoritative per-member values are in members[])
original_size int original file size in bytes (mode series: 0; see members[])
original_sha256 string sha256 of the original file, the correctness check (mode series: ""; see members[])
byteorder string "<" or ">" — pixel byte order for re-serialization
pixel_codec string codec id, e.g. "blosc:zstd:byteshuffle:clevel=5" (informational)
pages list per-page layout records (mode D). See below.
plane_order list storage-order names of 0's leading axes, e.g. ["z","c"]. 0 is stored in canonical NGFF order; this inverts the permutation in §5. Empty is a reader-side back-compat case only (⇒ plain unravel, §5); current writers always populate it (≥ ["z"]). Consolidated series: describes the shared 0.
husk_raw_size int uncompressed husk length in bytes
notes string free text
members list per-member records (mode "series"). See below.
series_consolidated bool true = shared 0 + per-member husks; false = nested

Page record (pages[], and members[].pages[]):

field type meaning
index int this page's position in storage (page-enumeration) order; mapped to a canonical array coordinate via plane_order (see §5)
shape list decoded page shape: [Y, X] or [Y, X, S]
dtype string NumPy dtype name (e.g. "uint16")
dataoffsets list[int] file offsets of the page's real-pixel byte segments
databytecounts list[int] byte length of each segment (parallel to dataoffsets)
samplesperpixel int S
planarconfig int 1 = chunky/interleaved, 2 = planar/separate
predictor int 1 = none (only 1 is emitted; others route to verbatim)
rowsperstrip int striped geometry (0 when tiled)
tilewidth, tilelength int tiled geometry (0 when striped)

For tiled pages, dataoffsets/databytecounts list only the real-pixel segments (edge padding excluded — it is in the husk), in the order §5 serializes them.

Member record (members[]):

field type meaning
relpath string path of this member relative to the series base dir
original_size, original_sha256, byteorder — as above, per member
husk_array string consolidated: name of this member's husk array ("husk_0", …)
husk_raw_size int this member's husk length
pages list consolidated: page records whose index is the global index into the shared 0
subarchive string nested: name of this member's own bundle ("m0.zarr.zip")

5. Reconstruction (mode D)

Inputs: the pixel array 0, the husk (read sequentially), and a page list (pages, or a member's pages).

Page index → array coordinate. 0's leading axes are stored in canonical NGFF order (time → channel/custom → space); pages enumerate in the file's storage order, recorded as plane_order (the leading axis names in that order). Let page_ndim = len(page.shape), leading = 0.shape[:0.ndim − page_ndim], and perm = the indices that stable-sort plane_order into canonical order (so canonical[j] = plane_order[perm[j]]). Then

storage_shape = [ leading[perm.index(i)] for i in range(len(perm)) ]
def read_page(i):
    s = unravel_index(i, storage_shape)                 # C-order, storage axes
    return 0[ (s[perm[0]], s[perm[1]], …, s[perm[-1]]) ]  # canonical coord

When plane_order is empty or already canonical this reduces to 0[ unravel_index(i, leading) ], and a flat stack to 0[i]. The permutation changes only which leading coordinate a page occupies — pixel bytes are identical — so byte-exactness is independent of it (guarded per-archive by the pack-side reconstruct-and-verify gate). plane_order is load-bearing for a permuted store: a reader that ignores it mis-orders a non-canonical file.

Re-serialize a page's on-disk pixel bytes (serialize_page), using the file byte order and the page's layout. Let a = read_page(i) normalized to (Y, X, S) in the file dtype:

  • striped, chunky (planarconfig=1): a.tobytes() (row-major, interleaved samples).
  • striped, planar (planarconfig=2): concatenate each sample plane's bytes, a[:,:,s].tobytes() for s in 0…S-1, in sample order.
  • tiled: iterate tiles sample-major (if planar) then grid row-major. For each tile emit only its real pixels: a full-width tile's rows are contiguous (ys rows × row-stride); a right-edge tile emits each real row separately (column padding excluded). Tile counts come from ceil(X/tilewidth) × ceil(Y/tilelength).

The result is the page's pixel bytes with the same total length as Σ databytecounts for that page.

Splice pixels into the husk. Build a region list and walk it in offset order:

regions = []
for pm in pages:
    pos = 0
    for off, cnt in zip(pm.dataoffsets, pm.databytecounts):
        regions.append((off, cnt, pm.index, pos)); pos += cnt
regions.sort()                      # ascending file offset

cur = 0; hp = 0; out = []
for off, cnt, pidx, ppos in regions:
    if off > cur:                   # a non-pixel gap → pull from the husk
        out.append(husk[hp : hp + (off - cur)]); hp += off - cur; cur = off
    out.append(serialize_page(read_page(pidx), …)[ppos : ppos + cnt]); cur = off + cnt
if cur < original_size:             # trailing husk (e.g. metadata after pixels)
    out.append(husk[hp : hp + (original_size - cur)])
original = b"".join(out)
assert sha256(original) == original_sha256

Streaming note: pages and husk are consumed in offset order, so the format is streamable in bounded memory — a conformant reader holds at most one page + one husk block regardless of file size.

Verbatim / series. Verbatim: concatenate the original/ array and check the hash. Series: reconstruct each member independently — consolidated members via the walk above (reading 0 at their global page indices and their own husk array); nested members by opening their sub-bundle.

6. Versioning

format is the schema version (currently 1). Additive, reader-ignorable keys (e.g. pyramid level arrays, extra .zattrs) do not bump it. Any change to the manifest schema or to the reconstruction algorithm in §5 does. A reader MUST refuse a format it does not implement rather than guess.

The plane_order key + canonical layout were folded into format 1 (not a bump) because decant is pre-release and no format-1 archives predate them: there is no older format-1 definition to diverge from. The §5 algorithm above is a strict generalization — it reproduces the previous plain-unravel exactly when plane_order is empty — so it also reads any archive written before the key existed.

Compatibility policy. The reader's supported-format set only ever grows: a newer decant reads every archive an older decant could (restore.SUPPORTED_FORMATS ⊇ every shipped format), and the read path for an older format is never removed. An older decant, conversely, refuses an archive whose format it predates rather than risk a wrong reconstruction. And because every archive is recoverable from this spec alone (§7), a lost or too-old tool is never fatal — the bytes outlive the program.

7. Independent recovery

Nothing above requires decant. A ~50-line reader — open the Zarr group, load .decant/manifest.json, run the §5 walk — recovers every original byte-for-byte using only zarr, numcodecs, and numpy. The original_sha256 in the manifest is the built-in proof that the recovery is exact.

For interactive viewing (as opposed to byte-exact recovery), the pixel side is a standard OME-Zarr group any NGFF reader opens — for example the decant-fiji plugin opens .zarr.zip archives directly in Fiji.