fturader

RF-DETR + SAM2 pipeline for FTU (Functional Tissue Unit) instance segmentation in histopathology images — H&E brightfield and multiplex fluorescence (CODEX / OME-TIFF) via a Beer-Lambert pseudo-H&E front end.

One call turns a tissue image into an (H, W) integer FTU label map — segment_ftu(image, channel_names, tissue_type). Install → drop in the weights → run.

• New here? Read Quick start and run a getting-started notebook.
• Deep Python API — loader decision tree, the low-level FTURader model, multiplex loaders, pseudo-H&E recipes, and the full support matrix: docs/python_api.md.
• CLI reference: docs/cli.md · Pseudo-H&E internals: docs/multiplex_pseudo_he.md.

---

Requirements

Requirement	Version
Python	3.10
NVIDIA GPU + CUDA	≥ 11.8
torch	≥ 2.5.1
rfdetr	≥ 1.6.5 (validated on 1.8.0)
SAM-2	install from GitHub (pulled in automatically as sam-2@git+…)
supervision	≥ 0.26
tifffile	≥ 2024.1

The DINOv2 patch-size / positional-encoding warnings printed when RF-DETR loads a checkpoint are benign and version-independent — the FTU checkpoints were trained at patch_size=16 / resolution=1024 (not stock DINOv2's 14/518), so RF-DETR (any version) reports it is not loading stock backbone weights. The full RF-DETR weights load fine.

---

Installation

From source (editable):

pip install -e .[dev]      # [dev] adds pytest, jupyter; omit for runtime only

This pulls torch, rfdetr, supervision, tifffile, and installs SAM-2 from GitHub (sam-2@git+https://github.com/facebookresearch/sam2.git), so the install host needs network access to GitHub and a working C/CUDA toolchain for SAM-2's extensions. The optional [zarr] extra (pip install -e .[zarr]) adds lazy windowed reads for very large arrays.

Pre-trained weights

Running inference requires pre-trained model weights under weights_root (DEFAULT_WEIGHTS_ROOT = ~/.deepcell/models/ftusam_v0.1/, also the CLI's default --weights-root). The directory must contain per-organ RFDETRLarge_resolution_1024_<organ>.pth (kidney / largeintestine / lung / prostate / spleen) and a shared sam2.1_hiera_b+_epoch_300.pth. Two ways to get them, pick one:

1. Place local weights (no token). If you already hold the .pth files, drop them into ~/.deepcell/models/ftusam_v0.1/. segment_ftu and the CLI load from there by default — no network, no token.

2. Download from Deepcell (needs a token). Create an access token at users.deepcell.org, then set it as an environment variable in your shell (not in a notebook or source) and download:

export DEEPCELL_ACCESS_TOKEN=<your-token>
python -c "import fturader; fturader.download_model_weights()"   # → ~/.deepcell/models/ftusam_v0.1/

download_model_weights() fetches and unpacks the archive into ~/.deepcell/models/ftusam_v0.1/. Authentication goes through the DEEPCELL_ACCESS_TOKEN environment variable (_auth.py); a missing token raises a clear ValueError pointing back to users.deepcell.org.

Security: never commit a token or write it into a notebook cell / source file. Use a shell environment variable (or a secrets manager). The download path has not been smoke- tested here because no token was available; the local-weights path is the one exercised.

Smoke test (no GPU / no weights)

Verify the install can import and that the CLI parses, without a GPU or weights:

python -c "import fturader; from fturader.multiplex import SingleImageDataset, infer_multiplex; print('import OK')"
fturader --help
pytest -q -m unit          # fast, no GPU, no external checkpoints

A weight-free end-to-end dry run (channel classification only, no model) on any multichannel OME-TIFF:

fturader --input stitched.ome.tif --print-classification

---

Supported Organs (defaults)

Organ	min_mask_area (px²)	score_threshold
prostate	40	0.33
largeintestine	200	0.36
lung	1 000	0.25
kidney	5 000	0.25
spleen	10 000	0.38

Lung default raised in R3 (score_threshold 0.13 → 0.25; min_mask_area 1 000). The old 0.13 produced 400+ detections per crop with slow NMM merge; 0.25 gives cleaner, faster output (a 6000²-px crop @ 0.25 ≈ 219 FTUs / 41 s) with no hand-set threshold.

Tuning for fewer, cleaner detections: raise score_threshold and/or min_mask_area. CLI: --score-threshold / --min-mask-area (0 = organ default). Python: segment_ftu(..., score_threshold=…, min_mask_area=…) or any infer_wsi kwarg.

---

Quick start — segment_ftu

The shortest path to a result. Hand us a multichannel image as (C, H, W), the channel names, and the tissue type; get back an (H, W) integer FTU label map. This is the most general entry point — recipe="auto" chooses the pseudo-H&E channels per panel, and the model always runs the large-image tiling engine, so it works on the big mosaics (thousands of px on a side) that this pipeline targets.

import numpy as np
from fturader import segment_ftu

# A multiplex fluorescence mosaic: C channels, then height, width.
image = ...                          # np.ndarray, shape (C, H, W); C is unrestricted
channel_names = ["Hoechst1", "Cytokeratin", "Vimentin", "CollIV", ...]   # len == C

labels = segment_ftu(
    image,
    channel_names,
    tissue_type="largeintestine",    # one of SUPPORTED_ORGANS (selects the RF-DETR weights)
    recipe="auto",                   # auto-pick channels (recipes: docs/python_api.md)
    pixel_size_um=0.377,             # µm/px — pass it; detections are scaled by it
)
# labels: (H, W) int, 0 = background, 1..N = FTU instances

segment_ftu packages from_image → pseudo-H&E synthesis → tissue_type → weights → infer_wsi → rasterize labels. It needs pre-trained weights present under weights_root (see Pre-trained weights).

Useful options (full reference in src/fturader/api.py docstring):

Argument	Meaning
mask=	optional (H, W) boolean ROI; output labels are intersected with it (outside → 0). None = whole image.
recipe=	"auto" (default) / "auto:mean" / "auto:max" / "ck" / an eosin marker list e.g. ["Cytokeratin", "Vimentin"] / a PseudoHEConfig. "v1" / "v2" are deprecated aliases.
he=	None (default) synthesizes a named fluorescence panel; True passes an already-H&E (H, W, 3) image through unchanged. Channel count never decides H&E — he does.
pixel_size_um=	µm/px; None falls back to 0.4 with a synthesis warning.
weights_root=	weights directory; default DEFAULT_WEIGHTS_ROOT = ~/.deepcell/models/ftusam_v0.1.
return_report=True	also return a machine-readable report dict {"load": …, "synthesis": …, "infer": …} (loader facts + structured synthesis config + inference plan with n_detections).
**infer_kw	forwarded to infer_wsi (e.g. score_threshold, overlap_ratio); the organ default score_threshold is used otherwise.

Client notes (read before running).

• Working size: best at roughly 3000–10000 px on the long side. Far smaller crops give the tiler too little context; far larger only costs time.
• auto can pick poorly on unusual panels. If the pseudo-H&E looks washed out, inspect describe() (via return_report=True) and specify channels manually — pass an eosin marker list, e.g. recipe=["Cytokeratin", "Vimentin"], or recipe="ck" for an epithelial organ that carries Cytokeratin.
• Resource cost (U5–U9 profiling): VRAM ~1.4 GB; host RAM peak scales with area × channel count, roughly 16–35 GB for a ~10000²-px multi-channel mosaic; wall time scales with the number of detected FTUs.

For finer control over loading (separate tile files, in-memory arrays, block-wise focus) or for the infer_image / infer_wsi model objects directly, see docs/python_api.md.

Transparency: report & profiling

Three ways to see what a run will (or did) do — pick by what you want:

want	call	shape
human-readable summary of what synthesis will do	dataset.describe()	concise string (one load line + one synthesis line); describe(verbose=True) adds the full channel list, normalization, Beer-Lambert and best-focus z
machine-readable transparency bundle	segment_ftu(..., return_report=True)	one dict {"load": …, "synthesis": …, "infer": …}
just the tiling/route plan (no model run beyond geometry)	model.infer_plan(image_hw, pixel_size_um)	the infer section standalone (no pixels read, no inference run)

dataset.describe() covers the load + synthesis stages only; the infer stage (tiling, route, threshold, detection count) needs the model, so it lives in return_report=True / infer_plan. (The structured dict behind describe() is the private _describe_synthesis, exposed programmatically as report["synthesis"].)

Profiling — where wall time goes (and the NMM bottleneck). Pass profile=True to record per-stage timings:

model.infer_wsi(rgb, pixel_size_um=0.377, profile=True)
prof = model.last_profile                               # dict of per-stage seconds
# or, via the one-call entry:
labels, report = segment_ftu(..., profile=True, return_report=True)
prof = report["infer"]["profile"]

The CLI writes the same block as profile into the bundle summary.json. Keys: scale_s, n_tiles, tile_forward_s (= RF-DETR + SAM2 tile inference), assemble_s (cross-tile concat), merge_s (the NMM/NMS dedup — timed directly by wrapping sv.Detections.with_nmm), slicer_overhead_s, rescale_s, total_s, n_detections.

Empirical conclusion (R3 runs): tile_forward_s (inference) stays bounded at ~2–19 s across samples, while merge_s (the NMM merge) scales with detection count and dominates wall time at high counts:

sample	n_detections	merge_s
HBM573	6	0.02 s
CODEX (large intestine)	~26	~1.0 s
lung 6000²-px @ 0.25	~219	~27 s
HBM288 full	247	99 s (≈80 % of a 124 s run)

8× more detections (26 → 209) cost ~46× the merge time, with inference roughly unchanged. This is why raising score_threshold / min_mask_area makes runs both cleaner and faster.

---

Notebooks

Notebooks come in two tiers — start minimal, then go comprehensive:

Tier	Python	CLI
Getting started (minimal — one call, copy-paste)	notebooks/getting_started_python.ipynb — a single segment_ftu call	notebooks/getting_started_cli.ipynb — one fturader command, with the 3 key params (--pixel-size / --channel-names / --he) explained up front
Comprehensive (full tour)	notebooks/walkthrough_python.ipynb — segment_ftu, H&E + CODEX + OME, recipe comparison, bundle I/O	notebooks/walkthrough_cli.ipynb — the CLI equivalents

Read a getting-started notebook first for the shortest working path, then move to the matching comprehensive guide for the full feature tour. The kidney example was removed from the notebooks (kidney remains a fully supported organ in the API and the defaults table). Deprecated notebooks are kept under notebooks/legacy/.

The getting-started notebooks run on a small bundled crop with no download. The walkthrough notebooks are best read as a reference tour — every cell runs on the full raw HuBMAP slides, so to actually re-execute them you must first download the datasets they list (HuBMAP Data Portal links are in each walkthrough).

---

CLI

The unified fturader command handles both H&E brightfield and multiplex fluorescence (CODEX / OME-TIFF) inputs through a single --input. It always runs infer_wsi internally (tiling kicks in for large inputs).

Deprecated aliases kept for backward compatibility: fturader-infer (H&E only) and fturader-multiplex (multiplex only) share the same logic as fturader and will be removed in a future release.

Full parameter reference: docs/cli.md.

Quick examples

# H&E brightfield (kidney)
fturader \
    --input  /path/to/image.ome.tiff \
    --he true \
    --organ  kidney \
    --output-dir ./out/ \
    --save-overlay

# Multiplex CODEX directory (large intestine; manual eosin channels, or recipe=ck)
fturader \
    --input  /path/to/processed/ \
    --channel-names /path/to/extras/channelnames.txt \
    --pixel-size 0.377 \
    --recipe Cytokeratin,Vimentin \
    --organ  largeintestine \
    --output-dir ./out/

# Already-stitched OME-TIFF (channel names + pixel size read from OME-XML)
fturader \
    --input  stitched.ome.tif \
    --recipe auto \
    --organ  largeintestine \
    --output-dir ./out/

# Dry run: print channel classification only (no GPU / no weights)
fturader --input stitched.ome.tif --print-classification

Output bundle

File	Description
labels.tif	(H, W) uint16/uint32 instance map (zlib-compressed); 0=background, 1..N; high-confidence on top
detections.npz	compact-RLE per-instance masks + xyxy / confidence / rf_score / sam_score
summary.json	organ, pixel_size_um, n_detections, overlap_strategy, checkpoint paths, pseudo_he_report (effective pseudo-H&E config), …
overlay_full.jpg	exp4-style turbo filled-mask overlay (--save-overlay)
overlay_2000.jpg	same, short-side-2000px JPEG (--save-overlay)

Reload in Python:

from fturader.io import load_detections
dets, meta = load_detections("./out/")   # sv.Detections + metadata dict

---

Results

Instance segmentation (COCO AP / F1@)

Segmentation quality (pixel-level Dice & IoU)