Read other compressed input formats (bz2 / zstd / xz): evaluate niffler #218

Dependency discussion per CONTRIBUTING.md, sub-issue of #201.

Goal

Accept compressed FASTQ input beyond gzip: bzip2, xz, zstd, without the user having to route it
through --readFilesCommand.

Where we are today

src/io/fastq.rs:88-110:

• compression is detected by file extension (.gz / .gzip), not by content
• the only decoder is gzip, via flate2::read::GzDecoder
• anything else has to go through --readFilesCommand (external process, as STAR does)

Two consequences: a .fq.zst is read as plain text and fails on garbage, and a gzip file with an
unusual name silently gets the same treatment. Related: the single-member gzip bug filed alongside
this issue.

Candidate

niffler 3.0.1, MIT/Apache-2.0, 532k downloads, released
2026-04-29.

• Detection by magic bytes, not extension: bytes2type peeks 5 bytes and matches 1f 8b
  (gzip), 42 5a (bzip2), 28 b5 2f fd (zstd), fd 37 7a 58 5a (xz), falling back to
  Format::No = pass through as plain.
• Gzip goes through flate2::read::MultiGzDecoder (src/basic/compression.rs:54), so adopting it
  fixes the multi-member truncation bug by construction.
• niffler::send::get_reader / from_path return Box<dyn Read + Send>, which drops straight into
  the existing Box<dyn BufRead + Send> in FastqReader::open via a BufReader::with_capacity.

The part that needs a decision: backends

niffler's default feature set is bgz + bz2 + gz + lzma + zstd, each with the upstream
crate's default features. That pulls in four more codecs, and for a crate published to crates.io
and built on five platforms including Windows, the backend of each is the whole question:

So the recommended shape if we adopt it is explicit, not default:

niffler = { version = "3", default-features = false, features = ["gz", "bz2"] }

and then a deliberate decision on zstd (C zstd via niffler, or ruzstd wired in ourselves) and on
xz (C only, in practice).

Alternative: do the sniff in-tree

The sniff is a 5-byte peek and a match; the decoders are one crate each. In-tree we could pick
ruzstd for pure-Rust zstd decode, which niffler cannot give us. Against that: niffler is 532k
downloads of battle-testing on exactly this problem, including the Send plumbing and the
pass-through case, and it is one dependency instead of three.

Reasonable split: take niffler for detection + gz + bz2, and treat zstd as its own decision.

STAR-compatibility note

STAR itself does not sniff; it expects --readFilesCommand (zcat, bunzip2, …). Auto-detection
is already a local convenience for gzip, so extending it is an extension of an existing divergence,
not a new one. Worth a line in DIVERGENCE.md and maintainer sign-off, since it changes what input
the tool silently accepts.

Checklist

• Decide the format list actually wanted: bz2 and zstd are the realistic ones, xz is rare
• Decide zstd backend: C zstd (via niffler) vs pure-Rust ruzstd (decode-only, in-tree)
• Confirm no C toolchain requirement is introduced on any of the five CI platforms
• Explicitly disable niffler's bgz feature; BGZF stays on noodles-bgzf
• Keep --readFilesCommand working and taking precedence over sniffing
• Detection must work on stdin/pipes too, or be documented as file-only
• Tests: one fixture per accepted format, plus a multi-member gzip case
• DIVERGENCE.md entry + sign-off for the auto-detection behaviour