Dependency discussion: optional rapidgzip feature for parallel gzip input

[BenjaminDEMAILLE]

Raising this before the PR, as CONTRIBUTING.md requires for a new dependency, especially a
non-Rust one.

Proposal

Add rapidgzip 1.2.1 (MIT OR Apache-2.0, released
2026-04-01) as an optional cargo feature, off by default, for parallel decoding of gzipped
--readFilesIn. It is a safe Rust wrapper over rapidgzip-sys, which vendors ~20 MB of C++
(rapidgzip + zlib + rpmalloc) and builds it with CMake.

What it costs

• A CMake and C++17 toolchain requirement for anyone building with the feature on. This is why
  it must be a feature and not a default dependency: the crate is published to crates.io and builds
  on five platforms including Windows.
• A second allocator (rpmalloc) inside a process that installs mimalloc globally.
• A young crate: first release 2026-03-25, ~2.3k downloads.

What it buys

Standalone, decoding a 73 MB level-6 .fq.gz (427 MB out) on 16 logical cores: flate2+zlib-rs
1300 MB/s on one thread, rapidgzip 548 MB/s on one, 1524 MB/s on four, 1705 MB/s on eight. Full
tables in #223.

End to end inside the aligner, 20 Mb genome, 2 M reads, --runThreadN 8, median of three:

decode threads	wall	user CPU	peak RSS
0 (flate2)	3.18s	25.3s	1242 MB
2	3.21s	25.7s	2069 MB
4	3.33s	26.2s	2100 MB

No wall-clock gain on this workload, and ~800 MB more resident for the decoder's chunk cache.
That is consistent with the profile: decode already runs on its own thread overlapped with
alignment, gzipped input costs only ~11% of wall, and the serial writer stage is ~34% (#223).

So why add it at all

For the input-bound shape this does not cover: fast alignment, slow single-stream decode, spare
cores. And to make that measurable rather than hypothetical. It stays off by default and inert
until RUSTAR_GZ_DECODE_THREADS is set, so nobody pays for it accidentally.

The zero-dependency alternative that already exists: --readFilesCommand "rapidgzip -dc" (or
igzip -dc, pigz -dc).

Question for the maintainer

Is an optional feature with a C++/CMake build path acceptable at all here, or would you rather this
stayed outside the tree and be handled through --readFilesCommand? PR is ready either way; say
the word and I will close it.

[BenjaminDEMAILLE]

Correction, and it changes the proposal.

The crate I evaluated above (rapidgzip on crates.io, by alekseizarubin) is an FFI wrapper whose -sys crate vendors ~20 MB of C++ and builds it with CMake. That is not the project I should have been looking at. rapidgzip-core 0.3 (BSD-3-Clause AND MIT, COMBINE-lab, rapidgzip-rust repo) is a pure-Rust implementation of the same marker/window algorithm. Its dependencies are bon, crossbeam-deque and libz-rs-sys, and that last one is zlib-rs, the same inflate backend flate2 already uses here.

So the main objection in this issue, a CMake and C++17 toolchain requirement on five platforms, does not apply. There is also no single-thread penalty: 1581 MB/s on one thread against 1552 MB/s for flate2, where the FFI crate managed 548 MB/s.

Updated numbers on the same 73 MB level-6 file (427 MB out), 16 logical cores, median of three:

decoder	throughput
flate2 + zlib-rs, 1 thread	1552 MB/s
rapidgzip-core, 1 thread	1581 MB/s
rapidgzip-core, 4 threads	1872 MB/s
rapidgzip-core, 8 threads	3356 MB/s
rapidgzip-core, 12 threads	4051 MB/s

What has not changed is whether it helps this aligner today. We consume decompressed FASTQ at ~140 MB/s (2 M reads in ~3.0 s at --runThreadN 8) against ~1550 MB/s from one flate2 thread, so decode sits at under a tenth of its capacity, and enabling it measured 3.15 s vs 3.03 s with ~360 MB more resident.

#225 is updated accordingly: pure-Rust dependency, still feature-gated and still inert until RUSTAR_GZ_DECODE_THREADS is set. The question for you is now much narrower, since the cost is one small pure-Rust dependency rather than a toolchain requirement.