Summary
Gzipped input is decoded with flate2::read::GzDecoder, which stops at the end of the first
gzip member. A multi-member .gz is therefore read partially, silently: no error, no warning,
just fewer reads. flate2::read::MultiGzDecoder is the variant that consumes concatenated members.
This matters in practice because multi-member gzip is normal in this domain: bcl2fastq output,
any cat a.fq.gz b.fq.gz > merged.fq.gz, and every BGZF file (BGZF is multi-member gzip).
Reproduction
On main, using only the fixtures bundled in the repo:
# 3 reads total, split across two gzip members
head -8 test/reads.fq | gzip -c > /tmp/multi.fq.gz
tail -4 test/reads.fq | gzip -c >> /tmp/multi.fq.gz
# same 3 reads, one member
gzip -c test/reads.fq > /tmp/single.fq.gz
cargo run -- --runMode genomeGenerate --genomeDir /tmp/idx \
--genomeFastaFiles test/simple.fa --genomeSAindexNbases 4
for f in single multi; do
cargo run -- --genomeDir /tmp/idx --readFilesIn /tmp/$f.fq.gz \
--outFileNamePrefix /tmp/out_$f/ --outSAMtype BAM SortedByCoordinate
grep 'Number of input reads' /tmp/out_$f/Log.final.out
done
| Input |
gunzip -c | wc -l |
Number of input reads |
single.fq.gz |
12 |
3 |
multi.fq.gz |
12 |
2 |
Both files decompress to the same 12 lines with gunzip/Python. The aligner sees one read fewer,
and exits 0.
Affected sites
All four read paths use the single-member decoder:
src/io/fastq.rs:106 — FASTQ input (the one that matters most)
src/solo/whitelist.rs:524 — barcode whitelist
src/solo/count.rs:2090
src/bin/emptydrops.rs:92
Fix
GzDecoder → MultiGzDecoder at those four sites. No API change, no new dependency; the type is
already in the flate2 version in the tree.
Suggested regression test
An integration test that builds a two-member .fq.gz from the bundled fixture and asserts the read
count matches the plain-text run. Cheap, and it locks the behaviour against a future refactor
picking the wrong decoder again.
Note
Detection is also extension-based today (path_str.ends_with(".gz"), src/io/fastq.rs:92), so a
compressed file with an unexpected name is read as plain text and a plain file named .gz fails.
That is a separate question, tracked with the multi-format input issue.
Summary
Gzipped input is decoded with
flate2::read::GzDecoder, which stops at the end of the firstgzip member. A multi-member
.gzis therefore read partially, silently: no error, no warning,just fewer reads.
flate2::read::MultiGzDecoderis the variant that consumes concatenated members.This matters in practice because multi-member gzip is normal in this domain:
bcl2fastqoutput,any
cat a.fq.gz b.fq.gz > merged.fq.gz, and every BGZF file (BGZF is multi-member gzip).Reproduction
On
main, using only the fixtures bundled in the repo:gunzip -c | wc -lNumber of input readssingle.fq.gzmulti.fq.gzBoth files decompress to the same 12 lines with
gunzip/Python. The aligner sees one read fewer,and exits 0.
Affected sites
All four read paths use the single-member decoder:
src/io/fastq.rs:106— FASTQ input (the one that matters most)src/solo/whitelist.rs:524— barcode whitelistsrc/solo/count.rs:2090src/bin/emptydrops.rs:92Fix
GzDecoder→MultiGzDecoderat those four sites. No API change, no new dependency; the type isalready in the
flate2version in the tree.Suggested regression test
An integration test that builds a two-member
.fq.gzfrom the bundled fixture and asserts the readcount matches the plain-text run. Cheap, and it locks the behaviour against a future refactor
picking the wrong decoder again.
Note
Detection is also extension-based today (
path_str.ends_with(".gz"),src/io/fastq.rs:92), so acompressed file with an unexpected name is read as plain text and a plain file named
.gzfails.That is a separate question, tracked with the multi-format input issue.