Skip to content

GzDecoder truncates multi-member gzip input: reads are silently dropped #217

Description

@BenjaminDEMAILLE

Summary

Gzipped input is decoded with flate2::read::GzDecoder, which stops at the end of the first
gzip member
. A multi-member .gz is therefore read partially, silently: no error, no warning,
just fewer reads. flate2::read::MultiGzDecoder is the variant that consumes concatenated members.

This matters in practice because multi-member gzip is normal in this domain: bcl2fastq output,
any cat a.fq.gz b.fq.gz > merged.fq.gz, and every BGZF file (BGZF is multi-member gzip).

Reproduction

On main, using only the fixtures bundled in the repo:

# 3 reads total, split across two gzip members
head -8 test/reads.fq | gzip -c  > /tmp/multi.fq.gz
tail -4 test/reads.fq | gzip -c >> /tmp/multi.fq.gz
# same 3 reads, one member
gzip -c test/reads.fq > /tmp/single.fq.gz

cargo run -- --runMode genomeGenerate --genomeDir /tmp/idx \
    --genomeFastaFiles test/simple.fa --genomeSAindexNbases 4
for f in single multi; do
  cargo run -- --genomeDir /tmp/idx --readFilesIn /tmp/$f.fq.gz \
      --outFileNamePrefix /tmp/out_$f/ --outSAMtype BAM SortedByCoordinate
  grep 'Number of input reads' /tmp/out_$f/Log.final.out
done
Input gunzip -c | wc -l Number of input reads
single.fq.gz 12 3
multi.fq.gz 12 2

Both files decompress to the same 12 lines with gunzip/Python. The aligner sees one read fewer,
and exits 0.

Affected sites

All four read paths use the single-member decoder:

  • src/io/fastq.rs:106 — FASTQ input (the one that matters most)
  • src/solo/whitelist.rs:524 — barcode whitelist
  • src/solo/count.rs:2090
  • src/bin/emptydrops.rs:92

Fix

GzDecoderMultiGzDecoder at those four sites. No API change, no new dependency; the type is
already in the flate2 version in the tree.

Suggested regression test

An integration test that builds a two-member .fq.gz from the bundled fixture and asserts the read
count matches the plain-text run. Cheap, and it locks the behaviour against a future refactor
picking the wrong decoder again.

Note

Detection is also extension-based today (path_str.ends_with(".gz"), src/io/fastq.rs:92), so a
compressed file with an unexpected name is read as plain text and a plain file named .gz fails.
That is a separate question, tracked with the multi-format input issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions