Skip to content

fix(io): decode multi-member gzip input instead of truncating it - #219

Open
BenjaminDEMAILLE wants to merge 1 commit into
scverse:mainfrom
BenjaminDEMAILLE:fix/multigz-decoder
Open

fix(io): decode multi-member gzip input instead of truncating it#219
BenjaminDEMAILLE wants to merge 1 commit into
scverse:mainfrom
BenjaminDEMAILLE:fix/multigz-decoder

Conversation

@BenjaminDEMAILLE

Copy link
Copy Markdown
Contributor

Fixes #217.

The bug

flate2::read::GzDecoder decodes one gzip member and then reports EOF. Compressed input was
read through it, so a .gz made of concatenated members was silently truncated: no error, no
warning, just fewer reads. Multi-member gzip is not exotic here — it is what bcl2fastq writes,
what cat a.fq.gz b.fq.gz produces, and what every BGZF file is.

Measured on main with the bundled fixtures, where both files decompress to the same 12 lines
under gunzip:

Input Number of input reads
single.fq.gz (one member, 3 reads) 3
multi.fq.gz (two members, 3 reads) 2

The fix

GzDecoderMultiGzDecoder at the four read paths:

  • src/io/fastq.rs — FASTQ input
  • src/solo/whitelist.rs — barcode whitelist
  • src/solo/count.rs
  • src/bin/emptydrops.rs

No API change, no new dependency: the type ships in the flate2 version already in the tree.
MultiGzDecoder is a strict superset of GzDecoder for reading — single-member files decode
identically, which is why the existing test_fastq_reader_gzip still passes unchanged.

Test

test_fastq_reader_gzip_multi_member in src/io/fastq.rs writes two gzip members into one file and
asserts both reads come back. I checked it in both directions: it fails on the old decoder
(second gzip member must be decoded, not silently truncated) and passes on the new one.

Full local gate: cargo fmt --check clean, cargo clippy --all-targets -- -D warnings clean,
cargo test 631 passed / 0 failed.

Not in this PR

Compression is still detected by file extension (.gz / .gzip), so a compressed file with an
unexpected name is still read as plain text. Magic-byte sniffing and other input formats are #218.

`flate2::read::GzDecoder` stops at the end of the first gzip member and
reports EOF. A `.gz` written as several concatenated members was
therefore read partially, with no error and no warning: bcl2fastq
output, `cat a.fq.gz b.fq.gz > merged.fq.gz`, and every BGZF file are
all multi-member.

Measured on the bundled fixtures before the fix: a two-member
`test/reads.fq` gzip (3 reads, 12 lines, identical to the single-member
file under `gunzip`) reported `Number of input reads | 2`.

Switches the four read paths to `MultiGzDecoder`: FASTQ input, the solo
barcode whitelist, solo counting, and the `emptydrops` binary.

Adds `test_fastq_reader_gzip_multi_member`, which builds a two-member
gzip and asserts both reads come back. Verified it fails on the old
decoder and passes on the new one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GzDecoder truncates multi-member gzip input: reads are silently dropped

1 participant