fix(io): decode multi-member gzip input instead of truncating it - #219
Open
BenjaminDEMAILLE wants to merge 1 commit into
Open
fix(io): decode multi-member gzip input instead of truncating it#219BenjaminDEMAILLE wants to merge 1 commit into
BenjaminDEMAILLE wants to merge 1 commit into
Conversation
`flate2::read::GzDecoder` stops at the end of the first gzip member and reports EOF. A `.gz` written as several concatenated members was therefore read partially, with no error and no warning: bcl2fastq output, `cat a.fq.gz b.fq.gz > merged.fq.gz`, and every BGZF file are all multi-member. Measured on the bundled fixtures before the fix: a two-member `test/reads.fq` gzip (3 reads, 12 lines, identical to the single-member file under `gunzip`) reported `Number of input reads | 2`. Switches the four read paths to `MultiGzDecoder`: FASTQ input, the solo barcode whitelist, solo counting, and the `emptydrops` binary. Adds `test_fastq_reader_gzip_multi_member`, which builds a two-member gzip and asserts both reads come back. Verified it fails on the old decoder and passes on the new one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #217.
The bug
flate2::read::GzDecoderdecodes one gzip member and then reports EOF. Compressed input wasread through it, so a
.gzmade of concatenated members was silently truncated: no error, nowarning, just fewer reads. Multi-member gzip is not exotic here — it is what
bcl2fastqwrites,what
cat a.fq.gz b.fq.gzproduces, and what every BGZF file is.Measured on
mainwith the bundled fixtures, where both files decompress to the same 12 linesunder
gunzip:Number of input readssingle.fq.gz(one member, 3 reads)multi.fq.gz(two members, 3 reads)The fix
GzDecoder→MultiGzDecoderat the four read paths:src/io/fastq.rs— FASTQ inputsrc/solo/whitelist.rs— barcode whitelistsrc/solo/count.rssrc/bin/emptydrops.rsNo API change, no new dependency: the type ships in the
flate2version already in the tree.MultiGzDecoderis a strict superset ofGzDecoderfor reading — single-member files decodeidentically, which is why the existing
test_fastq_reader_gzipstill passes unchanged.Test
test_fastq_reader_gzip_multi_memberinsrc/io/fastq.rswrites two gzip members into one file andasserts both reads come back. I checked it in both directions: it fails on the old decoder
(
second gzip member must be decoded, not silently truncated) and passes on the new one.Full local gate:
cargo fmt --checkclean,cargo clippy --all-targets -- -D warningsclean,cargo test631 passed / 0 failed.Not in this PR
Compression is still detected by file extension (
.gz/.gzip), so a compressed file with anunexpected name is still read as plain text. Magic-byte sniffing and other input formats are #218.