feat(ipc): support writing LZ4 and ZSTD compressed IPC streams and files - #935
Conversation
… streams and files
…ionTypeFromString()
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #935 +/- ##
==========================================
+ Coverage 79.29% 79.49% +0.20%
==========================================
Files 106 106
Lines 16316 16564 +248
Branches 1925 1986 +61
==========================================
+ Hits 12937 13167 +230
- Misses 2163 2164 +1
- Partials 1216 1233 +17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ession error paths
paleolimbot
left a comment
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you!
A few conceptual questions (feel free to push back of they don't make sense!)...I'll follow up with a review focused on correctness with the help of the robots after
This needs a rebase or a merge from main (sorry!) from the initial work on Dictionary writes.
…support) into ipc-write-compression.
…return NULL from ToString
…D_BODY in schemas
…and harden the encoder
|
Thanks for the review @paleolimbot! Merged with the latest main and walked through your points. |
paleolimbot
left a comment
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you!
Codex turned up an unrelated issue surfaced here we can tackle separately ( #936 ).
| // don't leave queued work referring to our buffers behind | ||
| struct ArrowError ignored; | ||
| NANOARROW_UNUSED( | ||
| private->compressor.compress_wait(&private->compressor, -1, &ignored)); |
There was a problem hiding this comment.
tiny nit: You should be able to pass NULL for the error here.
3 participants
This PR adds LZ4 and ZSTD compression to nanoarrow’s IPC encoder and writer, allowing applications to write compressed Arrow streams and files. The motivation comes from the duckdb-arrow extension, where we want nanoarrow to handle compression while the extension takes care of the SQL options. To support this, callers can select a codec and compression level, query the supported level range, and convert between codec names and enum values without needing to include the compression libraries’ headers themselves.
Compression happens per buffer within each record batch, following the Arrow IPC format. If compressing a buffer does not make it smaller, we store it uncompressed, and empty buffers remain empty. The implementation provides a serial compressor by default, with an interface for applications to supply their own. Unsupported codecs and out-of-range levels are rejected when configuring the built-in compressor, so callers can report these errors before writing data. The added tests cover compression and decompression roundtrips, compressed streams and files, compression levels, and builds with either codec unavailable.