Add Zstd "in-place" encoding plugin - #31
Open
gsmecher wants to merge 4 commits into
Open
Conversation
gsmecher
force-pushed
the
zstd
branch
2 times, most recently
from
July 30, 2026 00:00
f4a724d to
4f986a6
Compare
Contributor
Author
|
(Updated to abstract most tests out of copy/paste duplicates of existing tests to the enc_* template framework. There are a few exceptions.) |
Contributor
Author
|
(adherence to c89 coding style - no mid-body variable declarations, plus a missing use_encdata when writing out zstd field definitions in a fragment) |
This allows each encoding's open() calls to parse these configuration options and embed them within the gd_raw_file struct for use later on. These structs are freed by the associated close(). Because other encoding API calls are bracketed by open()/close() pairs, memory ownership is clear. The exception is the encoding.name() call, which occasionally happens with an unopened field (and hence already receives the encoding string as a separate argument.)
…sent Because seek()+read() and seek()+write() are common pairings, adding a pread/pwrite alternative allows us to cut down on syscall volume.
This is an "in-place" encoding scheme - we are primarily interested in supporting efficient streaming writes of compressed data (presently only supported by RAW and SIE).
Contributor
Author
|
This latest push
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On-line compression (that is, concurrent, reader-visible updates to a compressed dirfile) is a useful and unserved operating point for getdata: network bandwidth is cheap, the SSDs needed to sustain these rates aren't bottomless, and clients like kst rely on low-latency refresh. However, current compression plug-ins are "out-of-place", meaning data isn't visible to readers until the writer has flushed or closed the dirfile -- this doesn't help for interactive uses.
Zstd is an in-place plug-in that compresses in configurable blocks and maintains the existing "read-while-writing" behaviour that RAW fields give. Zstd has a few technical attributes (decompressed size in frame header at the beginning of the frame; skippable metadata frames; frame scans without decompression; checksums) that make it particularly well-suited to this scheme.
The main design quirk in this implementation is the maintenance of an in-memory "seek table" that's built up with a linear frame scan when a file is opened. This frame scan takes ~seconds on multi-GB compressed fields but does grow linearly. It could be improved in a future revision with a journal-type scheme. Another alternative is a "chunking" scheme at the getdata (not encoding) layer that spreads fields across multiple files; this would address similar limitations for other plug-ins as well. Both of these approaches are out-of-scope for now.
Also included are