perf: reduce exporter memory footprint with disk staging and streaming generation - #5919
Open
another-rex wants to merge 13 commits into
Open
perf: reduce exporter memory footprint with disk staging and streaming generation#5919another-rex wants to merge 13 commits into
another-rex wants to merge 13 commits into
Conversation
another-rex
marked this pull request as draft
August 26, 2026 04:20
…om vulnMeta, and write CSV/Vanir in-memory
…oadThenProcessor, stream cleanups, and fast-fail on disk error
…ry helper functions
…linter in AGENTS.md
michaelkedar
left a comment
michaelkedar
left a comment
Member
There was a problem hiding this comment.
Seems okay.
I wonder how much slower this will be with the disk reading/writing?
Hopefully it doesn't push the exporter above 15 minutes...
|
|
||
| // csvEntry holds the unix timestamp in nanoseconds and the relative entry path. | ||
| type csvEntry struct { | ||
| modified int64 |
Member
There was a problem hiding this comment.
changing this to a time.Time (or the RFC string) will probably make logic around this neater
…remove file deletion in writer
another-rex
marked this pull request as ready for review
August 27, 2026 00:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rather than keeping everything in memory, then writing it out at the end in order. Save the records into a temporary location on disk in JSON, then io.copy them into a zip in order. Streaming the output to GCS.