casblob-convert: offline .v1 to zstd casblob converter - #45
Draft
shreyas-blacksmith wants to merge 3 commits into
Draft
shreyas-blacksmith wants to merge 3 commits into
shreyas-blacksmith wants to merge 3 commits into
Conversation
Converts legacy uncompressed CAS blobs to zstd casblobs in place, one object at a time, so zstd-mode nodes stop paying compress-on-read for pre-existing hot blobs. Runs only while bazel-remote is stopped (the in-memory index pins exact filenames); the next startup scan indexes converted entries natively. Idempotent and resumable; verifies content hashes during conversion and deletes corrupt blobs (removal lets the client re-upload and heal); preserves atimes so LRU order survives the rescan; skips zero-byte blobs (casblob cannot represent them). Co-authored-by: Cursor <cursoragent@cursor.com>
A quiesced node has no reason to convert gently: with -workers the tool saturates the NVMe instead of trickling. Measured on a real 9.2M-file store: 1.16 GB/s raw (~5,900 objects/s) at -workers 12, converting a full 1.4TB node in ~25 minutes. Co-authored-by: Cursor <cursoragent@cursor.com>
…aming fixes Staging e2e validation surfaced that the stop-the-world conversion window is avoidable entirely, and caught two bugs: - New -shadow mode converts .v1 files into a parallel tree while bazel-remote keeps serving (the server's index pins live filenames and never rescans, so an outside tree is invisible to it). New -swap mode then consumes the shadow tree under a brief per-node stop: pure renames (measured ~8,500/s), discarding entries the server evicted since the shadow write and refreshing atimes from the source so overnight recency survives into LRU order. Rollout becomes rolling per node with the ring covering each window - no fleet-wide off period. - New -min-free-gb guard stops shadow dispatch when the filesystem's free space drops below a floor, so a space-tight node (e.g. 92% full) converts in bounded shadow/swap cycles: the shadow tree is transient, so each swap returns free space to baseline and the next cycle covers the next slice. - Fix: converted casblobs now copy the source's owner and mode. The converter runs as root while the server runs as its own user; CreateTemp's root:0600 output was unreadable by the service. - Fix: casblob names now carry the logical size (<hash>-<size>-<random>, per FileLocation). Legacy .v1 names have no size field, and naively stripping the suffix made the startup scan index converted entries with their on-disk (compressed) size, so size-checked requests would spuriously miss. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion tool for the fleet-wide
storage_mode: zstdflip (FastActions/fa#5588). Replaces the closed runtime-fallback approach (#44): MinIO as the L1 backing source is being retired, so instead of hydrating from the v1 keyspace at serve time, each node's existing local blobs are converted offline, once.Modes
-shadow DIR(server may keep serving): converts legacy.v1blobs into a parallel tree outside the data dir. The server's in-memory index pins live filenames and never rescans, so the shadow tree is invisible to it. Corrupt sources are logged and left alone (the live server owns its tree); sources evicted mid-run are skipped. A-min-free-gbfloor stops dispatch before the shadow tree can squeeze the live cache.-swap DIR(server stopped, seconds-to-minutes): renames shadow casblobs over their.v1sources — pure metadata ops, measured ~8,500 renames/s. Shadow entries whose source was evicted since the shadow write are discarded (never resurrect), and atimes are refreshed from the source at swap time so recency accrued during the shadow window survives into LRU order..v1s are deleted (miss + client re-upload heals).Per conversion: content sha256 verified by
casblob.WriteAndClose, owner/mode copied from the source (converter runs as root, server does not), atime/mtime preserved, atomic rename. Casblob names carry the logical size (<hash>-<size>-<random>, matchingFileLocation); legacy names have no size field. Zero-byte blobs stay legacy (casblob cannot represent them). Every mode is idempotent and resumable; mixed stores are natively supported so every stopping point is safe.Fleet rollout (zero customer-visible outage)
-shadowruns on every serving L1 (throttled workers).-swap→ start withstorage_mode: zstd(fa#5588). The ring's health checks route around each node during its 2–5 minute window.Validation (staging-1, real 1.8T disk at 81%, 9.2M-file store)
Accept-Encoding: zstdstreaming stored bytes — the path that eliminates the recompression tax).bazel-remote:664, copied from source).