Skip to content

Post step: stop writing the multi-pack-index during mirror maintenance - #57

Open
piob-io wants to merge 1 commit into
mainfrom
devin/1789773117-maintenance-no-midx
Open

piob-io wants to merge 1 commit into
mainfrom
devin/1789773117-maintenance-no-midx

Conversation

@piob-io

@piob-io piob-io commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Since v1.8.0 the post step runs git repack -d -l -n --geometric=2 --write-midx on every job. On large mirrors (a few GiB, millions of objects) the fleet shows a bimodal post step: most runs finish maintenance in well under a second, but a recurring ~25% take 6–16 s inside that repack command with the same inputs (same two kept packs, nothing deferred, one small incoming pack). The roll-up itself is tiny in those runs; what scales with the mirror is --write-midx: git rewrites the multi-pack-index whole each time, reading every pack's .idx (tens of MiB per kept pack) and writing about as many bytes again, and on a freshly mounted sticky disk those reads are cold.

Local measurement on a 2.8M-object / 1.3 GiB two-kept-pack fixture with one small incoming pack, page cache dropped before each run: --geometric=2 --write-midx 0.55–0.59 s, --geometric=2 alone ~0.01 s; same ratio on a 1M-object fixture. The end-to-end confirmation on a real staging sticky disk is the steady-state legs added to the checkout-mirror-maintenance-e2e workflow in test-workflows (baseline main vs this branch on the same seeded mirror).

Change (runMirrorMaintenance):

-  repackArgs = ['-a', '-d', '-l', '-n', '--write-midx']        // reclaim
+  repackArgs = ['-a', '-d', '-l', '-n']
-  repackArgs = ['-d', '-l', '-n', '--geometric=2', '--write-midx']  // incremental
+  repackArgs = ['-d', '-l', '-n', '--geometric=2']
+  if (!(await removeMultiPackIndex(mirrorPath))) return fail(false, ...)  // before the repack

Lookups across the handful of packs the geometric repack leaves (kept base packs + one or two roll-ups) are fast without a midx, and bitmaps were already off. The index a v1.8.0 run left on existing mirrors is removed up front rather than left to go stale; git repack -d would drop it itself on the first run that folds something, but not on a no-op run.

Everything else is unchanged: .keep selection and byte bound, the 14-day reclaim, the fail-open commit, stale-file cleanup.

Tests: existing real-git tests now assert no midx is present after incremental and reclaim runs; new test seeds a midx (as an earlier version would have) and checks it is removed and every pack it listed is still rolled up, fsck clean. cleanup test asserts --write-midx is not passed.

Link to Devin session: https://app.devin.ai/sessions/a3b42ef15226438faf3f9cdde9662918
Open in Devin Desktop: https://app.devin.ai/desktop/session/a3b42ef15226438faf3f9cdde9662918?variant=devin
Requested by: @piob-io


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Every --write-midx rewrites the index whole: a cold read of every pack's
.idx and a write of about the same size. On mirrors with millions of
objects that costs seconds per job on the sticky disk, more than the
roll-up itself, while lookups across the handful of packs the geometric
repack leaves are fast without it. Remove any index an earlier version
left behind instead of letting it go stale.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@piob-io
piob-io marked this pull request as ready for review September 18, 2026 23:37
@piob-io
piob-io requested a review from ajwerner September 18, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant