Skip to content

Fetch without an object filter when the workspace shares the mirror - #52

Draft
piob-io wants to merge 3 commits into
mainfrom
devin/1788700715-mirror-drop-filter
Draft

piob-io wants to merge 3 commits into
mainfrom
devin/1788700715-mirror-drop-filter

Conversation

@piob-io

@piob-io piob-io commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

A filter: input (or the blob:none that sparse-checkout implies) is now dropped whenever the workspace verifiably shares the mirror's object store (sharesMirrorObjects, the same gate the shallow negotiation uses). Without a mirror the filter is passed through unchanged.

Why. With the mirror attached as an alternate, every object a filter would omit is already local, so the filter cannot save any transfer — but it does make the workspace a partial clone, and that is expensive with an alternate behind it. Since git 2.48, fetch-pack runs index-pack --promisor for a filtered fetch, and index-pack --promisor repacks every locally available, non-promisor object linked from the received pack into a new promisor pack (repack_local_linkspack-objects --exclude-promisor-objects-best-effort). The fetched commit's parent lives in the mirror, so that repack walks the mirror's history reachable from it and writes it into the workspace's .git/objects. There is no config to turn this off.

Reproduced on an 8.5 GB mirror with git 2.51 (trace2), --filter=blob:none --depth=2 of a merge commit whose parent is the mirror tip:

upload-pack + pack-objects (server):          0.30 s, 17 objects, 62 KB      <- the actual fetch
index-pack --promisor
  └ pack-objects --exclude-promisor-objects-best-effort: 100.9 s
      -> pack-c271….pack  3.7 GB, 7,260,634 objects, .promisor
same fetch without --filter:                  0.59 s total

In production this shows up as a flat ~60 s per filtered fetch that receives any objects, independent of depth or ref count (proportional to mirror size), plus several GB written to the runner's root disk per job; unfiltered fetches of the same shape take ~1 s. A filtered fetch that receives no objects (ref already in the mirror) is unaffected, which is why the cost is bimodal.

Change (git-source-provider.ts):

const sharesMirrorObjects = !!cacheInfo && await blacksmithCache.sharesMirrorObjects(repositoryPath, mirrorPath)
const filter = settings.filter ?? (settings.sparseCheckout ? 'blob:none' : undefined)
if (filter && sharesMirrorObjects) core.info(`[git-mirror] Fetching without --filter=${filter}: ...`)
else if (filter) fetchOptions.filter = filter

Side effect: fetch-depth: 0 + filter: now qualifies for the direct mirror ref copy (fetchRefsFromMirror, previously gated on !fetchOptions.filter), so it no longer does a +refs/heads/* network fetch either. The resulting workspace is a regular clone (no remote.origin.promisor / partialclonefilter); the only behavioural difference for later steps is that objects the mirror lacks are not lazily fetched — with a full mirror behind the alternates there are none, and dissociate: true is unaffected.

Tests

  • __test__/mirror-filter-fetch-git.test.ts (real git): unfiltered full and depth-2 fetches into an alternates workspace store ≤ 6 objects, write no .promisor, leave no partial-clone config, and check out the whole tree; a blob:none fetch into the same workspace does produce a promisor pack, and on git ≥ 2.48 its object count exceeds the mirror's history. Passes on git 2.34 and 2.51.
  • test-blacksmith.yml: fetch-depth: 0 + filter: blob:none and a sparse-checkout checkout on the real mirror; asserts no promisor packs, no partial-clone config, < 1000 local objects, fsck clean, sparse cone applied.
  • README / action.yml filter description updated.

Link to Devin session: https://app.devin.ai/sessions/a3b42ef15226438faf3f9cdde9662918
Open in Devin Desktop: https://app.devin.ai/desktop/session/a3b42ef15226438faf3f9cdde9662918?variant=devin
Requested by: @piob-io


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled. (Staging)

With the mirror attached as an alternate every object a filter would omit
is already local, so the filter saves nothing. It does make the workspace a
partial clone, and since git 2.48 index-pack --promisor repacks every
locally available object linked from the received pack into a promisor
pack: with a full mirror behind the alternates that copies the mirror's
reachable history into the workspace on every filtered fetch that receives
objects, costing time proportional to the mirror rather than to the delta.

Drop the filter (explicit or the blob:none that sparse-checkout implies)
whenever the workspace verifiably shares the mirror object store. As a side
effect fetch-depth 0 with a filter now takes the direct ref-copy path
instead of a full network fetch of all heads and tags.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

piob-io and others added 2 commits September 6, 2026 13:27
… mount

A later checkout step in the same job mounts its own copy of the sticky
disk over the same path, so objects that only the earlier mount's sync
added are no longer visible through the earlier workspace's alternates.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…directory

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant