fix(server): commit-keyed rev refs, gix index-pack fallback, archive reuse guard#68
Open
russellromney wants to merge 7 commits into
Open
fix(server): commit-keyed rev refs, gix index-pack fallback, archive reuse guard#68russellromney wants to merge 7 commits into
russellromney wants to merge 7 commits into
Conversation
Add benchmark/run_shaped_sweep.sh and benchmark/fly_shaped_benchmark.sh to measure ripclone vs native git clone across shaped network links (50-1000 Mbps) or unshaped warm-cache runs (SHAPED=0). Include benchmark/plot_ratios.py and the generated benchmark/shaped_ratios.png. Update README.md and docs/BENCHMARKS.md with the resulting tables for oven-sh/bun, pandas-dev/pandas, and torvalds/linux (1000 Mbps only), and rewrite benchmark/README.md around the new canonical harness. Delete superseded scripts: - benchmark/baseline.sh - benchmark/remote.sh - scripts/benchmark_clone_compare.sh - scripts/benchmark_fly_client.sh Fix files mode to request the full clonepack (archive chunks) in rust/src/bin/cli.rs. Update tests/fly/Dockerfile.client to install the traffic-shaping tools and copy the harness scripts; keep the dev Fly machines warm during long sweeps in tests/fly/fly.client-dev.toml and tests/fly/fly.server-dev.toml.
For very active repos like pandas-dev/pandas the default branch can advance while the sweep is waiting for full-history artifacts. Add BENCH_REF support to run_shaped_sweep.sh / fly_shaped_benchmark.sh so a tag or commit can be synced and cloned instead of the moving default branch. Update benchmark/README.md with usage notes.
- Apply the files-mode clonepack fix to rust/src/bin/cli.rs using the main API (mode.needs_archive). - Fix tests/fly/*.toml dockerfile paths so works with current flyctl (paths are relative to the config file). - Add missing ripclone-worker stub to tests/fly/Dockerfile.client.
russellromney
changed the title
…reuse guard - Use commit-keyed ref-store keys for rev-targeted builds so stale rev-keyed refs don't block future syncs and same-commit revs share a build. - Implement commit-keyed load_build for FileRefStore and S3RefStore. - Don't reuse a completed build that is missing archive chunks unless archive generation is still in progress. - Fall back to git index-pack when gix fails on ref-delta packs (oven-sh/bun). chore(benchmark/docs): prefer RIPCLONE_SERVER_TOKEN and fix header - shaped harness prints resolved commit in header. - benchmark scripts and docs updated to RIPCLONE_SERVER_TOKEN.
russellromney
force-pushed
the
benchmark/shaped-bandwidth
branch
from
e6b1e40 to
a79e0bd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
sync --at <rev>/sync?rev=<rev>now store artifacts under{branch}#{commit}so stale/incomplete rev-keyed refs don't block future syncs and different revs that resolve to the same commit share a build.RefStore::load_buildforFileRefStoreandS3RefStoreso a sync of one branch can reuse a completed build of another branch at the same commit.reuse_existing_buildno longer reuses a completed build that lacks archive chunks (unless archive generation is still in progress), preventing files-mode clones from polling forever.oven-sh/bun), fall back to stockgit index-pack.RIPCLONE_SERVER_TOKEN(falling back to deprecatedRIPCLONE_TOKEN).Verified
cargo test --libserver/ref_store/git tests pass.ripclone-server-devandripclone-client-dev.pandas-dev/pandas@v2.2.2andoven-sh/bun@df92f8fd...both complete all 5 shaped modes @ 1000 Mbps.