fix(download): residential saturation + transient failure hardening#95
Open
jacderida wants to merge 2 commits into
Open
fix(download): residential saturation + transient failure hardening#95jacderida wants to merge 2 commits into
jacderida wants to merge 2 commits into
Conversation
Hardens the download path against two distinct failure modes observed
running `ant file download` against the production network: residential
link saturation and per-peer / DHT transient errors that previously
fatally aborted multi-hundred-chunk downloads.
End to end, this takes a residential `ant file download` from "aborts
on the first 256-wide concurrent batch's saturation event" to
"completes 11/11 files including 2+ GB downloads," and on a fat-pipe
droplet from "matches baseline" to "matches baseline" — no regression
on the warm-start path that production downloaders actually exercise.
Six related changes:
1. retry-on-Ok(None) with unanimous-NotFound threshold in chunk_get.
When the close group returns Ok(None) (no peer has the chunk),
retry once with a fresh find_closest_peers lookup, unless every
queried peer responded with an authoritative NotFound (the only
safe stop for genuine data absence). The previous behaviour treated
Ok(None) as fatal on first occurrence, which on a saturated link
meant any single chunk's transient close-group exhaustion aborted
the whole download.
2. rebucketed_unordered in file.rs instead of buffer_unordered for the
in-flight chunk fetches. The adaptive limiter's cap can now shrink
the in-flight count mid-batch under sustained pressure;
buffer_unordered snapshotted the cap once at pipeline build and
ignored later Decrease decisions.
3. observe-outer with Ok(None) -> Outcome::Timeout instead of
observe-per-peer. The controller sees one observation per chunk_get
(not one per peer attempt), classified via a new chunk_get_outcome
helper that treats Ok(None) as a load-shedding signal. Avoids the
per-peer noise floor on the production network where some peers in
any K=7 close group are unreachable from any given client even on a
healthy link — that noise was driving spurious Decrease decisions
on the droplet and pinning steady-state cap low.
4. ChannelStart::fetch: 64 -> 4 cold-start. The 64-wide initial burst
saturated residential connections before the controller had any
observation to act on. 4 is the value confirmed safe on a real
residential link. On droplets the cost is a one-off cold-start
warm-up of ~16 min on the first 2.5 GB file; subsequent files
warm-start from the persisted client_adaptive.json snapshot (which
the controller cleanly grows to cap=256, the channel ceiling).
5. Deferred-retry pass in streaming_decrypt's consumer. When chunk_get
returns Ok(None) for a chunk during a batch, the chunk is deferred
rather than aborting the batch. After the main batch settles, the
deferred chunks are retried serially with sleeps of 10/30/60 s.
This rides out transient saturation events that hit multiple
in-flight chunks at once — by the time the batch has drained and
the first sleep elapses, the link has usually settled. A chunk
only becomes fatal after all 3 deferred attempts fail.
6. Per-peer protocol-error tolerance and deferred-retry transient-error
tolerance. A single peer returning Error::Protocol (e.g. "Chunk
verification failed" from a corrupted local copy) no longer aborts
the close-group sweep — the loop counts it and continues to the
next peer. Similarly, an Err(_) from chunk_get_observed during a
deferred-retry attempt logs and falls through to the next attempt's
longer backoff rather than escalating.
Also: latency_inflation_factor default 2.0 -> 4.0. Natural close-group
fallback latency on the production network routinely doubles vs the
EWMA baseline (a single peer hitting fallback adds ~10 s on top of a
sub-second median), and was firing spurious Decrease decisions even
on the droplet. 4.0 is the value validated on the previously-merged
tune-latency-inflation-factor branch.
Test plan:
- 296 ant-core unit tests pass.
- End-to-end residential download (PROD-LOCAL-DL-04): 11/11 files
completed including 2.51 GB (42m 43s) and 2.76 GB (46m 26s).
During an earlier residential run 14 chunks went through the
deferred-retry path and every one recovered on attempt 1/3 after
the 10 s sleep.
- End-to-end droplet download (PROD-DL-05): 20/20 files completed.
The first 2.5 GB file paid 16m 9s of cold-start cost; subsequent
multi-GB files ran in 3-6 min each, near the pre-change ~5 min
baseline on PROD-DL-02. No retry mechanism fired across the
20-file run — healthy-network success.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jacderida
force-pushed
the
home-download-fixes
branch
from
e8b1de5 to
13070d9
Compare
The previous commit fixed residential saturation and abort-on-first- failure, but field reports from fast connections (e.g. an Oracle VPS that gets full speed on the released client) showed the opposite problem: the fetch cap stayed pinned at ~13-24 across an entire 36-file run and never climbed toward the 256 ceiling, so multi-GB files took ~22 min each instead of ~5. Three compounding causes, all from having tuned exclusively for the saturated-home case: 1. Cap can't grow. AIMD exits slow-start permanently on the first Decrease, then grows +1 per 32-observation window. On a link with a steady ~4% close-group-exhaustion trickle, intermittent Decreases fire often enough that additive +1 never gets ahead — equilibrium ~20. Additive growth simply cannot reach a useful cap from a low base before a file finishes. Fix: add `LimiterConfig::slow_start_ramp_threshold`. Below it, a Decrease still halves the cap but keeps slow-start armed, so the next healthy window doubles back up instead of crawling. The fetch channel sets it to the channel ceiling, so download concurrency tracks the connection's real capacity. Default 0 preserves the original behaviour for quote/store. 2. The p95-latency Decrease misfires on fetch. `chunk_get_observed`'s latency includes the internal 1 s retry sleep and the slow retry sweep for chunks that needed one, so a window with a couple of retry-path chunks has a wildly inflated p95 that reads as congestion. Fix: add `LimiterConfig::latency_decrease_enabled`, false for fetch. Genuine fetch congestion still surfaces via the Ok(None) -> Timeout rate, which the timeout_ceiling check catches. 3. The deferred-retry pass was a throughput sink. It retried deferred chunks SERIALLY with a mandatory 10 s pre-sleep each; a batch that deferred ~20 chunks burned minutes of near-zero throughput even though every chunk succeeded on its first retry (the 10 s sleep was pure waste — the deferrals were peer-side noise that clears in <1 s). Fix: retry deferred chunks in CONCURRENT rounds reusing the fetch limiter, with the first round firing immediately and later rounds backing off (0/15/45 s) only for chunks that survive a round. Both Ok(None) and transient errors re-defer to the next round; only the final round's leftovers are fatal. Quote and store channel behaviour is unchanged (threshold 0, latency-decrease enabled). New unit tests cover protected-vs-additive recovery, the disabled latency check, and that the controller applies the download tuning to fetch only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the download path against two distinct failure modes observed running
ant file downloadagainst the production network: residential link saturation and per-peer / DHT transient errors that previously fatally aborted multi-hundred-chunk downloads. Together the changes take anant file downloadrun on a residential connection from "aborts on the first 256-wide concurrent batch's saturation event" to "completes 11/11 files including 2+ GB downloads," and on a fat-pipe droplet from "matches baseline" to "matches baseline" — no regression on the warm-start path that production downloaders actually exercise.What's in here
Six related pieces:
retry-on-Ok(None)with unanimous-NotFound threshold inchunk_get. When the close group returnsOk(None)(no peer has the chunk), retry once with a freshfind_closest_peerslookup, unless every queried peer responded with an authoritative NotFound (which is the only safe stop for genuine data absence).rebucketed_unorderedinfile.rsinstead ofbuffer_unorderedfor the in-flight chunk fetches, so the adaptive limiter's cap can shrink the in-flight count mid-batch under sustained pressure.Ok(None) → Outcome::Timeoutinstead of observe-per-peer. The controller sees one observation perchunk_get(not one per peer attempt), classified via a newchunk_get_outcomehelper that treatsOk(None)as a load-shedding signal. Avoids the per-peer noise floor on the production network where some peers in any K=7 close group are unreachable from any given client even on a healthy link.ChannelStart::fetch: 64 → 4cold-start. The original 64-wide initial burst would saturate residential connections before the controller had any observation to act on. 4 is the value confirmed safe on the operator's home link. On droplets the cost is a one-off cold-start warm-up of ~16 min on the first 2.5 GB file; subsequent files warm-start from the persisted snapshot (which reachescap=256cleanly).streaming_decrypt's consumer. Whenchunk_getreturnsOk(None)for a chunk during a batch, the chunk is deferred rather than aborting the batch. After the main batch settles, the deferred chunks are retried serially with sleeps of 10/30/60 s, giving the link time to clear any transient saturation. A chunk only becomes fatal after all 3 deferred attempts fail.Error::Protocol(e.g., "Chunk verification failed" from a corrupted local copy) no longer aborts the close-group sweep — the loop counts it and continues to the next peer. Similarly, anErr(_)fromchunk_get_observedduring a deferred-retry attempt logs and falls through to the next attempt's longer backoff, rather than escalating.Also:
latency_inflation_factordefault2.0 → 4.0(cherry-picked from the previously-validatedtune-latency-inflation-factorbranch — natural close-group fallback latency on the production network routinely doubles vs the EWMA baseline, and was firing spurious Decrease decisions on the droplet).What's not in here
chunk_get_from_peer. The retry happens at the close-group sweep level (once insidechunk_get, once at the deferred pass).Evidence
The most recent end-to-end runs both completed cleanly:
Local download (residential connection,
PROD-LOCAL-DL-04)11/11 files completed including 2.51 GB and 2.76 GB downloads:
Files 4 and 5 are the multi-GB workloads that previously aborted on the first close-group exhaustion within the first few minutes. They now complete via the deferred-retry mechanism — during the earlier successful home test, 14 chunks were deferred and every single one recovered on attempt 1/3 after the 10 s sleep.
Droplet download (production,
PROD-DL-05)20/20 files completed:
File #4 is the cold-start cost: the adaptive limiter ramps from
ChannelStart::fetch=4through doublings to the channel ceiling of 256, and the snapshot persists at 256 for subsequent runs. Files #5 onwards run at near-baseline speeds: 3-6 min per 2+ GB file, vs the pre-change baseline of ~5 min onPROD-DL-02.Grepping the per-file logs on
PROD-DL-05shows none of the new retry mechanisms fired across the 20-file run — everychunk_getsucceeded on its first close-group sweep. So this is a healthy-network success, not a "saved by deferred retry" success. The deferred-retry mechanism is proven on the home runs (14 successful recoveries inPROD-LOCAL-DL-03); the droplet just didn't need it.Test plan
ant-coreunit tests pass (cargo test -p ant-core --lib).PROD-LOCAL-DL-04): 11/11 files completed.PROD-DL-05): 20/20 files completed.🤖 Generated with Claude Code