Conversation
The orphan-leave sweep was reachable for exactly one epoch per orphan, so an orphan that survived a boundary became permanently unsheddable. Being orphaned is what expires an allocation: no worker bound means no proofs, so it misses its per-epoch re-confirm at the next boundary and `AllocationBuckets::from_allocations` moves it out of `active` into `expired_epoch`. The sweep drew its candidates from `active` alone, so from that moment nothing could propose a Leave for it — not the orphan path, and not `plan_leaves`, which scores `allocated_descriptors`, itself filtered by `active`. The node then holds allocations it cannot serve, indefinitely, with no automatic way out. Observed on a node with 35 allocations against 15 workers: the 20 unbound ones all read `re-confirm!` and no Leave was ever proposed. Draw orphan candidates from `active` UNION `expired_epoch`, in a `orphaned_allocation_filters` helper. `expired_epoch` deliberately does not join `active` itself — that set is coverage accounting, and an allocation that has not re-confirmed genuinely does not count — so this widens only who may be shed. The existing exclusions still apply, on the expired path as much as the active one: a filter with a worker bound (including an in-flight join), an operator-pinned filter, and one already mid-Leave are all left alone. The leave block's own `!active.is_empty()` guard is widened the same way, so a prover whose every allocation expired still sweeps. An unbound allocation stays on-chain Active, so a Leave is valid for it; only the derived status reads expired. Not changed: the per-epoch re-confirm still queues every `expired_epoch` filter, so an orphan is briefly both re-confirmed and proposed for Leave. Re-confirming re-encodes replicas from local storage, and whether that is reachable without a bound worker is not established here, so narrowing it is left out of this fix. The Leave lands within an epoch, after which the allocation is `Leaving` and out of `expired_epoch`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011K67csMrXEr4ovegwy2Fmq
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The orphan leave sweep considered only active allocations, but orphaned allocations become
ExpiredEpochand were therefore unreachable after one epoch.Include expired allocations in the orphan leave candidates without counting them as active coverage.
Tests:
cargo test --locked -p quil-engine --libBase:
4eaf1f79f4eb022b9a9d3ccc072064050e5781b6