fix: run the frame-record gap backfill on every role, at runtime - #620
Open
blacks1ne wants to merge 2 commits into
Open
Conversation
blacks1ne
force-pushed
the
blacks1ne/backfill-below-the-lowest-stored-frame
branch
from
August 18, 2026 22:54
9492a76 to
2d9231b
Compare
blacks1ne
changed the title
blacks1ne
force-pushed
the
blacks1ne/backfill-below-the-lowest-stored-frame
branch
2 times, most recently
from
August 26, 2026 13:09
f21034b to
513b89f
Compare
blacks1ne
force-pushed
the
blacks1ne/backfill-below-the-lowest-stored-frame
branch
2 times, most recently
from
September 6, 2026 15:00
b72e7d7 to
e33ae7f
Compare
A non-archive never ran the frame-record gap backfill at all, and no node ran
it after bootstrap. Verified on a live mainnet regular: zero `gap scan`,
`restart gap scan` or `record-only` lines in six hours of uptime. Both gates
did that — `run_all_gap_backfill` was behind `if sync_archive_mode`, and it was
spawned once inside the bootstrap closure.
That matters because the hole it would have closed is large and permanent. A
freshly-wiped mainnet node holds the genesis record at 244200, then whatever
gossip delivers once it adopts the head — on the node examined, a contiguous
block at 701050..=701741 — and nothing in between. 456,849 absent frames. The
poller cannot help: forward-fill only climbs from its cursor, which sits at the
head. A regular's app-shard storage attestation anchors rho_N to an exact
global frame that must be present locally, and below the cursor it is not.
The same shape is reached by several routes — a successful state jump (which
writes its target frame and nothing beneath it, at boot or via the runtime
far-behind rescue), a failed jump plus gossip head adoption, or a restart. The
scan already reported this as an internal hole. Nothing ever ran the scan.
Changes:
- The scan runs for every role, on a loop (`GAP_RESCAN_INTERVAL`, 10 min).
Serving ranges is the archive's reason to want contiguity, not the only one;
`forward_fill` is already enabled network-wide on the same argument.
Bootstrap-only missed the runtime jump entirely.
- Gaps are descended in chunks of 512 rather than handed over whole.
`run_record_only_backfill` materializes `(lo..=hi)` into a `Vec<u64>` and
fetches serially — fine for a 3-frame restart leftover, not for 456k.
Descending, because each chunk's `parent_selector` anchor is the record the
chunk above it just filled.
- A chunk nobody can serve stops that gap's descent, since below the migration
boundary every frame fails the genesis-prover allowlist. The signal is
`filled == 0 && unresolved > 0`, not `filled == 0`: a chunk the poller closed
between the scan and the backfill also fills nothing, and stopping on it
would abandon a fillable descent. Hence `BackfillOutcome`.
- Non-archives bound the descent to three epochs below a gap's TOP —
the slots a leaf-root registration vertex retains ({prev, current, next}),
so every epoch an in-flight opening can be validated against. Measured from
the top because the heights needed soonest are those just under the records
the node already has. Without it a regular would serially fetch ~31k frames
down to the migration boundary.
- `find_global_frame_record_gaps` also reports the hole below the store's
lowest record, which it structurally could not see before (it emitted a range
only between two iterated keys). The caller MUST clamp that to the network's
genesis frame and now does: `bootstrap_genesis` writes the genesis record on
every node, so `earliest` IS genesis and the reported floor range is entirely
fictional — 244,199 mainnet heights that never existed. Unclamped it costs a
chunk of pointless fetches at every startup and then blacklists itself.
Tests: six fail before this change and pass after — the scan reporting the
floor hole, that hole filled from local candidates, a descent spanning more
than one chunk, the depth bound applying to an internal gap measured from its
top, and the fictional sub-genesis range being discarded rather than fetched.
The rest pass on both sides, including the pair separating an already-complete
range (`{filled: 0, unresolved: 0}`) from an unservable one
(`{filled: 0, unresolved: 3}`). `-p quil-store -p quil-node` 131/131.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Field report on the parent commit, from a wiped mainnet non-archive: the descent filled ~29k records down to the flag day, then stalled there and never moved again. Frames at or below GLOBAL_FLAG_DAY_LAST_LEGACY_FRAME (669975) were produced by pre-migration provers, so step 1 of `archive_frame_is_valid` — the genesis-prover allowlist — drops every one of them. Its own comment already said so: "expected for legacy pre-migration frames". Archives serve those records correctly; this node fetches them and then discards them locally. The parent commit created the stall. Before it, only archives ran the scan, and archives migrate from the flag-day head so they never descend below it. Un-gating the scan for non-archives pointed the new 3-epoch depth cap at [667816, 669975] — 2160 heights that are 100% unfillable by construction. - Floor the descent at the flag day on mainnet, folded into the existing `floor_frame` so the clamp itself is unchanged. Other networks never rewound and keep genesis; the constant sits above every frame they have. - Clamp before cap, via one shared helper, so a cap deeper than the distance to the floor cannot re-admit legacy heights. - Report `intended_frames` next to `missing_frames`. The raw hole size is not the objective: on the reporting node it read 669,974 and never moved, of which 244,199 is the fictional sub-genesis range and 425,775 is the legacy range. The true objective was zero. - Tally WHY heights went unresolved (invalid / unavailable / timed_out / store_failed / connect_failed). A locally-rejected frame and a frame no peer holds were one indistinguishable number, which sent the reporter hunting an archive-side gap that does not exist. - Drop the "likely uncommitted/orphaned (correctly not canonical)" inference. It was written for the reseed path, which fetches above the canonical head; it is provably wrong for a gap-scan range, whose hi+1 is a present canonical record — the chain is contiguous by frame number, so heights beneath it cannot be orphaned. The log field `unrecoverable` is now `unresolved`. Proven RED without the fix: the floor test reports 244200 vs 669976, and the intent count reports 2160 vs 0 — 2160 being exactly the reporter's stalled window (669975 − bounded_lo 667816 + 1). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blacks1ne
force-pushed
the
blacks1ne/backfill-below-the-lowest-stored-frame
branch
from
September 8, 2026 07:02
e33ae7f to
25870f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
A node can retain a high global-frame head while missing the contiguous records beneath it. Forward frame polling cannot repair that shape; the gap backfill must run beyond bootstrap and must avoid retrying frames this node cannot validate.
Validation