Skip to content

fix(core): queue all future blocks of a batch instead of failing - #2550

Open
gzliudan wants to merge 5 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-future-block-batch
Open

fix(core): queue all future blocks of a batch instead of failing#2550
gzliudan wants to merge 5 commits into
XinFinOrg:dev-upgradefrom
gzliudan:fix-future-block-batch

Conversation

@gzliudan

@gzliudan gzliudan commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

insertChain treats a mid-batch ErrFutureBlock as a fatal verification error: both future-block loops only accepted ErrUnknownAncestor, so a batch whose first or middle block was in the future stopped at its second block and returned consensus.ErrFutureBlock. The downloader wraps any non-nil InsertChain error into errInvalidChain (eth/downloader/downloader.go) and Synchronise answers that by dropping the delivering peer, aborting the round and retrying with another peer. Near the chain tip this repeats and continuously evicts innocent peers.

Root cause

The XDPoS engines check the header timestamp before the parent lookup and with zero tolerance (header.Time > now), unlike upstream geth's 15-second allowedFutureBlockTimeSeconds. Consequently the children of a future block fail the future check too — they surface as ErrFutureBlock, never as ErrUnknownAncestor, so the inner loops (which only accept ErrUnknownAncestor) stop at the second block and the rest of the batch is silently skipped.

Before #2534 the same scenario silently dropped the tail and failed one batch later with a bogus bad-block report.

Fix

Accept consensus.ErrFutureBlock in both future-block loops of insertChain (first-block path and tail path), so the whole tail enters the futureBlocks queue and the import reports success. No error is swallowed: the loops drain naturally to (nil, nil) and genuine verification errors still propagate. The queued blocks are imported by procFutureBlocks (100 ms ticker) once their timestamps are reached, matching the upstream geth design intent.

Blocks beyond the now+30s future window still return an error (abort + peer drop), which is the documented behaviour for a local clock more than 30 seconds behind — an environment problem, not an invalid chain.

A genuine verification error on a block after the queued future tail (e.g. a malformed validator field, which the XDPoS engines check before the timestamp) also surfaces: it is recorded as a bad block and returned, so only a fully drained tail reports success. Future, pruned-ancestor and known blocks remain exempt from the bad-block report — they are legitimate chain states. The same applies to a genuine error after the queued prefix when the batch starts with a future block: it is recorded and returned as well. When the known-block import PR lands, its final-return filter reports the same block again; WriteBadBlock deduplicates by (number, hash), so the only effect is one extra BAD BLOCK log line.

Tests

New core/blockchain_futureblocks_test.go (self-contained, own failRangeEngine with an atomic.Uint64 fail range because the chain's future-block loop calls VerifyHeaders concurrently):

  • TestInsertChainQueuesMidBatchFutureBlocks — a batch rejected as future mid-way returns n == len(blocks), err nil, head stays at blocks[1], the whole tail is in futureBlocks, no bad-block records.
  • TestInsertChainQueuesFutureBatchFromFirstBlock — same for a batch whose first block is already in the future (pre-existing scenario).
  • TestInsertChainProcFutureBlocksResumesImport — once headers verify again, the queued tail imports and the head reaches the batch tip (head polled with a deadline because InsertChain uses TryLock and the background loop may be mid-import).

All three are mutation-verified: reverting either loop condition alone makes the corresponding test fail. go test ./core/ and go test ./eth/downloader/ pass, also under -race; gofmt/go vet clean.

Compatibility

No consensus-rule changes: block/receipt structure, state transition and wire protocol are untouched. Only the error handling of the import path changes. No migration or node-operator action required. Suggested label: consensus.

@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 06bcda29-2a4d-4d99-be58-a6bfdc83230b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A verification error following queued future blocks is still discarded by the final nil-error return.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Queues entire future-block batches to prevent downloader peer eviction near the chain tip.

Changes:

  • Accepts ErrFutureBlock while queuing descendants.
  • Adds tests for first/mid-batch queuing and resumed import.
File summaries
File Description
core/blockchain.go Expands future-block queue handling.
core/blockchain_futureblocks_test.go Tests queuing and later processing.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread core/blockchain.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two error paths can omit bad-block reporting or silently report an incomplete import as successful.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

core/blockchain.go:1959

  • These exemptions currently suppress both the bad-block report and the return. If iteration stops on ErrPrunedAncestor or ErrKnownBlock, block is still unconsumed but execution falls through to the final nil return, recreating a silent partial import. Exempt legitimate states only from reportBlock; every non-nil iterator error must still be returned.
		if err != nil && !errors.Is(err, consensus.ErrFutureBlock) &&
			!errors.Is(err, consensus.ErrPrunedAncestor) && !errors.Is(err, ErrKnownBlock) {
			bc.reportBlock(block, nil, err)
			return it.index, events, coalescedLogs, err
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread core/blockchain.go Outdated
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch 22 times, most recently from 15971f6 to e960998 Compare September 6, 2026 08:21
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch 12 times, most recently from 9823b56 to 380e6c9 Compare September 6, 2026 21:16
@gzliudan gzliudan added the WIP work in process label Sep 6, 2026
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch from bcc88c0 to c6bb2ed Compare September 6, 2026 21:55
…chain.go

Replace direct == / != comparisons against sentinel errors
(consensus.ErrPrunedAncestor, consensus.ErrFutureBlock,
consensus.ErrUnknownAncestor, ErrKnownBlock, ErrStopPreparingBlock)
with errors.Is, which also matches wrapped errors.
stats is a function-local insertStats and stats.report is only called
from the main import loop, so every stats write on a path that returns
before reaching the loop is dead and never logged:

- stats.queued += it.processed() on the first-block future path is not
  only unreachable for reporting but off by one: it.processed() returns
  it.index+1, which over-counts by one both when the whole batch is
  drained (it.index == len(chain) at exhaustion) and when the loop
  aborts on a non-exempt error (it.index points at the block that was
  never queued). Both queued counters date back to ff435e0.
- stats.queued++ in the tail future loop (which also skipped the block
  that triggered the queueing) and stats.ignored += it.remaining() on
  both future paths are written then dropped on return.
- stats.ignored += len(it.chain) in the first-block error abort has
  the same write-then-return shape.

Remove all of them. Drop the insertStats.queued field and the "queued"
log context in report() that could never fire once the counters are
gone, matching upstream geth whose insertStats no longer carries a
queued field, and remove the now-unused insertIterator helpers
processed() and remaining(). The live stats.ignored accounting (the
ErrKnownBlock skip loop that falls through to the import loop and the
InsertReceiptChain stats) is untouched.
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch from c6bb2ed to 5b4e565 Compare September 6, 2026 22:25
Add regression tests asserting that header verification checks the
timestamp first and answers ErrFutureBlock for blocks too far ahead,
without requiring their parent to be resolvable. XDPoS v1/v2 (under
full verification) order the checks this way; these tests pin the
behavior the future-batch tail queueing (queueFutureTail) relies on:
children of a future block surface as ErrFutureBlock, not
ErrUnknownAncestor.

No production code is changed.
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch 4 times, most recently from 1752135 to ad6bab5 Compare September 7, 2026 03:32
…canonical

Problem

importBlockResults fed the last fetched block to the proposed-block
handler as long as the batch import returned nil, taking a nil error to
mean the last block reached the canonical chain. A nil InsertChain does
not imply canonicality: a fork batch writes its blocks as side-chain
entries that a mere existence check cannot tell from canonicality, and
processQC overwrites highestQuorumCert before its own existence check
(GetHeaderByHash), so a master node can end up voting for a block that is
not in the canonical chain.

Changes

- Gate the proposed-block handler on storage and canonicality: the batch
  tail must be present as a full block and be the canonical block at its
  own height, via a new GetCanonicalHash method on the downloader's
  BlockChain interface. The canonical hash is keyed by height, so it is
  immune to a concurrent import advancing the head past the tail between
  the insertion and this read (a CurrentBlock comparison would spuriously
  skip the handler in that window); only a reorg changes it, and a
  reorged-away block must not reach the handler. HasBlock covers the fast
  sync header phase, where the header chain marks a height canonical
  before its body is imported.
- downloadTester gains storeBlock/GetCanonicalHash, a canonical
  number-to-hash table advanced the way HeaderChain.WriteHeader and
  insertChain advance theirs, plus parkTailOnce and extendTailAfterInsert
  hooks so the tests can present batches that never reached the chain or
  were later reorganized away. Both hooks are consumed by the single
  InsertChain call they apply to. GetCanonicalHash picks the head by total
  difficulty, the way the real chain resolves a reorg, instead of by
  insertion order. HasBlock now also requires the stored block to match
  the requested height, like the real chain.

Tests

- TestImportBlockResultsProposedBlockHandler covers six scenarios: a batch
  that never reached the chain must not trigger the handler, a fully
  imported batch triggers it exactly once on the batch tail, a stored fork
  batch re-delivered after the local chain grew past it must not trigger
  it again, a concurrent import advancing the head past the canonical tail
  must not skip the handler (height-keyed lookup), a heavier fork stays
  canonical once a lighter chain is stored after it, and a height that the
  fast-sync header phase made canonical without a body must not trigger
  it either.
Problem
-------

XDPoS engines check the header timestamp before the parent lookup with
zero tolerance, so the children of a future block also fail verification
with ErrFutureBlock rather than ErrUnknownAncestor. The insertChain
future-block loops only accepted ErrUnknownAncestor, so a batch whose
first or middle block was in the future stopped at its second block and
returned the error, which the downloader wraps into errInvalidChain and
answers by dropping the peer on a valid delivery. Before XinFinOrg#2534 the same
scenario silently dropped the tail and failed one batch later with a
bogus bad-block report.

Changes
-------

* Accept ErrFutureBlock in both insertChain future-block loops so the
  whole tail enters the futureBlocks queue and the import reports
  success; the queued blocks are imported by procFutureBlocks once their
  timestamps are reached. Queueing is bounded by addFutureBlock's
  maxTimeFutureBlocks (30s) window: a tail whose timestamps span past it
  aborts the batch with the enqueue error (non-sentinel, so the
  downloader still drops the peer) — the same visible outcome as before —
  while the in-window prefix stays queued.

* Gate HandleProposedBlock on canonicality instead of block existence
  (importBlockResults now requires the tail to be the current head). A
  nil InsertChain no longer implies the last block reached the chain, and
  a queued tail can sit at or below the head height (re-delivered history
  on a side chain); feeding such a block into processQC overwrites
  highestQuorumCert before its block-existence check, so a master node
  could end up voting for a block that is not in the chain.

* In the fetcher, move the sign hook, the proposed-block handling and the
  final broadcast behind a "block exists in the chain" gate.

* Evict parked blocks that fail to import: procFutureBlocks used to keep
  every failed block, so garbage delivered inside the future window could
  fill the queue permanently and be re-verified and re-reported as bad
  blocks on every futureBlocksLoop tick forever. A block is now evicted
  unless it is still in the future or its parent is itself parked; blocks
  are sorted by number, so evicting a failed parent cascades to its
  queued children within the same pass.

* Collapse the duplicated enqueue loops of the two insertChain future
  paths into the shared queueFutureTail helper, and drop the dead
  insertStats accounting on the abort paths (stats.report is only called
  from the main import loop).

Tests
-----

Future-block regression tests live in core/blockchain_futureblocks_test.go.
@gzliudan
gzliudan force-pushed the fix-future-block-batch branch from ad6bab5 to 4862bd0 Compare September 7, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

WIP work in process

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants