feat(snap): add EIP-8189 state bootstrap - #11
Draft
lean-apple wants to merge 117 commits into
Draft
Conversation
Adds `reth-engine-snap`, the client half of snap/2 (EIP-8189): a pivot tracker that follows the chain head reported by the engine, and a streaming downloader that pulls accounts, storage and bytecodes at the pivot root. Every response is verified before it is written. Range proofs are checked by reconstructing the trie root from the returned leaves plus the proof subtrees outside the range, so a peer cannot omit a leaf in the middle of a range that proves both endpoints. Storage ranges are additionally checked against each account's storage root, and bytecodes against their code hashes. When a peer can no longer serve the pivot root the download reports where it stopped, the pivot advances to a fresher block, and the download resumes from that account rather than restarting. BAL catch-up between the final pivot and the head, the final state-root check, and reorg recovery are not implemented yet.
The four write helpers each repeated the same three provider bounds. Moving them onto a struct that holds the factory states those bounds once on the impl and lets the downloader thread one value instead of a factory reference.
Adds BAL replay, EIP-8189's replacement for snap/1 trie healing: once the state at the pivot is downloaded, the blocks between the pivot and the head are brought forward by writing the post-block values their access lists commit to, with no transaction execution and no trie-node round trips. A block's access list only carries the fields that block changed, so entries are merged onto the stored account rather than overwriting it. The post-block value of a field is the change with the highest block access index, read from the index itself rather than from list order. `catch_up_with_bals` works against a head snapshot taken at entry so it always terminates; the caller re-invokes to follow a head that moved meanwhile.
Replaces hand-rolled equivalents of things reth and alloy already provide: hashed writes go through `HashedPostState` instead of ad-hoc tuples, storage ranges are checked with `reth_trie::root::storage_root` instead of a proof-free range verification, `Account::into_trie_account` and `From<TrieAccount>` do the account conversions, and the range cursor uses `U256` arithmetic. Moves snap/2's slim account encoding onto `AccountData` in eth-wire-types so the server and the downloader share one codec rather than each keeping a copy, and gives `StorageData` the matching value accessors. Groups the loose download helpers onto `StateDownloader` and `StorageRoots`, and the proof-free response checks onto the latter, so the provider bounds are stated once per impl instead of on every function.
Closes the sync with the check the per-range proofs cannot give: each response was only proved against the root it was served at, so nothing until now ruled out gaps between ranges served at different pivots, or a block access list applied wrongly. `SnapStateWriter::finalize_sync` rebuilds the state trie over the assembled hashed state and compares its root to the block header. The same pass yields the intermediate trie nodes, which are written on success because a node cannot serve proofs or extend the chain from hashed state alone. A mismatch returns both roots and commits nothing, so a retry starts from the hashed state rather than a half-built trie. Also propagates the `alloy-trie` features added with the shared slim account codec, which zepter flagged as missing on no_std builds.
…rand Applying a block access list writes post-block values without recording what they replaced, so an orphaned block cannot be rewound the way an executed one can, and re-applying the new chain only corrects the keys that chain happens to touch. `AppliedChain` remembers which keys each applied block wrote. Catch-up compares each block's parent against what was applied below it and stops on a mismatch; the orphaned blocks' keys are marked stale, and re-applying the new chain clears whatever it rewrites. What remains is state from a chain that no longer exists. Reading those keys back from peers is the follow-up. Until then a reorg leaves `stale_keys` non-empty, which the final state-root check turns into a failure rather than silent corruption.
The downloader dropped the peer id with `into_data()`, so a peer answering with the wrong message type, a bad range proof, non-monotonic keys or unrequested bytecode was never penalized and kept being selected. Every other reth downloader reports these through `DownloadClient::report_bad_message`; snap now does the same at each validation point. A block access list that does not match the header commitment reports the peer too, but only when the list came from one: lists delivered by the engine with the payload have no peer to hold to account.
Reporting a peer and returning an error still ended the whole sync on the first bad response, so a single peer serving a wrong message type or an unusable proof could stop snap sync outright. Penalizing without retrying does not keep the download going; `bodies/request.rs` reports the peer and resubmits. Each request kind now retries up to `MAX_REQUEST_ATTEMPTS` times, reporting the peer between attempts so the network layer routes the retry elsewhere, and only fails once the attempts are used up. Response validation moves into the request methods, since a response is only worth returning once it has been checked. `StorageRoots::verify_response` checks a whole storage-ranges reply, which drops the per-call-site penalize helpers.
Completes reorg recovery. Detection already identified the keys an orphaned chain wrote that the surviving chain never rewrites; those are now read back from peers so a reorg costs a handful of lookups instead of failing the sync at the state-root check. Snap has no request for a specific key, so each is fetched as a range whose origin and limit are that key. Those replies are checked with a single-key proof rather than the range verifier: a range proof asserts completeness through to the end of the trie, which a one-key reply does not claim and cannot support. An absence proof is a real answer, not a gap. An account that no longer exists is deleted and its storage wiped, and an absent slot reads back as zero, so a reorg that removed state removes it here too.
Adds an opt-in `--snap` flag that advertises the `snap/2` capability. The capability was reachable only through `NetworkConfigBuilder::with_snap`, which the node launch path never called, so no reth node advertised snap/2. It stays off by default: snap is not reth's sync path, and advertising commits the node to serving those requests. This covers the serving half of activation. Driving a snap sync still needs the engine to feed `SnapSyncEvent`s, so the flag makes a node a snap/2 server rather than a snap/2 client. Also drops a doc link from public `orphan_from` to the crate-private `record`, which fails the `docs` CI job under `-D warnings`.
Snap sync consumes forkchoice information but is not Engine API processing. Living under crates/engine invited the coupling it had: payload buffering, canonicality, downloading and persistence all reaching into each other. Renames reth-engine-snap to reth-snap-sync and moves it to crates/snap-sync. `SnapSyncSession` is now the single serialized owner, with pivot advancement and reorgs as transitions of its state machine rather than separate subsystems, so pivot.rs and reorg.rs are gone. `CanonicalChainSource` gives the crate the narrow view of the chain it needs without depending on engine-tree internals. Two defects stop being expressible. Blocks are identified by hash everywhere, so a target, header or access list can no longer be resolved to whichever block happens to sit at a height; and the head comes from forkchoice, so it may move sideways or backwards instead of only forward. `store::reset` gives a session a clean generation rather than inheriting a genesis allocation or a failed run. Drops the key-only reorg recovery. Re-reading a stranded key at the target root restores the value from before the fork, which is wrong whenever that key also changed between the target and the common ancestor; a session now restarts. Protocol layers stay where they are: eth-wire-types owns the messages, p2p the request traits, network the peer selection. This crate owns synchronization policy and database assembly only.
Addresses the review of 585e46c. Stale targets can now advance. `advance_target` picks a fresher target and applies the access lists between the old and new one to the already-downloaded prefix, which is EIP-8189's rolling transition. Without it the only way past a stale root was `start`, which discards all progress, and resuming at a new root would have left a prefix from one state beside a suffix from another. Finalization re-anchors against forkchoice. The head can move while access lists are being applied, and a root matching an orphaned block matches nothing the node will build on. Bytecodes are re-requested until none are outstanding. A short reply is legitimate because servers cut responses at a size limit, so dropping the hashes it omitted left accounts pointing at code the database does not have — which the state root check cannot catch, since code lives outside the trie. Complete zero-origin storage tries now replace rather than merge, so slots absent from the downloaded trie cannot survive a target advance or a repeated account write. Also: an empty state root is a valid answer rather than an unavailable one; an empty reply from one peer spends an attempt instead of ending the request; BAL requests retry against another peer; and an account left empty by an access list is deleted per EIP-161 instead of being written as an empty leaf, using the existing `Account::is_empty` rather than a local predicate.
…the trie walk Addresses the review of c97b1b7. A served account range is now committed in micro-batches, and nothing becomes durable until that batch's accounts, their complete storage and every bytecode they reference are written in one transaction. Writing accounts before their storage let a stale root strand them above the resume point: the rolling target transition applies access lists only below the covered prefix, and a range at a fresher root does not mention accounts that were deleted, so the old leaf survived. Finalization reads a forkchoice token before the state trie is rebuilt and compares it afterwards. Rebuilding walks the whole state, which is long enough for the head to move and leave the earlier canonicality check stale, committing trie updates for an orphaned block and marking the session complete. Deleting an account under EIP-161 now emits a wiping `HashedStorage`. Reth clears an account's storage rows only when the entry is marked wiped, so a deleted account that changed no slots kept its old ones and a later recreation at the same address would inherit them.
The wasm job builds every workspace crate that is not excluded. Snap sync depends on reth-provider and reth-network-p2p, which pull in tokio features and C libraries that do not build for wasm32-wasip1, so the new crate has to sit with the other native-only ones.
`finalize_sync` built the whole state trie with a single `root_with_updates` call, holding every trie node for the entire state in memory before writing any of them. Drive `root_with_progress` in a loop instead, the way `MerkleStage` walks a full rebuild: each chunk's nodes are written and dropped, so peak memory no longer scales with total state size. All chunks share one transaction, so a root mismatch still discards every node written along the way.
The network layer rejects a snap request outright while no connected peer advertises `snap/2`, so a session starting before peers connect burned its whole retry budget in microseconds and reported a failed sync. Treat that rejection as its own outcome: the step records how far it got and returns `WaitingForPeers`, leaving the session resumable once a peer shows up.
A session starts by wiping the hashed state, so a crash part-way left tables that look like a healthy node's while holding a partial download, with nothing on disk to say so. Record the generation in the same transaction as the wipe and clear it in the same transaction as the root check, so the marker is present exactly while the state is untrustworthy. `interrupted_generation` reports it on startup.
The e2e test carried a hand-written decode counterpart of the slim account body to assert the wire shape. Drop it, along with the open-coded slot value decoding, and test the codec next to the wire type instead.
lean-apple
force-pushed
the
poc/eip-8189-snap-sync
branch
from
August 1, 2026 11:24
907abaa to
6afa30d
Compare
Applying a block access list wrote hashed state and deployed bytecodes in separate transactions, so a crash between them left an account's code hash pointing at bytecode the database does not have. The final root check cannot catch that, because code lives outside the trie.
Snap responses are keyed by hashed address with no preimage, so only the v2 layout — hashed tables as the canonical state representation — can be assembled from them. On v1 the state providers read plain tables, so a snap-synced node would verify a correct root and then execute against empty state. Checked before the wipe, so a v1 node's existing state is left untouched.
Engine-tree prewarming and snap-sync healing each carried their own reading of an EIP-7928 account entry — last write per field, merge onto the parent account, slot hashing, deployed-code collection — and the copies had already drifted: one took the last list element where the other took the highest block access index. Move the shared reading to `reth_trie_common::bal` behind an `eip7928` feature and consume it from both. The index decides everywhere now; how the result is used stays with each caller.
The chain source carried its own `cached_bal` hook, duplicating what the node's `BalStoreHandle` already provides. Hand the session the store instead, so payload-received lists and snap sync read the same cache and the chain source shrinks to canonicality alone.
A height alone does not identify a block across a reorg, so a marker carrying only the target number could not say which block the partial state on disk belongs to. Persist the target hash and state root next to the checkpoint row, in the same transactions that create and clear it.
Every access list request went out with request id 0. The ids exist to correlate requests with responses, which stops working the moment more than one is ever outstanding.
The marker was cleared inside `finalize_sync`, in the same commit as the trie tables — before the session re-checks that the block survived the trie walk. A reorg during the walk therefore left the database unmarked while holding the complete state of an orphaned block. A matching root proves whose state this is, not that the node accepted it. Clear the marker in an explicit completion step after the canonicality re-check, and rewrite it when the rolling transition moves the target, so it always names the block the state on disk is converging on.
A list fetched from a peer and verified against the header commitment is as trustworthy as one a payload carried, but it was dropped after use — a session that walks the same segment again re-fetched it. Insert it into the node's BAL store, the same place the session already reads from.
Adds `HeaderChain`, the first `CanonicalChainSource` implementation. A snap syncing node answers SYNCING to the consensus layer, so forkchoice reaches it as a bare head hash; numbers, parent links, state roots and access list commitments all come from peers' headers instead. Every header is verified against the hash it was requested by — the first against the request, each next against its predecessor's parent — so a peer cannot answer with a block other than the one asked for, and a segment walk either lands on its anchor or proves the two are on different chains.
Documents what EIP-8189 support exists today: serving behind --snap over a 128-block window, the experimental unwired sync side, the v2 storage layout requirement, and the SnapSync marker's meaning.
…apple/reth into feat/snap-account-range-downloader
# Conflicts: # crates/engine/tree/src/tree/payload_processor/prewarm.rs
…nap-sync # Conflicts: # crates/net/downloaders/src/snap/mod.rs
`RawBal::ensure_hash` and `DecodedBal::from_raw_bal` replace the manual hash comparison and `Vec<AccountChanges>` decode. Carrying one `RawBal` through the path also hashes the list once instead of twice, and the alloy decoder rejects trailing bytes.
`reth-trie-common` already re-exports alloy's `ProofNodes`, which maps trie paths to nodes; this one maps commitments to blobs.
`ancestor` and `segment` opened a read transaction per header, so selecting a pivot cost 17 of them and a catch-up segment one per block.
Driving a step from the wrong state was reported as a network failure, which reads as a peer problem and is retried like one.
Provider, database, chain and RLP failures were flattened into strings, which dropped the source chain and made every call site carry a `map_err`. Chain lookups were also reported as network failures.
The check needs neither a client nor a database, and being an associated function forced the test to invent a `StateDownloader<'_, (), ()>` alias.
Reth's block-keyed APIs take `NumHash`, which the session was building by hand from the two fields.
A peer that answered badly was only reported before the retry went back through normal peer selection, which weighs request state, response quality and latency rather than that report, so the same fast peer could take every attempt and stale a pivot another connected peer could serve. Snap requests now carry the peers already tried for the same logical request and the fetcher skips them. Once every capable peer has been tried the request fails with `NoEligiblePeers` instead of waiting for peer churn.
Trie reconstruction ran synchronously between the two canonical-token reads, but the token only changes when the outer select processes the head channel, which cannot happen while the session future blocks. A forkchoice update queued during that window left the token unchanged, so the session accepted state for the old head. Rebuilding now runs on the blocking pool so head updates keep landing, and acceptance re-checks canonicality immediately before clearing the generation marker. A head that only moved forward reopens healing instead of failing the sync, so the assembled state is replayed rather than downloaded again.
The dedicated snap connection was selected only for exactly eth + snap/2, so negotiating any third capability routed the session through the satellite multiplexer, which reported no snap support and rejected snap sends. snap/2 was silently unusable against such peers. Snap is now installed as a satellite protocol alongside the other handlers and bridged to the session, so the multiplexed connection reports and serves snap like the dedicated one.
lean-apple
force-pushed
the
poc/eip-8189-snap-sync
branch
from
August 17, 2026 19:32
d67cc77 to
08aa4ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds an opt-in EIP-8189 snap/2 state bootstrap for fresh or interrupted hashed-state databases. Once the state is verified and accepted, Reth resumes its existing pipeline.
Introduces a snap sync session that selects a hash-anchored pivot, downloads account, storage, and bytecode ranges, applies verified BALs, follows the canonical head, and verifies the final state root (session, advancement and healing).
Extends snap/2 networking with BAL messages, slim account encoding, request routing, response limits, and account/storage/code/BAL serving (wire types, request serving).
Reuses Reth infrastructure for header synchronization and normal backfill through
PipelineSync, together with existing provider, hashed-state, trie, stage checkpoint, and BAL abstractions (bootstrap wrapper).Adds crash-safe persistence through a generation marker, atomic state batches, a chunked trie rebuild, final root verification, and a RocksDB-backed BAL store (state writer, finalization, BAL store).
Modifies shared components only where reusable, including BAL post-state extraction and the committed trie-root rebuild helper (BAL state merge, trie rebuild).
Tests the complete bootstrap handoff, including adversarial range proofs, stale-pivot advancement, snap-to-pipeline continuation, and persisted BAL serving (proof tests, E2E bootstrap).
Snap remains default-off and is activated with
--snap.Towards Reth
The implementation can be upstreamed through several parallel tracks after landing the shared BAL semantics and codecs:
The corresponding upstream changes are:
Share BAL post-state interpretation through
BalAccountStateand keep the snap slim-account codec ineth-wire-types.Add snap/2 message-ID validation, including removal of trie-node messages, and the BAL request/response RLP types (message IDs, BAL messages).
Extend the existing snap client and network request handler with account, storage, bytecode, and BAL routing, retaining Reth’s provider abstractions and response limits (client API, server routing).
Add the
BlockAccessListsRocksDB table and the buffered persistent BAL store, then register it with the existing RocksDB provider.Expose the committed, chunked state-root rebuild from
reth-trie-dbso snap finalization and existing database initialization share the same implementation.Introduce the snap-sync crate around a provider-backed canonical chain source, hash-based ancestor walking, and pivot tracking (crate boundary, canonical chain source).
Add the authenticated range-proof verifier and account-range downloader, including omitted-leaf and ordering tests.
Add the storage-range downloader and bytecode downloader, with peer retry, request matching, and adversarial response validation.
Add the snap state generation lifecycle: clean generation start, atomic account/storage/code writes, persisted interruption marker, pipeline handoff, and final root verification (writer and acceptance, finalization).
Add the session state machine that sequences pivot selection, authenticated download, rolling target advancement, BAL catch-up, final verification, and acceptance (session runner, advancement and healing).
Integrate the bootstrap as a decorator around Reth’s existing
PipelineSync, gate it behind--snap, and verify snap-to-pipeline handoff through E2E tests (bootstrap integration, launch gating, E2E coverage).Remaining work
The snap/2 bootstrap path is implemented, but the following remains before considering EIP-8189 production-ready:
Complete reorg recovery. The session detects when the applied BAL segment leaves the canonical chain, but currently discards the generation and restarts. It does not yet roll back the old fork with retained BAL state and apply the new fork (current restart behavior).
Support mixed RLPx capabilities. Snap is currently recognized only on the dedicated ETH+snap connection variant. A connection negotiating ETH, snap, and another satellite capability is not treated as snap-capable (current capability check).
Persist orphaned BALs needed for recovery. Canonical BALs are flushed to RocksDB, while BALs retained for executed forks can remain memory-only. Durable fork BAL retention is needed by the complete rollback/reapplication path (canonical persistence handoff, persistent BAL store).
Decide interrupted-generation resumption. A restart safely detects partial state, but begins again from a clean generation rather than resuming the verified downloaded prefix (generation marker, clean generation start).
Define historical-data behavior. Snap acceptance advances pipeline and static-file frontiers without downloading pre-snap bodies, receipts, or change sets. The production design must explicitly retain checkpoint-style semantics or schedule historical backfill afterward (current acceptance behavior).
Complete production E2E coverage. Add tests for reorgs through and before the pivot, restart during state generation, persisted BAL serving after restart, malicious-peer failover, mixed capabilities, and historical RPC behavior. Existing coverage proves bootstrap, state-root equality, stale-pivot advancement, BAL serving, and normal pipeline continuation (current E2E coverage).
Enabling snap by default should follow completion of the recovery path and mainnet-scale performance testing.
cc @mattsse for review