Skip to content

fix(exex): skip missing notification files when filling the WAL block cache - #17

Closed
jowparks wants to merge 2 commits into
backport/v2.5.1/mainfrom
jowparks/exex-wal-skip-missing-notification
Closed

fix(exex): skip missing notification files when filling the WAL block cache#17
jowparks wants to merge 2 commits into
backport/v2.5.1/mainfrom
jowparks/exex-wal-skip-missing-notification

Conversation

@jowparks

@jowparks jowparks commented Aug 25, 2026

Copy link
Copy Markdown

A hole in the ExEx WAL file ID range currently makes the node unbootable: fill_block_cache walks min..=max and propagates WalError::FileNotFound out of Wal::new, aborting startup. Warn and skip the missing file instead; other WalError variants still abort.

Hit on the Base shadow builder, which crash-looped on boot until <datadir>/exex/wal was deleted by hand:

called `Result::unwrap()` on an `Err` value: notification 251460 not found

Holes are reachable because WalInner::commit consumes the next file ID before writing, so a failed write (the shadow builder had filled its SSD) burns the ID permanently. Not addressed here.

Second commit fixes two unrelated clippy lints in reth-network that a newer toolchain flags; they fail CI on this branch regardless of this change. deny is also red on this branch for RUSTSEC-2026-0258 (h2), which needs a lockfile bump and is out of scope here.

… cache

`WalInner::commit` consumes the next file ID before writing the notification,
so a failed write (e.g. a full disk) leaves a permanent hole in the file ID
range. On the next boot `fill_block_cache` walks `min..=max` and fails with
`WalError::FileNotFound`, which propagates out of `Wal::new` in
`ExExLauncher::launch` and aborts node startup. The node then crash-loops
until the WAL directory is deleted by hand.

Log a warning and skip the missing file instead. Other `WalError` variants
still abort as before.
`needless_range_loop` in `testnet.rs` and `needless_bool` in `fetcher.rs`
fail CI on this branch independently of the WAL change.
@jowparks jowparks closed this Aug 25, 2026
@jowparks
jowparks deleted the jowparks/exex-wal-skip-missing-notification branch August 25, 2026 21:45
@jowparks

Copy link
Copy Markdown
Author

Closing in favour of an upstream report: paradigmxyz#26813.

The diagnosis in this PR is wrong about the cause. It blames WalInner::commit burning a file id when a write fails (full disk). That path is real but rare, and is not what we hit.

The actual cause is Wal::finalize. BlockCache::notification_max_blocks is a min-heap keyed by (max_block, file_id), so remove_before evicts by block number while file ids are assigned by write order. A ChainReverted from an execution-stage unwind carries the abandoned fork's high tip, which pins those files while later, lower-height commits get evicted from underneath them — deleting a file from the middle of the id range. The read path then assumes min..=max is dense and startup dies. No disk failure involved.

There is a deterministic unit test in the upstream issue that reproduces this on unmodified main with no filesystem tampering.

The warn!-and-skip fix here is still roughly the right shape, but it is option 2 of 3 upstream; enumerating the ids that actually exist is the better fix. Leaving that to upstream rather than carrying a fork patch built on a wrong rationale.

Branch jowparks/exex-wal-skip-missing-notification stays pushed if we need to revive it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant