Harden event-watcher boot: resolve boot ledger in the retried poll loop#130
Merged
Conversation
… loop A transient RPC failure while resolving the boot start ledger could permanently disable the watcher. start() resolved the boot ledger via an un-retried getHealth() RPC before scheduling the poll loop; if that call rejected, start() rejected, the loop never ran, the error was swallowed, and the watcher stayed dead for the life of the process — blinding the provider to all live on-chain membership/channel events until a manual restart. Move boot-ledger resolution into the first poll(), inside the existing try/catch + retry machinery. A transient failure is now caught and retried on the next tick, exactly like any poll error, instead of being terminal. Boot-sync semantics are unchanged: oldest available, or the pinned BOOT_SYNC_START_LEDGER_BLOCK override — never "latest"; converge-by-query stays the recovery path; no durable-cursor behaviour. Add a test proving a transient failure on the first boot resolution does not kill the watcher: it retries and, once the RPC recovers, polls and processes events normally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The event-watcher could be permanently disabled by a single transient RPC failure at boot.
start()resolved the boot start ledger via an un-retriedgetHealth()RPC (resolveBootStartLedger, taken whenBOOT_SYNC_START_LEDGER_BLOCKis unset / "sync-all") before scheduling the poll loop. If that RPC blipped,start()rejected beforescheduleNext(), the poll loop never ran, the error was logged-and-swallowed bystartEventWatcher, and the watcher stayed dead for the life of the process — blinding the provider to all live on-chain events (provider add/remove, channel enable/disable) until a manual restart. Poll-loop errors, by contrast, were already caught and retried; only the start path was terminal.Fix
Move boot-ledger resolution out of
start()and into the firstpoll(), inside the existing try/catch + retry machinery. A transient failure resolving the boot position is now caught and retried on the next tick — exactly like any poll error — instead of rejectingstart().start()no longer makes an un-retried network call; the in-memory cursor staysnulluntil the first poll resolves it.Semantics preserved
BOOT_SYNC_START_LEDGER_BLOCKoverride — never falls back to "latest".Test
EventWatcher - transient boot-ledger failure retries instead of killing the watcher: the firstgetHealth()throws → the watcher does not die → on RPC recovery it resolves the boot ledger, polls, and processes aprovider_addedevent; the cursor advances. Verified failing against the pre-fix code and passing after.Verification (Deno 2.8.2, the CI toolchain)
deno fmt --check— cleandeno lint— cleansrc/http/v1/pay/) — 116 passedPatch version bump
0.9.6 → 0.9.7in this PR.