Skip to content

Harden event-watcher boot: resolve boot ledger in the retried poll loop#130

Merged
AquiGorka merged 2 commits into
mainfrom
fix/watcher-boot-resilience
Jun 24, 2026
Merged

Harden event-watcher boot: resolve boot ledger in the retried poll loop#130
AquiGorka merged 2 commits into
mainfrom
fix/watcher-boot-resilience

Conversation

@AquiGorka

Copy link
Copy Markdown
Contributor

Problem

The event-watcher could be permanently disabled by a single transient RPC failure at boot. start() resolved the boot start ledger via an un-retried getHealth() RPC (resolveBootStartLedger, taken when BOOT_SYNC_START_LEDGER_BLOCK is unset / "sync-all") before scheduling the poll loop. If that RPC blipped, start() rejected before scheduleNext(), the poll loop never ran, the error was logged-and-swallowed by startEventWatcher, and the watcher stayed dead for the life of the process — blinding the provider to all live on-chain events (provider add/remove, channel enable/disable) until a manual restart. Poll-loop errors, by contrast, were already caught and retried; only the start path was terminal.

Fix

Move boot-ledger resolution out of start() and into the first poll(), inside the existing try/catch + retry machinery. A transient failure resolving the boot position is now caught and retried on the next tick — exactly like any poll error — instead of rejecting start(). start() no longer makes an un-retried network call; the in-memory cursor stays null until the first poll resolves it.

Semantics preserved

  • Boot-sync from oldest available, or the pinned BOOT_SYNC_START_LEDGER_BLOCK override — never falls back to "latest".
  • Converge-by-query stays the recovery path; no durable-cursor behaviour changed.
  • This is purely about making the start path retry instead of dying.

Test

EventWatcher - transient boot-ledger failure retries instead of killing the watcher: the first getHealth() throws → the watcher does not die → on RPC recovery it resolves the boot ledger, polls, and processes a provider_added event; the cursor advances. Verified failing against the pre-fix code and passing after.

Verification (Deno 2.8.2, the CI toolchain)

  • deno fmt --check — clean
  • deno lint — clean
  • event-watcher suite — 56 passed
  • full unit suite — 133 passed
  • CI pay suite (src/http/v1/pay/) — 116 passed

Patch version bump 0.9.6 → 0.9.7 in this PR.

… loop

A transient RPC failure while resolving the boot start ledger could
permanently disable the watcher. start() resolved the boot ledger via an
un-retried getHealth() RPC before scheduling the poll loop; if that call
rejected, start() rejected, the loop never ran, the error was swallowed,
and the watcher stayed dead for the life of the process — blinding the
provider to all live on-chain membership/channel events until a manual
restart.

Move boot-ledger resolution into the first poll(), inside the existing
try/catch + retry machinery. A transient failure is now caught and
retried on the next tick, exactly like any poll error, instead of being
terminal. Boot-sync semantics are unchanged: oldest available, or the
pinned BOOT_SYNC_START_LEDGER_BLOCK override — never "latest";
converge-by-query stays the recovery path; no durable-cursor behaviour.

Add a test proving a transient failure on the first boot resolution does
not kill the watcher: it retries and, once the RPC recovers, polls and
processes events normally.
@AquiGorka AquiGorka merged commit 8c8f0ee into main Jun 24, 2026
7 checks passed
@AquiGorka AquiGorka deleted the fix/watcher-boot-resilience branch June 24, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant