Skip to content

Frontend reconnect hardening: tab wake & reconnect probe - #172

Open
ndisidore wants to merge 2 commits into
mainfrom
fix/reconnect-hardening
Open

Frontend reconnect hardening: tab wake & reconnect probe#172
ndisidore wants to merge 2 commits into
mainfrom
fix/reconnect-hardening

Conversation

@ndisidore

@ndisidore ndisidore commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Handles 2 cases the server cannot automagically help with recovery on:

  • Reconnecting flicker: probe each candidate with getServerConfig() and only publish a stub that has round-tripped. This stops false reporting a reconnected state just to go back to disconnected
  • Probe the connection on tab wake. visibilitychange/online triggers a probe of the current stub
Screenshot from 2026-08-12 11-48-33 _note reconnecting chip_

Testing

Types, unit tests, lint. Manually against the local dev stack:

  • Outage (backend down ~3 min with a workspace open): chip appears once, no flicker; exactly one RPC connection lost / one RPC connection restored; measured attempts at +0.2s, +1.3s, +3.2s, +6.8s, +14s, then ~10s apart (correct jittered doubling backoff); chat works after recovery.
  • Zombie (kill -STOP on workerd, then a wake event): probe times out at ~10s, disposes the stub, and the app recovers on its own once the process thaws (no user action needed).
  • Tab-switch noise (counted probe frames by hooking WebSocket.send): a burst of 8 wake/online events yields exactly 1 probe past the 15s idle window, 0 within it.

handleBroken published a fresh stub every backoff cycle before it had
round-tripped: capnweb queues sends while the socket is CONNECTING, so
the unproven stub looked fine until every effect pipelined onto it
failed at once, once per cycle. And because useAuth pipelines
authenticate() without awaiting, markConnectionRestored() fired
instantly and the Reconnecting chip flickered off/on all outage long.

Rework handleBroken into a recovery loop that probes each candidate
with getServerConfig() (20s timeout) and only publishes it once the
probe round-trips. During the outage the published stub stays the dead
one, so stub-keyed effects don't re-fire per attempt; subscribers hear
exactly twice per outage — lost, then restored. The proof replaces
markConnectionRestored(), so delete it and its __root.tsx effect.

Keeps today's fast recovery from one-off blips: the first backoff is
skipped when the dying connection had been up longer than the initial
backoff.

The workspace editor's Reconnecting chip previously appeared only as a
side effect of the churn: each republish re-fired the workspace-open
effect, whose failure set the workspace-level connectionLost. With the
dead stub staying published, drive the chip from the socket-level flag
too.
Nothing detected a socket killed during laptop sleep or background tab
throttling: the user woke the tab and their first action hung on a
zombie socket before recovery even started.

On tab-visible and network-online signals, probe currentStub with
getServerConfig() under a 10s timeout — skipped while reconnecting,
while a probe is already in flight, or when the connection was proven
alive within the last 15s (so rapid tab switches stay silent). On
timeout, dispose the stub; that fires onRpcBroken and the recovery
loop takes over, whose skip-first-backoff path retries immediately —
right for "the network just came back".
@github-actions github-actions Bot added the workshop/frontend Changes to the Workshop frontend label Aug 12, 2026
@ndisidore ndisidore changed the title Frontend reconenct hardening Frontend reconnect hardening: tab wake & reconnect probe Aug 12, 2026
@ask-bonk

ask-bonk Bot commented Aug 12, 2026

Copy link
Copy Markdown

useEffect is still used at line 135. No unused import.

Now let me assess one final real concern: the getServerConfig() probe refetch interaction. AppWithConnection's effect (line 194-208) also calls rpcState.stub.getServerConfig() on every stub change. When probeOnWake calls getServerConfig() on the live stub, that's an extra RPC but harmless. When handleBroken swaps currentStub and notifies, the effect re-runs getServerConfig() — a second call after the probe's own call. Minor redundancy (two getServerConfig per reconnect), not a bug.

Let me double-check the jitter/backoff produces the PR's claimed measurements to validate correctness of the loop, and confirm skipSleep doesn't cause a tight infinite loop when the server is down.

Trace: server down, connection was up > 1s → skipSleep=true. First iteration: no sleep, try connect, probe fails (times out at 20s or connects-then-fails fast). If the WS fails fast (connection refused), the probe rejects quickly → continue. Second iteration: skipSleep=false, sleep ~1s, backoff→2s. This matches "+0.2s, +1.3s, +3.2s...".

But there's a subtle issue: if getServerConfig() on a refused socket rejects immediately every iteration, the loop still sleeps between attempts (backoff grows), so no tight loop. Good. However, if a candidate's WS connects but the backend hangs, the probe waits the full 20s timeout each attempt — acceptable per the comment.

I've completed a thorough review. My findings are minor/low-confidence; none are clear defects introduced by the change. Let me summarize.

Overall this is a well-constructed change. Let me note the observations I have.

Review Summary

This is a solid, well-reasoned change. The design correctly separates the two concerns (socket-level RPC state via useConnectionLost() vs. workspace-level reopen failures) and the comments are excellent at explaining the why. The handleBroken refactor to prove-before-publish is a genuine improvement over the old connect-then-hope flow, and moving isConnectionLost clearing out of the auth path (markConnectionRestored) into the probe is cleaner.

I found no clear defects introduced by this change. A few low-confidence observations (none blocking):

  1. onRpcBroken registration moved after the probe (main.tsx:133). In the old code the broken handler was attached immediately after startConnection(); now it's attached only after getServerConfig() round-trips. This introduces a narrow window (probe success → onRpcBroken registration) where a break could be dropped. This is only safe if capnweb invokes an onRpcBroken callback immediately for an already-broken stub. That's the conventional capnweb/Cap'n-Proto semantic, so this is almost certainly fine — but it's now load-bearing where it wasn't before. Worth a one-line comment noting the reliance, if the semantic is confirmed.

  2. withTimeout leaves its sleep timer running (main.tsx:79-80) after the wrapped promise wins the race. Not an unhandled rejection (Promise.race still observes the loser), but on a successful probe a stray setTimeout fires up to 20s later and throws into an already-settled race. Harmless, just slightly untidy; a clearTimeout in a finally would tighten it.

  3. Double getServerConfig() per reconnect. handleBroken probes with getServerConfig(), then swaps currentStub and notifies, which re-runs the AppWithConnection effect (main.tsx:194-208) that calls getServerConfig() again on the same stub. Redundant but cheap and correct.

  4. lastProvenAt initialized to Date.now() at module load (main.tsx:75) treats the not-yet-proven initial socket as freshly proven, so an immediate tab-switch inside the 15s window won't probe a zombie initial connection. Edge case, acceptable.

The removed markConnectionRestored/notifyCurrentStubUpdated exports have no remaining references, and the useEffect import in __root.tsx is still used elsewhere — no dangling code.

LGTM!

github run

@kentonv

kentonv commented Aug 12, 2026

Copy link
Copy Markdown
Member
  • I think there's a subtle change in behavior here. Previously, when the stub was detected broken, it was immediately replaced by a stub representing the eventual new connection, so new RPCs made while the reconnection handshake was in flight would be queued and delivered once the connection was up. After this change, I believe we leave the old, broken stub in place until after we've reconnected and probed the new connection to prove that it is good. This means in the meantime a bunch of RPCs might fail, but it would probably be better to have those RPCs wait for the new connection. I believe you should be able to accomplish this by setting the top-level stub to an RpcStub constructed from a Promise<RpcStub>, where you later resolve the promise to the final stub once the probe succeeds.

  • getServerConfig() is not exactly free on the server side. It does a KV read and some other stuff, which is all wasted in the case that it's being used as a probe. I think we should instead add a ping() method that just returns nothing.

@kentonv

kentonv commented Aug 12, 2026

Copy link
Copy Markdown
Member

Hmm, it looks like Cap'n Web actually might not currently expose a way to construct an RpcPromise/RpcStub from a Promise (or generally a Promise). We should probably add that. In raw Cap'n Proto we use it all the time.

Either RpcPromise should have a constructor from Promise or we should have some sort of makePipelined(p: Promise<T>): RpcPromise<T> function...

@ndisidore

Copy link
Copy Markdown
Contributor Author

Hmm, it looks like Cap'n Web actually might not currently expose a way to construct an RpcPromise/RpcStub from a Promise (or generally a Promise). We should probably add that. In raw Cap'n Proto we use it all the time.

Either RpcPromise should have a constructor from Promise or we should have some sort of makePipelined(p: Promise<T>): RpcPromise<T> function...

I actually have a branch that does exactly this: let me see if I can get it over the line

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

workshop/frontend Changes to the Workshop frontend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants