Back off NYM mixFetch setup after a failure

[j0ntz]

Technical Design Document

zano-migration-off-module-queue.md

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

• Yes
• No

Dependencies

none

Description

QA reported an iOS test account logging itself out every minute or so with auto-logout set to one hour. The attached logs show it is not the logout timer: each re-login is preceded by EdgeContext closed and then EdgeContext opened about a second later, with no JS bundle relaunch. That is EdgeCoreWebView.webViewWebContentProcessDidTerminate calling visitPage(), so iOS killed the core WebView's WebContent process for memory and the core reloaded itself, dropping the session.

What exhausts the memory is the NYM mixFetch setup path. initMixFetch sets its cached promise back to null whenever a setup fails, so the next privacy fetch immediately builds another client. createMixFetch spawns a web worker that loads about 13 MB of WASM before it ever contacts the gateway, and the library exposes no way to terminate that worker, so each failed attempt leaves one behind. With an engine poll loop driving a request every few seconds against a gateway that is refusing the client, roughly thirty workers accumulate inside a minute and the WebContent process is killed.

In the reported logs every crash-looping session carries Initializing mixFetch... about every 2.5 seconds, each ending in gateway client error (5rXcNe2a...); the one long healthy session in the same file has no mixFetch lines at all.

This change gives a failed setup a cooldown, starting at 30 seconds and doubling to a 5 minute ceiling. Callers arriving during the cooldown reject immediately instead of building another client, so a dead gateway costs one worker per window instead of one per request.

The reported symptom needs both an account with NYM privacy enabled and a failing gateway, which is why it does not appear on a default account.

What review changed

Four rounds with @peachbits reshaped the details, and the notes are worth carrying here because each one is load-bearing:

• The cooldown is armed before the library is touched. The failure handler is no longer async: it nulls the cached promise, records the error and sets retryAfter first, then fires disconnectMixFetch().catch(() => {}) without awaiting it. Comlink's request path (index.js:342-351) builds a promise with no reject and no timeout, so a disconnect issued against a worker the OS already killed never settles. Awaiting it stranded every line below, the cooldown included, which is the freeze this PR exists to prevent. A test makes the fake disconnect hang forever and fails against the awaited form.
• A cooling-down caller gets its own error, not the stored instance. Re-throwing lastError handed callers a stack from a setup that ended minutes ago, so a 0 ms rejection read in a crash report as a fresh 60 second timeout. The rejection now names the remaining wait and carries the original as cause, which also drops the assumption that lastError is an Error at all: the library rejects with a raw MessageEvent on a worker error.
• One breadcrumb per cooldown window, not one per refused caller. The GUI caps Sentry at maxBreadcrumbs: 25 (src/app.ts:45), so a poll loop refusing every few seconds would evict the app's whole breadcrumb history inside a minute, blinding the crash report this fix exists to inform. The breadcrumb moved to the arming site and names the window actually armed. A test asserts the cardinality, since the failure mode is silent.
• The setup routine is built by a factory. makeMixFetchSetup(now = () => Date.now()) closes over the cooldown state, and export const initMixFetch = makeMixFetchSetup() keeps both io call sites unchanged. That removed a resetMixFetchForTests export and a mutable module clock from the shipped surface, and gives each test fresh state for free. Its one caveat is documented on the function: the cooldown state is per instance while the client, its worker and window.__mixFetchGlobal are process-global, so the single exported instance is the only supported arrangement.
• Dropped: clearing the cooldown on success. Those resets were unreachable. The cached promise is only nulled in the failure handler, so after a success the guard at the top of the routine is false forever and nothing reads the cooldown state again. The test that appeared to cover them went with them.

Testing

test/util/nym.test.ts drives the module against a controlled clock and counts how many clients get built. Five cases: twenty-five seconds of requests after a failure build exactly one client; a cooling-down caller is rejected with the naming error and the preserved cause; the breadcrumb fires once per window across eleven refusals and again on the next window; a retry happens once the window expires and the wait doubles after the second failure; and the cooldown still arms when the cleanup disconnect never settles. Each case was checked against the pre-fix code it guards.

The full suite reports 168 passing, npm run types is clean, and verify-repo.sh passed.

Driven in the app on an iOS simulator, with edge-core-js linked into edge-react-gui via updot and Ethereum's Network Privacy set to Nym Mixnet. The gateway failure is forced with a temporary uncommitted edit that rejects the setup after 2s, while still building the real client so each attempt still spawns its worker. Over 3m56s the engine's poll loop produced 82 privacy fetches, and the cooldown answered them with 4 client builds and 4 breadcrumbs, at 30s, 60s, 120s and 240s. The same drive with RETRY_BASE_MS set to 0 produced 47 builds and 47 breadcrumbs in 1m55s, one every ~2.4s, which is the cadence the QA logs carried. Screenshots are in the test-evidence comment below; both edits are reverted.

Still not verified: that WebContent memory stays flat across a long outage on a real device. The build rate is what this change controls and it is now measured, but the resulting memory ceiling was not observed on hardware.

Asana: https://app.asana.com/0/1215088146871429/1217733586839287
Parent: https://app.asana.com/0/1215088146871429/1217559756673909

---

Note

Medium Risk
Changes NYM privacy fetch initialization on failure paths; mis-tuned backoff could delay recovery when the gateway returns, but the behavior only affects accounts with NYM privacy and a failing setup.

Overview
Fixes an iOS symptom where NYM privacy with a failing gateway looked like repeated logouts: each failed createMixFetch left a WASM web worker alive, and initMixFetch cleared its cached promise on failure so poll-driven privacy fetches spawned a new worker every few seconds until the WebView process was killed for memory.

makeMixFetchSetup now owns per-instance cooldown state; production still uses export const initMixFetch = makeMixFetchSetup(). After a failed setup (including the existing 60s timeout), new attempts are refused until a backoff window passes—30s initially, doubling up to 5 minutes—instead of calling createMixFetch again. Callers during cooldown get a fresh “cooling down” error with the original failure on cause, not a re-thrown stale error. The cooldown is armed before best-effort disconnectMixFetch / global cleanup, which are not awaited so a hanging disconnect cannot block later callers. One Sentry breadcrumb is emitted per cooldown window. test/util/nym.test.ts covers client count, errors, breadcrumbs, backoff, and hang behavior with a fake mix-fetch module and injectable clock.

^{Reviewed by Cursor Bugbot for commit eec286f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[peachbits]

The cooldown is the right shape and the doubling matches the changelog. Four notes: one comment that misdescribes the library, one hazard the new timeout opens up in the pre-existing cleanup block, and one set of resets in the success path that turn out to be unreachable (which is also why the third test passes either way).

[j0ntz]

All four taken, no rejections. The comment at the top of the setup now describes what the library actually does, the failure handler arms the cooldown before firing an unawaited disconnect, and the unreachable success-path resets are gone with the test case that appeared to cover them.

The hang you flagged is now covered: a new case makes the fake disconnectMixFetch never settle, which times out against the awaited form and passes against the fix. Suite is 166 passing, same as before, and tsc --noEmit is clean.

[peachbits]

Correction to my earlier review. I diffed against a stale local master (2.46.1), which folded 2.47.x/2.48.0 into what I read as this PR's diff. Three notes therefore described pre-existing code as new — one of them on a factually wrong premise. Retracting that one and re-scoping the rest below. The notes on the .then resets (114-116), the third test case, and the *ForTests export were about code this PR actually adds and stand as written.

[j0ntz]

Re-scoping noted, and the three re-scoped notes are answered in their threads. Two produced code: the catch block now arms the cooldown before firing an un-awaited disconnectMixFetch, and the cooldown rejects with its own error plus a breadcrumb instead of re-throwing the stored instance. The __mixFetchGlobal comment was rewritten, but against a third reading, traced through index.js:446-449 in the installed copy, which does not match either of ours. The makeMixFetchSetup factory is declined, with the reason recorded in the code.

[j0ntz]

📸🪓 Test evidence: cooldown driven in the running app

🪓 Hack-forced evidence: the gateway failure is forced by a temporary uncommitted edit that rejects the setup after 2s (the real client is still built, so each attempt still spawns its worker); the second column additionally sets RETRY_BASE_MS=0 to reproduce the pre-fix cadence. Both edits reverted, worktree clean. Temporary uncommitted edit, reverted before commit; the marked frames prove the rendering, not the trigger.

🪓 HACK-FORCED: nym mixnet enabled

🪓 HACK-FORCED: nym cooldown ab

Captured by the agent's in-app test run (build-and-test).