Skip to content

Broadcast to every server at once, and stop leaking the NOWNodes key - #458

Open
peachbits wants to merge 2 commits into
masterfrom
matthew/broadcast-all-servers
Open

Broadcast to every server at once, and stop leaking the NOWNodes key#458
peachbits wants to merge 2 commits into
masterfrom
matthew/broadcast-all-servers

Conversation

@peachbits

@peachbits peachbits commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

Engine half of the "send failed in the UI but the money moved" incident. Asana: https://app.asana.com/1/9976422036640/project/1213880789473005/task/1217135300337949

GUI companion (independent, neither blocks the other): EdgeApp/edge-react-gui#6190

Two separate problems in ServerStates.broadcastTx, one commit each. The first commit stands alone and can be reviewed, or landed, without the second.


1. Send the NOWNodes API key only to NOWNodes servers

ServerConfig had a single type, blockbook-nownode, so when Edge's own Blockbook hosts were added in Dec 2024 (9962d7e) they were filed under it alongside the real NOWNodes entries. Every HTTP broadcast therefore sent the NOWNodes key to Edge's servers too.

They are not the same kind of server:

GET https://btc-wusa1.edge.app/api/v2   -> 200, no key needed
GET https://btcbook.nownodes.io/api/v2  -> 401 Unknown API_key

The type becomes blockbook | blockbook-nownode. The ten edge.app HTTP entries are now plain blockbook, the nownodes.io entries are unchanged, and the key is attached per target instead of to the whole batch.

This also fixes the no-key case. A missing nowNodesApiKey previously rejected the entire HTTP path with "Missing connection key for fallback servers", even though the public servers need no key. NOWNodes targets are now skipped with a warning and the public servers still carry the broadcast.

2. Broadcast to every server at once instead of gating HTTP on the sockets

The HTTP servers were used only when no WebSocket reported itself connected. A socket that looks connected but never answers sendTransaction therefore failed the whole broadcast, after the 30 second request timeout, without a single HTTP attempt being made. That is the shape of the incident behind this task.

The gate is worse than it looks: the NOWNodes WebSocket URI is docked 400 points at score load (ServerScores.serverScoresLoad penalises any URI carrying key params) and so rarely wins one of the two connection slots. HTTP is the realistic route to those servers, and it was reachable only when every socket was visibly down.

broadcastTx now fires at every cached blockbook and every HTTP server in parallel. Sockets still connecting are included, since a queued request transmits as soon as the socket opens and times out otherwise. The first success resolves; it rejects only once every attempt has failed, logging each failure against its server.

Testing

New test/common/utxobased/engine/ServerStates.spec.ts, seven tests in the first commit and one more in the second:

  • broadcasting over every HTTP server when no sockets are cached
  • first success wins when one server fails and another succeeds
  • rejecting only after every attempt fails
  • rejecting immediately when there is nothing to broadcast to
  • the api-key header reaching NOWNodes servers and not the public ones
  • skipping NOWNodes but still using public servers when no key is configured
  • rejecting when only NOWNodes servers exist and no key is configured
  • (second commit) a real WebSocket server that accepts sendTransaction and never answers, while HTTP still resolves in well under the 30 second socket timeout

Full suite: 1261 passing at the first commit, 1262 at the second. tsc and eslint clean at both.

Note for reviewers

This deliberately does not include the broadcast-failure classification from #455. Per the direction agreed for this task, the app now hard-fails an ambiguous broadcast and says so, rather than trying to infer whether the transaction landed. The saveTx "No addresses to process" fix from #455 is still valid and is not in this branch.

`ServerConfig` had a single type, `blockbook-nownode`, so when Edge's own
Blockbook hosts were added in Dec 2024 (9962d7e) they were filed under it
alongside the real NOWNodes entries. Every HTTP broadcast therefore sent
the NOWNodes key to Edge's servers as well.

They are not the same kind of server:

    GET https://btc-wusa1.edge.app/api/v2   -> 200, no key needed
    GET https://btcbook.nownodes.io/api/v2  -> 401 Unknown API_key

The type becomes `blockbook | blockbook-nownode`. The ten edge.app HTTP
entries are now plain `blockbook`, the nownodes.io entries are unchanged,
and the key is attached per target rather than to the whole batch.

This also fixes the no-key case. A missing `nowNodesApiKey` previously
rejected the entire HTTP path with "Missing connection key for fallback
servers", even though the public servers need no key. NOWNodes targets
are now skipped with a warning and the public servers still carry the
broadcast; the reject is reserved for having no usable server at all.

Adds test/common/utxobased/engine/ServerStates.spec.ts covering the
header routing, both no-key cases, and the existing HTTP broadcast
behavior it has to preserve.
`broadcastTx` used the HTTP servers only when no WebSocket reported
itself connected. A socket that looks connected but never answers
`sendTransaction` therefore failed the entire broadcast, after the 30
second request timeout, without a single HTTP attempt being made. That
is the shape of the incident this task exists for (Asana
1217135300337949): the send failed in the UI while the transaction was
already on the network.

The gate is worse than it looks, because the NOWNodes WebSocket URI is
docked 400 points at score load (ServerScores.serverScoresLoad penalises
any URI carrying key params) and so rarely wins one of the two
connection slots. HTTP is the realistic route to those servers, and it
was reachable only when every socket was visibly down.

broadcastTx now fires at every cached blockbook and every HTTP server in
parallel. Sockets that are still connecting are included, since a queued
request transmits as soon as the socket opens and times out otherwise.
The first success resolves. It rejects only once every attempt has
failed, logging each failure against the server that produced it.

Extends the ServerStates spec with a real WebSocket server that accepts
`sendTransaction` and never answers, asserting the broadcast still
resolves over HTTP in well under the socket timeout.
@peachbits
peachbits force-pushed the matthew/broadcast-all-servers branch from fa5f837 to f809273 Compare September 3, 2026 05:09
@peachbits
peachbits marked this pull request as ready for review September 3, 2026 19:52

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed alongside EdgeApp/edge-react-gui#6190; the findings are inline.

Nit: the commit subject Broadcast to every server at once instead of gating HTTP on the sockets is 71 characters, over Edge's 50-character limit. Broadcast to every server at once is 33 and the body already carries the rest. (Send the NOWNodes API key only to NOWNodes servers is exactly 50 and fine.)

if (blockbook == null) continue
attempts++
blockbook
.broadcastTx(transaction)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failing WebSocket attempt never settles, so this promise can hang instead of rejecting.

blockbook.broadcastTx goes through promisifyWsMessage (Blockbook.ts), which awaits a Deferred that is only ever resolved from inside its own generator. Every failure path in Socket.ts delivers the error as request.generator.throw(err).catch(e => log.error(e.message)): the error-response branch in onMessage, the 30s expiry in onTimer, and onSocketClose. That generator has no try/catch, so the rejection is logged and the deferred is left pending forever. No onFailure ever runs for that attempt, failures never reaches attempts, and the promise settles neither way.

sequenceDiagram
    participant SS as ServerStates.broadcastTx
    participant Sock as Socket blockbook
    participant Http as HTTP servers
    SS->>Sock: broadcastTx, attempts = 1
    SS->>Http: sendtx, attempts = 2
    Http-->>SS: rejects, onFailure, failures = 1
    Sock->>Sock: 30s expiry, generator.throw
    Sock->>Sock: Deferred left pending, no onFailure
    SS->>SS: failures 1 < attempts 2, never settles
Loading

The deferred bug is pre-existing, but the denominator change is what exposes it. Before this commit, the fully-disconnected case took the !isAnyBlockbookConnected branch, skipped sockets entirely and rejected off the HTTP count, so a genuinely failed broadcast surfaced as a failure. Sockets are now always counted, so an all-fail broadcast with any entry in serverStatesCache never rejects. Paired with EdgeApp/edge-react-gui#6190 that is a locked slider spinning forever with no message at all, since the scene's finally never runs.

The new spec covers socket-hangs-but-HTTP-succeeds; it is the all-fail variant that hangs. Either bound each attempt before counting it, or have promisifyWsMessage reject its deferred when the generator throws.

'api-key': nowNodesApiKey
},
body: transaction.signedTx
for (const { uri, headers } of httpTargets) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every send now discloses the transaction to the NOWNodes HTTP endpoint even when the sockets and Edge's own servers are healthy, where HTTP was previously a fallback only. For BTC that is btc-wusa1.edge.app, btc-eu1.edge.app and btcbook.nownodes.io on every broadcast, unconditionally.

Unblocking HTTP is the right call; making it unconditional on the happy path is a separate product decision that the PR description does not call out. Worth confirming it is intended, or staggering the third-party targets behind a short delay so they only fire when the first wave has not resolved.

@j0ntz j0ntz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correcting the verdict on my earlier review: those findings are blocking, not advisory.

The broadcast hang is the one that gates this. An all-fail broadcast with any entry in serverStatesCache settles neither way, and paired with EdgeApp/edge-react-gui#6190 the send scene has no error to show and no finally to unlock the slider.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants