Stop v2 rates failing silently when the currency code map has not synced

[peachbits]

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

• Yes
• No

Dependencies

none

Description

On 2026-07-29 an unattended glibc upgrade at 06:52:44 had needrestart bounce couchdb, redis-server and pm2 within one second of each other. ratesServer was listening again at 06:52:56, ahead of CouchDB, and every synced document failed to load. rates-wusa1 then answered /v2/exchangeRate with a null rate and a 200 status for roughly 17 hours, invisible to the heartbeat.

The cause is that asV2CurrencyCodeMapDoc defaults data to {}, so a document that never loaded is indistinguishable from a loaded one at the call site. currencyCodeToAsset then falls back to a fabricated pluginId — BTC rather than bitcoin — which resolves to no rate, and neither getRates nor ratesV3 throws on a miss.

Fall back to the bundled v2 currency code map. Prefer data/v2CurrencyCodeMap.json, the file that seeds rates_settings, over an empty map, so a CouchDB outage during startup cannot turn every v2 rate into null. Had this been in place on 2026-07-29, wusa1 would have answered those 17 hours of requests correctly.

hasV2CurrencyCodeMapSynced() keys on the map holding entries rather than on a sync having resolved, because syncedDocument.sync() resolves — and emits — even when it creates an empty document because none existed yet. getV2CurrencyCodeMap() calls that same predicate, so the two agree by construction.

When the map has not synced the heartbeat posts a Slack alert but deliberately still reports healthy. The load balancer health checks / every 5s with health_status 2xx, so returning 503 there would pull the instance at exactly the moment the bundled map is keeping v2 correct, and would pull every instance at once during a shared CouchDB outage — taking v3 down with it. Failing open and alerting is the better trade.

Retry synced documents that fail their initial load. bootstrapV3SyncedDocs collects its syncs with Promise.allSettled, so the bootstrap resolves successfully even when every document fails, leaving the 30 minute refresh interval as the only recovery path. Failed documents now retry on a five second to five minute backoff, and each rejection reason is logged at bootstrap rather than only on the next interval. This also covers crosschain, which has no bundled fallback and whose absence silently skips cross-chain canonicalization on v3 as well as v2.

Seed v2CurrencyCodeMap in the shape its cleaner expects. Unrelated to the incident, but more dangerous. setupDatabases.ts seeded the bare map parsed straight from the JSON file; run through the cleaner that shape yields 0 entries, against 485 when the map is wrapped in data. setupDatabase overwrites any document that does not match what it is asked to seed, and syncedDocument.sync() writes back a normalized document whenever the round-tripped value differs from what is stored. Running yarn setup against a live database would therefore have written the bare map, and the next sync would have overwritten the document with {"data":{}} — breaking every v2 rate lookup on every instance, with an ordinary _rev bump and no error raised anywhere.

The cleanFailStrategy / onCleanFail options added in edge-server-tools 0.2.24, which this repo pins, do not cover that case: they only fire when the cleaner throws, and asMaybe swallows the shape mismatch instead of rejecting.

Testing

• tsc clean, eslint clean on changed files, 82 tests passing (77 existing plus 5 new).
• test/v2CurrencyCodeMap.test.ts covers the bundled fallback, preference for the synced map once loaded, both branches of hasV2CurrencyCodeMapSynced, and that BTC resolves to bitcoin rather than a fabricated pluginId while unsynced.
• Seeder shape verified through the real cleaner: bare map produces 0 entries, wrapped produces 485.
• heartbeatV3 is not invoked directly by the suite: its unsynced branch calls slackPoster, and serverConfig.json is gitignored, so a developer with a live webhook would have the test run posting to Slack. The predicate driving that branch is covered both ways instead.

Known gaps, not addressed here

hasV2CurrencyCodeMapSynced() cannot detect CouchDB failing after a healthy start — v2CurrencyCodeMapSyncDoc.doc.data stays populated in memory once loaded, so the instance goes on reporting healthy while couch-backed historical lookups fail on both v2 and v3. Catching that needs the heartbeat keyed on sync recency, or a genuine CouchDB health check at the ops layer.

Because the bundled JSON is a seed rather than a maintained mirror, yarn setup against a live database will now revert entries curated directly in CouchDB to whatever the checked-in file holds. That is a revert rather than the pre-fix wipe, but the overwrite path is still live; dropping the document from the seed list for existing databases would close it.

Not covered here

This is the application half only. The restart ordering that triggered the incident (needrestart bouncing CouchDB, Redis and pm2 with no dependency ordering) and CouchDB refusing port 5984 for roughly 17 hours while systemd reported Started both remain open, and neither is fixable in this repo.

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1217799336105986

[j0ntz]

Reviewed the v2 fallback and synced doc retry work. Verified locally on 61b4dd9: the test suite passes (82 tests) and build.types is clean.

The one that matters is on retrySyncUntilLoaded: it stops retrying on a resolved sync(), and sync() resolves after creating an empty document, which is the exact semantic this PR documents on hasV2CurrencyCodeMapSynced. The rest are cleanups.

Two commit subjects are over the 50 character limit:

• Seed v2CurrencyCodeMap in the shape its cleaner expects (55). Suggested: Seed v2CurrencyCodeMap in its cleaner's shape (44)
• Retry synced documents that fail their initial load (51). Suggested: Retry synced docs that fail their initial load (45)

[j0ntz]

Warning: this success branch stops retrying on any resolved sync(), but a resolved sync does not mean the document loaded. That is the same semantic hasV2CurrencyCodeMapSynced documents over in router.ts.

syncedDocument.sync() in edge-server-tools swallows a not-found and inserts a fresh empty document:

const { _id, _rev, ...rest } = await db.get(id).catch(error => {
  if (asMaybeNotFoundError(error) == null) throw error
  return { _id: id, _rev: undefined }
})
// ...
if (_rev == null || !matchJson(dirty, rest)) { await db.insert({ _id, _rev, ...dirty }) }

So against a fresh or wiped rates_settings, the retry logs recovered synced doc and stops while the map is still empty. The insert is durable, so every later sync finds a matching { data: {} } and the 30 minute interval never populates it either. v2 serves the bundled fallback and the Slack alert repeats until someone reruns the setup script.

sequenceDiagram
  participant Boot as bootstrapV3SyncedDocs
  participant Retry as retrySyncUntilLoaded
  participant Couch as CouchDB
  Boot->>Couch: sync(v2CurrencyCodeMap)
  Couch--xBoot: ECONNREFUSED
  Boot->>Retry: schedule, 5s backoff
  Note over Couch: CouchDB returns,<br/>document was never created
  Retry->>Couch: sync()
  Couch-->>Retry: not_found, inserts { data: {} }, resolves
  Retry->>Retry: logs "recovered synced doc", stops
  Note over Couch: document now exists and matches
  Boot->>Couch: 30 minute interval sync()
  Couch-->>Boot: { data: {} }, unchanged
  Note over Boot: hasV2CurrencyCodeMapSynced() stays false

Loading

The helper is generic and most synced docs are legitimately allowed to be empty, so the fix is probably an optional isLoaded: () => boolean parameter (defaulting to always true) that the success branch checks before it stops backing off. v2CurrencyCodeMapSyncDoc would pass hasV2CurrencyCodeMapSynced.

[peachbits]

Fixed in 1623b9f, taking the isLoaded parameter you suggested — optional, defaulting to always-true, so documents that may legitimately be empty are unaffected. syncedDocs.ts maps v2CurrencyCodeMap → hasV2CurrencyCodeMapSynced through a Record. A resolved-but-empty sync now logs and keeps backing off instead of stopping.

The durability point is the part I had missed: I flagged the resolve-vs-loaded confusion in this helper while reviewing, then only fixed the router.ts half. The insert being durable is what turns it from slow into permanent.

[j0ntz]

Warning: the bundled map is the only source of v2 mappings during an outage, and it is the one path that skips asV2CurrencyCodeMap. The synced document goes through asV2CurrencyCodeMapDoc (types.ts:199), so a malformed entry there fails loudly and lands in the onCleanFail Slack alert. A malformed entry in data/v2CurrencyCodeMap.json ships silently under as V2CurrencyCodeMap and only surfaces as wrong v2 conversions while CouchDB is down.

asV2CurrencyCodeMap accepts the file as-is today (checked locally, all 485 entries clean), so this is a one line change that turns a silent drift risk into a startup failure:

const bundledV2CurrencyCodeMap = asV2CurrencyCodeMap(bundledV2CurrencyCodeMapJson)

It also proves the seed and the fallback agree on shape, which is the guarantee the comment above is asserting.

[peachbits]

Fixed. Combined with your setupDatabases.ts comment into a new src/v3/bundledCurrencyCodeMap.ts that loads and cleans the file once, imported by both router.ts (97f6b34) and setupDatabases.ts (8ddea60). Kept independent of router.ts as you suggested, so the setup script doesn't pull in config, redis and every provider.

Confirmed the cleaned map deep-equals the raw JSON, so what setupDatabase seeds and what matchJson compares are byte-identical to before — the change is validation only, not content.

[j0ntz]

Warning: this entry says the opposite of what the code does. heartbeatV3 deliberately stays 2xx when the map has not synced and only posts to Slack (router.ts:297-305), with a comment explaining that failing here would pull every instance during a shared CouchDB outage. Shipping this line tells operators the load balancer removes the instance when it does not.

Suggested:

- fixed: Alert on Slack when the v2 currency code map has never synced, while keeping the heartbeat healthy so a CouchDB outage does not pull every instance from the load balancer

[peachbits]

Fixed with your wording, in 8ddea60.

One note on where it landed: I first put it in the 0898976 fixup, where it belongs semantically, and the autosquash failed. The diff's trailing context is the Retry… / Log… / Seed… entries, which don't exist yet at that point in history, so it had to ride with the last fixup instead. The final squashed tree is the same either way.

[j0ntz]

Suggestion: watchDatabase on line 65 runs its own initial sync over the full document list before it resolves:

await Promise.all(syncedDocuments.map(async doc => await doc.sync(db)))

So during an outage the same failed documents are being synced by two paths in the same startup sequence.

The more useful consequence: that await is what rejects, so the catch on line 72 logs Failed to start rates_settings watcher and nothing retries the watcher. The retry added here recovers the documents but leaves the process with no changes feed for the rest of its life, which is the more expensive half of the same restart race. Worth putting watchDatabase on the same backoff.

[peachbits]

Fixed in 6eda83f, but narrower than this. Laying out why, since it's a disagreement about mechanism rather than about whether to fix it.

The feed is started before the rejection. In watch-database.ts, db.changesReader.start(...) is synchronous — it kicks off the async work() loop and returns. The await Promise.all(...) is a separate, later statement. So when that rejects, the feed is already running; the rejection is the redundant initial sync failing, not the feed failing to start.

nano retries connection errors itself (changesreader.js:214-225):

if (err && err.statusCode && err.statusCode >= 400 && err.statusCode !== 429 && err.statusCode < 500) {
  self.continue = false
} else {
  delay = delay ? Math.min(60000, delay * 2) : 5000
}

ECONNREFUSED carries no statusCode, so a restarting CouchDB takes the retry branch and backs off 5s→60s until it reconnects. The restart race does not leave the process without a changes feed.

What does is the fatal branch: a 4xx other than 429 sets continue = false, the loop exits and calls setDefaults(), which replaces this.ee with a fresh EventEmitter — so the handlers watchDatabase registered are orphaned and nothing re-arms. Plausible trigger is CouchDB answering 401 while _users is still loading, i.e. the same restart race through a different door. That is what 6eda83f restarts, on a 5s→5min backoff.

I avoided the blanket retry because start() guards on self.started and returns the existing emitter, so retrying a live feed would silently do nothing.

Also fixed the logging, which I think is the more valuable half: Failed to start rates_settings watcher was firing for a feed that was running fine and would recover — actively misleading during an incident. And the watcher no longer blocks bootstrap, since its initial sync duplicates the Promise.allSettled above it.

If you read the nano source differently I'll take the broader retry.

[j0ntz]

Suggestion: slackPoster throttles on a single global lastText / lastDate pair (utils/postToSlack.ts). This new call shares that one slot with 'Rates server heartbeat failed' further down the same function. During a CouchDB outage both conditions hold, the two texts alternate on every health check, each one resets lastText, and the 5 minute throttle never applies to either. Caddy polls this route continuously, so the channel floods during exactly the outage the alert exists to report.

Gating on the transition posts once per state change instead:

let warnedUnsynced = false
// ...
const synced = hasV2CurrencyCodeMapSynced()
if (!synced && !warnedUnsynced) {
  warnedUnsynced = true
  slackPoster('...').catch((error: unknown) => { console.error(error) })
}
if (synced) warnedUnsynced = false

[peachbits]

Fixed in 97f6b34 with the transition gating you suggested.

You're right, and I had understated it: I flagged the volume while reviewing but assumed the 5 minute throttle would hold at roughly one message per process per interval. It doesn't — the two texts alternate, each resets lastText, and neither is ever suppressed.

One refinement: the flood needs both alerts firing. If only the unsynced one does, lastText stays constant and the throttle works normally. It fails worst in the everything-is-broken case, which is where you least want the channel flooded.

[j0ntz]

Suggestion: the bundled map now loads twice through two different mechanisms: readFileSync plus JSON.parse here, and a JSON import at router.ts:5. The comment at router.ts:25 asserts the fallback is "the same data the synced document would have loaded", which currently holds only by convention.

A small module exporting the cleaned map, imported by both, makes that structural and lets this line read { data: bundledV2CurrencyCodeMap }. Worth keeping it separate from router.ts rather than importing from there, since this script would otherwise pull in config, redis, and every provider.

[peachbits]

Fixed — see the reply on router.ts:28. New src/v3/bundledCurrencyCodeMap.ts exports the cleaned map, imported by both this file and router.ts, and deliberately kept independent of router.ts so this script stays light. readFileSync and join are gone from here.

[peachbits]

Thanks for the review — all six addressed, and each claim checked against the source before acting rather than taken on faith. Every one held up.

Comment	Landed in
retrySyncUntilLoaded stops on a resolved-but-empty sync	1623b9f
Bundled map skips asV2CurrencyCodeMap	97f6b34 + 8ddea60
CHANGELOG entry contradicts the code	8ddea60
watchDatabase failure never retried	6eda83f — narrower, see thread
Slack throttle collision	97f6b34
Bundled map loaded twice	97f6b34 + 8ddea60

The first three and the last are fixups against the commit that introduced the code. The watcher fix is a standalone commit since it is a distinct change rather than a correction. git rebase --autosquash dry-runs clean to a four-commit branch with a tree identical to the current head.

Two of these were mine to have caught. The cleaner-validation gap I raised while reviewing my own diff and skipped as having no live trigger — then made it live with the fail-open heartbeat change and did not revisit. The retrySyncUntilLoaded semantics I wrote down explicitly and then fixed only the router.ts half of.

The one worth your attention is the watchDatabase thread. nano already retries connection errors on its own 5s→60s backoff, and the feed is started before the rejection you identified, so the restart race does not leave the process without a changes feed. The genuine permanent-death path is narrower — a 4xx other than 429, where nano exits its loop and swaps in a fresh event emitter, orphaning the handlers. That is what the fix targets. If you read changesreader.js differently, say so and I will take the broader retry.

Leaving the two long commit subjects as they are for now.

tsc clean, 82 tests passing, eslint clean on changed files.