catch up - #1
Open
Corey-Code wants to merge 1147 commits into
Open
Conversation
…s metric The "broadcasting new block ... to N channels" line logged at default level on every indexed block, flooding the logs. Replace it with the websocket_new_block_notifications counter, incremented by the subscribeNewBlock subscriber count on each block, demote the per-block detail to V(2), and add a "Websocket new block notifications per minute" panel to the Grafana Websockets section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… only what was already fetched
… and give it higher priority
… work in different goroutines
…ip on Free tier Historical daily backfill chose its backend with a binary gate on bootstrap state: the rate-limit-free Trezor CDN during the initial bootstrap, then the rate-limited CoinGecko Free tier forever after. A restart with downtime or a stale series therefore refetched a large gap through the Free tier and got throttled. Replace the binary gate with a per-series routing table keyed on the resolved range. The Free tier is now reached only for the chain tip (gap == 1 day); every higher-volume range (multi-day backfill, first-seen/capped, full-history bootstrap) goes to the CDN when configured, falling back to the Free tier only when no CDN URL is set. Metadata calls (supported_vs_currencies, coins/list) also prefer the CDN. - resolveHistoricalDays: tip window of 1 day; 2..365-day gaps become the new "backfill" range; >365 stays "capped". - getHistoricalTicker no longer takes a baseURL; it resolves the range, picks the backend via sourceURLForRange, and shares the per-day fill loop (fillFromMarketChart, with a fill-missing-only mode for the upcoming startup reconciliation). - New metrics fiat_rates_fetched_units_total / _fetched_tokens_total / _unable_total (labelled by phase/reason) wired into the runtime path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…torical rates
Gap detection only ever extended forward from the most recent ticker, so interior
holes (a day a past cycle dropped) were never repaired, and a restart after
downtime relied on the async daily loop to slowly catch up.
Add a blocking, at-startup self-healing pass that repairs missing daily rates —
interior holes and trailing gaps — within a 365-day window for the base coin
(every vsCurrency) and every token, fetching only from the CDN and filling only
the days a series lacks (existing data is never rewritten). It runs in two stages:
1. read the DB and classify every series. A series whose latest ticker is >= 90
days old is reported as gap_too_large and skipped (a months-long gap signals a
bug/misconfiguration, not routine catch-up); first-seen series are left to the
runtime downloader's initial population.
2. fetch the window per repair candidate, fill holes, persist in batches.
Wiring:
- RatesDownloaderInterface.ReconcileHistoricalRates(windowDays, maxGapDays);
implemented by Coingecko, reusing the shared fillFromMarketChart (fill-missing).
- FiatRates.ReconcileHistoricalRatesAtStartup honors a new per-coin config flag
reconcileHistoricalAtStartup (absent => enabled) and reuses the
fiat_rates_update_duration_seconds{stage="reconcile"} metric.
- blockbook.go gains runStartupSelfHealing, invoked before the periodic downloader
loops start so reconciliation runs via the CDN without Free-tier contention.
- It is a no-op during bootstrap or when no CDN URL is configured.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a "Fiat Rates" row with four panels driven by the new fiat-rate metrics, so
the CDN/Free-tier split and the startup self-healing are observable:
- Chain tip fetches (<=1 day, Free tier) — fiat_rates_fetched_units_total{phase=tip}
- Backfill fetches (>1 day, via CDN) — units + tokens for phase=backfill|bootstrap
- Startup reconciliation (self-healing) — units + tokens repaired for phase=reconcile
- Unable to fetch (by reason) — fiat_rates_unable_total split by phase/reason
Edits the generator sources (template.json layout + panels.yaml content) only;
configs/grafana/grafana.json is git-ignored and regenerated by render_grafana.py.
Validated with render_grafana.py --check and render_grafana_test.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ash-safe Self-review of the reconciliation pass surfaced a correctness-vs-spec gap and a serious startup cost: - The pass computed a per-series trailing gap and then fetched days=365 for every series (thousands for ETH) on *every* restart just to discover there were no holes, re-writing window records and doing a per-token FiatRatesFindLastTicker full-history scan each time. That diverges from the intended "read what is missing in the DB" design and would hammer every instance on every restart. Redesign around cheap, gap-driven detection: - Stage 1 now does a key-only scan (new db.FiatRatesGetTickerTimestamps) to find which whole days are missing within the window. Zero missing days -> returns immediately and makes NO network requests (the common, healthy case). 90+ missing days -> reported as gap_too_large with the count, no backfill. A young DB (only tip-zone records) and a >window trailing gap are distinguished so a fresh DB is not mis-flagged. - Stage 2 fetches only the missing-day range (days = span back to the earliest missing day, capped to the window) and fills only those days via a new targetDays filter on fillFromMarketChart, so the write set is bounded by the (sub-90) missing-day count regardless of series count. Robustness/observability: - ReconcileHistoricalRatesAtStartup recovers from panics so a fiat-rate bug can never brick startup, and only reloads the in-memory daily cache (a full scan) when something was actually repaired. - ReconcileHistoricalRates returns the number of points filled; added stage logs for healthy / gap_too_large / missing-day / per-failure cases. Tests updated/added: interior-hole repair (base + token, present days untouched), healthy-DB makes no requests, young-DB makes no requests, gap_too_large skips all fetching, no-CDN no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plumb blockbook's chanOsSignal into the startup self-healing reconciliation pass. A SIGTERM mid-repair now aborts promptly instead of waiting for a long backfill (potentially thousands of token series) to finish: - check the stop channel once after the cheap stage-1 scan (before any network call) and again between each series fetch in stage 2 - on abort, persist the days already fetched so the next startup resumes from the remaining gap rather than refetching from scratch, and log it - nil stop channel (tests) is never stopped Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…et via context
Thread a context.Context through the CoinGecko fetch path so the blocking
startup reconciliation can never stall startup. The context carries both the
shutdown signal and a configurable wall-clock budget (reconcileHistoricalMaxMinutes,
default 5 min); makeReq's otherwise-unbounded throttle-retry loop now returns
ctx.Err() when the context is done, and waitForRequestSlot's sleeps are
interruptible. Steady-state loops (current/hourly/5m/daily tip) pass
context.Background() and keep their retry-forever behavior unchanged.
This subsumes the previous stop-channel cancellation: a SIGTERM mid-retry now
aborts even while a single request is stuck retrying a 429, and a persistently
throttling CDN can no longer block blockbook from coming up.
- ReconcileHistoricalRatesAtStartup builds ctx = WithTimeout(budget) + cancel
on shutdown; ReconcileHistoricalRates takes ctx as its first arg
- a budget timeout is recorded as fiat_rates_unable_total{reason=
"budget_exhausted"} (count = unprocessed series) and logged; days fetched
before the abort are persisted so the next startup resumes from the gap
- context-cancelled requests are returned quietly (no error log / error-status
metric) since they are expected during shutdown/budget abort
- tests: cancelled-context aborts before any fetch; budget timeout aborts
mid-backfill without error; all makeReq/waitForRequestSlot call sites updated
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… race The RunDownloader loop split (runHistoricalLoop + runCurrentLoop running concurrently) made logTickersInfo, called on the historical goroutine, read the hourly/5-minute ticker maps and their bounds while the current goroutine replaced them under fr.mux -- a data race flagged by -race. Snapshot all cache fields under fr.mux.RLock() and format the log line outside the lock (matching the GetTickersForTimestamps pattern); also guard the unlocked fr.dailyTickers[fr.dailyTickersTo] read in runHistoricalCycle for consistency. Adds TestLogTickersInfo_ConcurrentWithCacheWriters, which fails under -race without the fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CDN routing only diverted ranges > 1 day, so the recurring daily "tip" range (a 1-day gap == yesterday's just-closed daily candle) stayed on the rate-limited Free tier. On the default Free plan with tokens enabled (e.g. ethereum), the daily UpdateHistoricalTokenTickers loop fetches one market_chart per platform token -- thousands of them -- each a tip-range request paced at 6s on the Free tier, taking many hours per day. Drop the tip special-case in sourceURLForRange so every daily/historical range, including the 1-day tip, uses the CDN when one is configured (no-CDN deployments are unchanged). Only genuinely live, low-volume calls stay on the Free/Pro endpoint: the current price (/simple/price) and the high-granularity hourly/5-minute charts, neither of which routes through sourceURLForRange. The Trezor CDN was verified to carry the latest completed daily candle fresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…void losing token series ReconcileHistoricalRates accumulates every series (base coin x each vsCurrency, then every token) into one shared per-day record, so a day record is only as complete as it will ever be once all targets have run. The previous code persisted the partial map on any early exit (shutdown/budget abort, Cloudflare ban, or DB fill error). After a base-phase or mid-token-phase abort that left a day with base rates but no token rates; because stage-1 detection is whole-day key-only, the day was then seen as "present" and never reconciled again -- silently dropping its token series for good. Make persistence all-or-nothing: write only after a clean, complete pass; on any early exit discard the partial map and report 0 filled, so the next startup re-reconciles the whole gap. Move the fetched-units/tokens metrics to the clean-completion path so discarded fetches are not counted, and generalize observeFetchedToken to take a count. Tradeoff: an aborted run no longer resumes from partial progress and refetches the gap next startup (a future per-day completion checkpoint can restore that). An isolated per-series failure on an otherwise-clean run still persists a partial-day hole -- tracked separately (see #1545). Tests: TokenPhaseAbortDiscardsBaseOnly (verified to fail on the old persist-on-abort behavior) and NoTokensPersistsBaseOnly (clean base-only pass still persists). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-batch progress log only fired at multiples of 100, so the final sub-100 batch was
stored silently and the sequence trailed off (e.g. "updated 800 of 866") with no clear end.
Persist the final batch inside the token block and log an explicit completion line with the full
count and a failure tally ("finished updating 866 of 866 token tickers (0 failed)"); on a
provider ban, log a clear "stopped after X of Y" warning instead of returning silently.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On fast chains every connected block produced three Info lines: resync: local at X is behind resync: synced at X HASH resync: finished in D Being "behind" is the expected, normal trigger for a resync rather than an anomaly, so it is no longer logged. The connectBlocks "synced at" line and the ResyncIndex "finished in" line are merged into one summary that carries the synced height, hash, and duration: resync: synced at X HASH in D This keeps the useful information while leaving the steady-state log at a single line per block, so non-standard events stand out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- skip txs younger than one check period in reconcileMempoolTxs, so a freshly submitted tx transiently unknown to a load-balanced provider node is not removed as a false positive; covered by a new test - treat unmarshal errors in getTransactionFromProviders like transport errors: record as firstErr and try the next provider instead of aborting the loop on one malformed response - use t.Errorf instead of t.Fatalf in the httptest handler, which runs outside the test goroutine where Fatal is not allowed - move blockbook_tx_removal.md to docs/tx-removal-decision.md and list it in the docs index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The alternative-provider send-tx path keeps its own mempool cache because private/Blink-style relays do not expose a mempool list to reconcile against. Until now a cached tx was only evicted when the provider dropped it (provider_missing), when it was mined, or on timeout. A tx whose nonce was already consumed by a *different* transaction (e.g. a replacement submitted outside Blockbook, or a wallet that picked a nonce below the account's current nonce) would keep being reported as pending by the provider and therefore sit in the cache until the timeout fired - it can never be mined. reconcileMempoolTxs now, for a tx the provider still reports as pending, fetches the confirmed account nonce via eth_getTransactionCount(from, "latest") and evicts the tx when its nonce is strictly below that count: the nonce is already spent on-chain, so the tx is permanently unmineable. Evicted via removeMempoolTx (both Blockbook mempool and cache), consistent with the mined/provider_missing removals, and counted under the new nonce_superseded metric action. Deliberately scoped to consumed nonces only: - nonce == confirmed count is the next mineable tx and is kept; - nonce > confirmed count is a gap that may still be filled, so it is kept and left to the timeout (no gap eviction - removing a legitimately pending tx would be worse than the bug being fixed). Safety: - confirmed nonce is the lowest value across all configured providers, so a lagging or misbehaving provider cannot evict a still-mineable tx; eviction requires unanimous agreement that the nonce is spent. - on any lookup failure the tx is kept; the timeout remains the safety net. - lookups are memoized per sender so each sender is queried at most once per reconcile cycle. - "latest" carries the usual chain-tip reorg caveat (documented); the exposure matches the existing mined-tx removal and is bounded - eviction only drops the cache entry, and a still-valid tx is re-indexed when actually mined. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nonce lookup Quality-only pass after a /simplify review; no behavior change. - getConfirmedNonce: drop the redundant (uint64, bool, error) -> (uint64, error). The bool was fully determined by the error (it could only be false when every provider failed, which already set firstErr), so the caller's awkward three-value comma-ok reassignment is gone. A no-result lookup now always returns a non-nil error. - tests: collapse the near-identical reconcile tests into two table-driven tests (liveness outcomes: dropped/mined/known/provider-error; nonce outcomes: below/equal/above/unparsable/lookup-failure) sharing one assertReconcileOutcome helper, and reuse the testAlternativeKnownTxResponse constant instead of re-spelling the pending-tx JSON literal. Net ~78 fewer test lines, same coverage (verified under -race). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The address API's `nonce` is sourced from eth_getTransactionCount at the "pending" tag, so it includes mempool transactions and jumps ahead of the last mined nonce whenever the account has pending activity. Consumers (e.g. trezor-suite) that need the last confirmed/minted nonce had no way to obtain it and were forced to infer it from the most recent confirmed tx, which is fragile with gaps, replacements, or externally-submitted txs. Add a sibling `confirmedNonce` field sourced from the "latest" tag, reflecting only mined transactions. The existing `nonce` semantics are unchanged; `confirmedNonce` equals `nonce` when there are no pending txs. Implementation notes: - Replace BlockChain.EthereumTypeGetNonce with EthereumTypeGetNonces, returning (pending, confirmed). Both nonces are fetched in a single JSON-RPC batch round-trip, so exposing the confirmed value adds no extra latency over the previous pending-only call. Falls back to two sequential calls when the RPC client does not support batching. - The alternative-provider path batches both tags too; on failure it falls back to the primary RPC (which still yields a correct answer). - Tron exposes only the latest nonce via NonceAt, so it returns the same value for both pending and confirmed. The field is additive and optional (omitempty); openapi.yaml and the generated blockbook-api.ts are updated in lockstep and verified by the OpenAPI parity typecheck. trezor-suite tolerates the extra field (no runtime schema rejection); it must sync its generated types to consume it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both callers build BackendInfo from an empty bchain.ChainInfo when
GetChainInfo fails (db.SyncWorker.updateBackendInfo, api.Worker.GetSystemInfo),
so a single failed query replaced the whole payload with zeros. Observed in
production over the last 7 days on Robinhood Archive (13 zero samples) and
Optimism Archive (2).
Consequences of the zero:
- blockbook_backend_best_height drops to 0, which both "backend stuck" rules
read as "the backend produced no blocks", and whose jump back to the real
height then blinds their delta() window;
- BackendTipLastAdvance is reset on the next successful query, because
0 -> real height counts as an advance, masking a stalled tip;
- the version/subversion labels of blockbook_app_info and the /api/ and
websocket getInfo responses are blanked.
Record the error on top of the last known good payload instead. The guard in
SetBackendInfo is narrow - it needs both a reported error and no height - so a
backend answering at genesis is still stored. blockbookAppInfoMetric skips the
backend-derived writes on the same condition, since it reads its own zeroed
copy rather than internal state.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blockbook_tip_age_seconds derived its age from BackendTipLastAdvance, which was not persisted, and the advance was detected by comparing against BackendInfo - also not persisted. Every process start therefore saw "0 -> real height", called it an advance, and reset the age. Measured during the 17-hour Tron crash loop on 2026-07-30: the gauge read 1.4e-5 s at every sample while the instance was restarting continuously. Persist the reference timestamp together with the height it was observed at, and compare against that height instead of the in-memory BackendInfo. Any change of height now counts as "the backend is answering", so a reorg or a backend replaced at a lower height cannot wedge the age at climbing-forever either. InternalState is stored as plain JSON, so the two added fields are compatible in both directions: an older database simply has them absent and seeds on first use. Also correct the metric help and the db/sync.go comment, which called a climbing tip age "the primary stall signal". It tracks the BACKEND; an indexer stalled behind a healthy backend keeps it near zero, which is precisely what happened on 2026-07-30. That case belongs to blockbook_best_height. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blockbook_synchronized, blockbook_best_height and blockbook_backend_best_height
were written only by blockbookAppInfoMetric, which runs at most every 15 minutes
because it needs three backend RPCs through GetSystemInfo. A stall was therefore
invisible for up to 16 minutes, and an alert on "synchronized == 0 for 15m" took
up to 31 minutes to fire.
Add api.RefreshSyncMetrics, which publishes those gauges plus initial_sync from
internal state and the cached backend info with no backend round trip, and call it
- on every storeInternalStateLoop tick (~60s), the guaranteed freshness floor;
- after every syncIndexLoop iteration, via defer so the early returns are
covered too - the interesting case is the quiet one, where resyncIndex keeps
returning syncNotNeeded and nothing else would move the gauges;
- at the two points is.InitialSync flips.
The inSync value comes from the same systemInfoInSync the /api/ response uses, and
the block cadence both normalize by is now derived by one shared syncBlockPeriod
helper, so the gauge and the API cannot drift apart.
backend_best_height is published only once a height has actually been observed:
the cached value is 0 before the first successful GetChainInfo, and a 0 reads as
"the backend produced no blocks" to the backend-stuck rules.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The only mempool alert available today reads the out-of-tree blockbook_api_* exporter, and the flag behind it - /api/ inSyncMempool - is cleared at the start of every reload. Measured over 14 days it reads "not syncing" for 18.6% of samples on btc-b11 and 7-19% on every other Bitcoin backend, so it is a resync-in-progress flag rather than a health signal and cannot carry an alert. blockbook_mempool_size is no substitute either: it is only written on success, so it freezes at the last good value instead of signalling a failure. Export the unix time of the last successful reload, so staleness is expressible as time() - metric. Written next to mempool_size in mempoolWithMetrics.Resync, which already gates on success and covers both the initial resync and the loop. Shaped as a timestamp rather than a computed age for the same reasons as alternative_fee_provider_last_sync_timestamp_seconds - the age keeps rising while reloads fail, and it survives a restart. Also record in the mempool_size help that healthy magnitudes span orders of magnitude between chains (Bitcoin peaks near 1.6e5, Polygon above 5e6), so a single absolute threshold cannot serve both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
is.AvgBlockPeriod is uint32 seconds, so every sub-second chain truncates to 0. Both live "backend stuck" rules are gated on blockbook_avg_block_period > 0, so that silently excluded three coins from either of them - measured today, Arbitrum Archive (~0.24s), BNB Smart Chain Archive (~0.45s) and Robinhood Archive (~0.10s) all publish 0. Compute a fractional-seconds average alongside the integer one and feed the metric from that. The integer field is untouched because it feeds the sync ETA; it also divides a span of avgBlockPeriodSample+1 intervals by avgBlockPeriodSample and so runs about 1% high, while the float form divides by the interval count it actually covers. The remaining zero case is genuine and now documented in the metric help: fewer than 102 block times loaded, i.e. early in an initial build or reindex and briefly after a restart. A rule gated on this gauge switches itself off in exactly those windows, which is why the help points at average_block_time_seconds - configured, and stable across restarts - as the normalizer to prefer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All five increment sites passed error="failure", so the label carried no
information and no alert could tell an unreachable backend from a failing block
connect - the distinction that decides whether to go and look at the backend or
at Blockbook. The dashboard panel already groups by (error) and legends
{{error}}, so it was drawing a single flat series called "failure".
Attribute each site to what actually failed there:
get_block_hash the backend did not answer GetBlockHash for a height being
connected (parallel and bulk connect loops)
get_block a retryable failure fetching a block from the backend
connect_block fetching or connecting a block failed inside a sync worker
resync the resync iteration returned an error, cause not attributed
Four stable values instead of one, so the cardinality cost is negligible and the
existing panel becomes readable without any dashboard change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blockbook_average_block_time_seconds is 0 for coins Testnet and Testnet4 in production, because neither config sets averageBlockTimeMs. That excludes them from any alert that normalizes by the configured cadence - which is the normalizer to prefer, since the observed blockbook_avg_block_period reads 0 during a reindex and after a restart. Use the protocol target of 600000ms, the same value bitcoin.json carries; the observed spacing is erratic on these chains by design, and this field is documented as the nominal cadence rather than a measurement. Signet is included so it does not arrive unmonitored when it is next deployed - that silent-gap failure mode is what this series of commits is about. Regtest is deliberately left alone: its blocks are mined on demand, so it has no cadence. For Bitcoin-type coins this is metrics-only. systemInfoInSync returns before its freshness check for any non-EVM chain, and TipStaleThreshold belongs to EthereumRPC, so no sync or watchdog behaviour changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GetSystemInfo forces inSync=false whenever GetChainInfo fails, but RefreshSyncMetrics never calls the backend and so missed that rule. Combined with the retained last-known-good height, systemInfoInSync would conclude "at the tip and fresh" against a dead backend and the gauge would publish 1 while /api/ reported 0 - the exact drift the shared helper exists to prevent. The backend height is still published, so the backend-stuck rules see a flat series rather than a drop to zero. While here, put is.InitialSync behind the state mutex. It is written from main and was read unlocked from api and server; the metrics refresh added two more concurrent readers, on the sync loop and on the state-store loop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…imes
Two coupled changes, because the second is only safe with the first.
systemInfoInSync derives its freshness window from the chain's cadence as
12 block times, with no lower bound. That is 3s on Arbitrum (250ms blocks) and
1.2s on Robinhood (100ms) - shorter than ordinary RPC, GC and scheduling jitter.
Floor it at 30s, the same bound eth.TipStaleThreshold already applies to its
analogous window.
averageBlockTimeMs is then measurably wrong on three coins. Measured over 14 days
from blockbook_backend_best_height, ignoring restarts and rollbacks, n=1344 each:
BNB Smart Chain configured 3000ms observed 450ms (p10 434, p90 468)
Avalanche configured 2000ms observed 1004ms
Polygon configured 2000ms observed 1500ms
The 6.7x error on BSC matters for more than the metric: an alert that normalizes
expected blocks by this value needs roughly a 33x slowdown before BSC trips it.
Effect of both changes together, per coin:
freshness window Arbitrum 3s->30s, Robinhood 1.2s->30s, Base/Optimism and
Avalanche/Polygon 24s->30s (all looser), Tron 36s and
Ethereum 144s unchanged, BSC 36s->30s - the only one that
narrows, and at 0.45s blocks 30s is still ~67 blocks of
slack against Ethereum's 12.
watchdog window BSC 90s->30s, Avalanche 60s->30s, Polygon 60s->45s; the
watchdog polls the tip and reconnects sooner on these three,
which is the intended behaviour at their real cadence and
stays inside the existing [30s, 5m] clamp.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight blocks introduced by the preceding commits ran to 6-18 lines. Trimmed to the load-bearing fact plus the reason, dropping the per-consequence enumerations and the incident dates, which live in the commit messages and in the metric help text where they are queryable. Comment-only change; no behaviour touched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
averageBlockTimeMs is not metric-only - it also drives eth.TipStaleThreshold (the tip watchdog stall/reconnect window) and the erc4626 negative-cache TTL, so the value must match the real cadence. BSC 450->750ms (post-Maxwell 0.75s) and Polygon 1500->1750ms, on both the archive and non-archive configs. Also replace the lone TAB indenting averageBlockTimeMs in bitcoin_signet.json with four spaces to match every other file under configs/coins. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- computeAvgBlockPeriod: guard the uint32 span subtraction. Header timestamps are not monotonic and RemoveLastBlockTimes re-runs it on a truncated array, so BlockTimes[last] < BlockTimes[first] wrapped to ~4.29e9 and spiked both the ETA and the blockbook_avg_block_period gauge; leave the last good average in place for such a window instead. - SetBlockTime/SetBlockTimes now return the average in fractional seconds under the write lock, so the hot per-block path in ConnectBlock takes is.mux once instead of re-entering GetAvgBlockPeriodSeconds; drop the now-dead uint32 returns and their stale doc comments. - Add GetSyncMetricsSnapshot so the sync gauges can be published from a single locked read; taking backend info, the initial-sync flag and the sync state through three separate accessors let a concurrent resync tear the backend-vs-indexed height comparison. - Stop persisting InitialSync (json:"-"). A SIGTERM during the initial build otherwise stored initialSync=true, and a later start without --sync never cleared it, latching blockbook_initial_sync at 1 and gating off the alert it exists to feed. A --sync run always sets it explicitly on startup. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
blockbook_mempool_last_sync_timestamp_seconds was a plain gauge, so it existed at 0 from process start and the alert its own help text prescribes (time() - metric) fired continuously until the first successful mempool reload - which only happens after the initial ResyncIndex returns, i.e. hours on a UTXO chain and days on an EVM archive. Make it a gauge_vec labeled by chain so the series is absent until the first successful reload writes it, matching the alternative_fee_provider_last_sync_timestamp_seconds pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Publishing fixes: - blockbookAppInfoMetric no longer computes blockbook_synchronized; it derived it from a live GetChainInfo tip while RefreshSyncMetrics uses the cached one, so the gauge oscillated on the ~15-min app-info period. RefreshSyncMetrics is now the sole writer and app-info just calls it. - Always publish blockbook_app_info: its blockbook_* labels are purely local and must identify the instance even when the backend is down. Backend labels fall back to the retained last-good BackendInfo instead of blanking the series. - Keep the sync gauges live during the initial build via a ~30s ticker; both republishing loops only start after the initial ResyncIndex returns, so a stall partway through was previously invisible on the new gauges. - RefreshSyncMetrics reads one locked GetSyncMetricsSnapshot and no longer forces synchronized=0 on a cached BackendError - that force was sticky for a full resync period after recovery and disagreed with /api (which re-queries live); a real outage is caught by tip_age and the backend-stuck rules. - syncBlockPeriod falls back to the fractional GetAvgBlockPeriodSeconds so a sub-second cadence is not truncated to 0 (which would skip every EVM check). systemInfoInSync: - The tip/freshness rescue now applies to any chain with a configured cadence, not only EVM. The raw IsSynchronized flag is cleared for the whole of every resync iteration, so a UTXO chain taking longer than the start grace to connect a batch read out-of-sync mid-iteration though its index was at tip. Drop the now-unused chainType parameter and rename the Ethereum-specific constants accordingly. Tests give each case a private prometheus registry and a fresh Metrics (the shared memoized one made them order-dependent and registered ~60 collectors on the global registry), and cover the generalized rescue and the no-cadence path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
450ms is the observed BSC mainnet spacing measured by the maintainers; revert the 750ms change. Only the Polygon correction (1500->1750ms) is kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… cadence Review findings on the branch, in one commit because the first is the only behaviour change. systemInfoInSync lost its chain-type gate, which extended the demotion (synced -> out-of-sync when the index has not advanced for 12 block times) from EVM to every chain with a configured averageBlockTimeMs - and this branch adds that setting to the Bitcoin test networks. EVM tips arrive over a newHeads subscription that can die silently and their spacing is near-deterministic, so 12 block times floored at 30s has large headroom. UTXO block arrival is Poisson: at 12 mean spacings a healthy chain crosses the window about 6e-6 of the time per block, and Bitcoin's configured 600s against an observed ~682s makes the real window tighter than nominal, so across the deployed fleet this demotes healthy nodes a few times a year - each one a high-severity blockbookApiInSync page, since that alert probes the /api/ field this computes. Restore the gate on the demotion only; the tip/freshness rescue stays chain-agnostic, which is what the mid-resync-iteration false negative needed. A stalled UTXO indexer is covered by comparing blockbook_best_height against blockbook_backend_best_height, which reads neither flag. Polygon averageBlockTimeMs 1750 -> 1500. 1500ms is what the 14-day measurement in 4da39a8 found and what rate(blockbook_backend_best_height) still reports over both 3h and 24h windows (0.6667 blocks/s); 3d1b180 raised it to a post-Bhilai target and 8eca3c6 already reverted the same commit's BSC half to the measured 450ms. This also restores the tip-watchdog window that commit documented (45s). blockbookAppInfoMetric called the package-level refreshSyncMetrics(), reaching for the internalState/chain/metrics globals while holding the same values as parameters; call api.RefreshSyncMetrics with the parameters instead. Its local `api` worker variable is renamed to `w` because it shadowed the package. Also: a Grafana panel for mempool_last_sync_timestamp_seconds, so the new metric is visible the way its alternative_fee_provider sibling is; a note that newRefreshTestMetrics swaps package globals and so cannot be used under t.Parallel(); and a stray blank line inside the metrics.yaml map. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebase onto master brought in the websocket IP blocklist panel (#1687), which also claimed panel id 350; render_grafana.py --check rejects the duplicate. Move the new panel to the free id 352. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… with blink MEV pool
…che uses 3h Explicit mempoolTxTimeout is still required: with an alternative provider enabled and no explicit value, mempool retention falls back to 10 minutes, below the 3h alternative cache, and the pending-nonce floor would outlive the pending view. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bumps [undici](https://github.com/nodejs/undici) from 6.27.0 to 6.28.0. - [Release notes](https://github.com/nodejs/undici/releases) - [Commits](nodejs/undici@v6.27.0...v6.28.0) --- updated-dependencies: - dependency-name: undici dependency-version: 6.28.0 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com>
…sc archive (#1707) With an alternative send-tx provider enabled, the absence of mempoolTxTimeout silently overrides mempoolTxTimeoutHours with a 10-minute default and the alternative cache runs at 5 minutes, so a relay-submitted transaction disappears from the pending view while the relay may still mine it. Align the alternative cache with the 3h window used on Ethereum and make the 12h mempool retention explicit again. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(eth): add per-chain opt-out for ENS reverse aliases Introduce disable_ens_aliases, a per-chain config flag that turns off ENS reverse aliasing (address->ENS-name labels) while leaving every other address alias intact. It is set on all EthereumType coin configs. Motivation: the reverse labels are an event projection of NameRegistered logs with no notion of expiry or ownership transfer, so a name lingers on its original owner address indefinitely and can appear on several addresses at once (stale/duplicate labels). The live, expiry-correct forward path (ResolveENS/CheckENSExpiration) is unaffected. - Write: processEventsForBlock skips getEnsRecord when UseEnsReverseAliases() is false, so no new ENS labels enter cfAddressAliases. - Read: the API worker keeps the Contract/token-name alias (separate cfContracts path) and skips only the GetAddressAlias ENS lookup, so existing labels stop being served without a reindex. address_aliases stays true, so contract/token-name annotations are preserved. cfAddressAliases is left dormant in the schema (no RocksDB migration). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(eth): warn that ENS reverse aliasing is not production-ready Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(eth): move all ENS code into ens.go Gather the ENS declarations scattered across contract.go, dataparser.go, ethparser.go, and ethrpc.go into a single ens.go: the NameRegistered signature and ENS contract addresses/selectors, getEnsRecord, SetEnsSuffix, UseEnsReverseAliases, and the forward path (ensNameHash, keccak256, parseENSAddressFromResult, ensContracts, ResolveENS, CheckENSExpiration). Pure move, no behavior change; struct fields (EnsSuffix, DisableEnsReverse, DisableEnsAliases) stay with their types. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(tron): honor disable_ens_aliases; unify naming; use EthereumLikeParser Review fixes: - Bug: NewTronRPC recreates the parser (NewTronParser) and never set the ENS reverse opt-out, so Tron kept recording/serving spoofable ENS labels despite disable_ens_aliases. Re-apply cfg.DisableEnsAliases to the Tron parser. - Rename parser field DisableEnsReverse -> DisableEnsAliases so the flag has one spelling across config and parser. - worker: assert eth.EthereumLikeParser (which now declares UseEnsReverseAliases) instead of a one-off anonymous interface. Adds a TronParser regression test for the opt-out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eth): make ENS reverse aliases opt-in via enable_ens_reverse_aliases Inverts the per-chain flag: disable_ens_aliases (default on, opt out) becomes enable_ens_reverse_aliases (default off, opt in). The reverse path is a spoofable event projection with no expiry or ownership tracking, so a new EVM chain should not inherit it by forgetting a config line - which had already happened: hyperevm_archive.json set address_aliases without the opt-out and kept recording NameRegistered-shaped labels. Because no chain opts in, the 14 per-chain config lines are dropped and the "disabled by default" warning on getEnsRecord is now true of the code, not just of the shipped configs. Contract/token-name aliases and the forward ResolveENS path are unaffected. Re-enabling stays a one-line config flip. The api worker's fallback flips to false as well, so a parser that does not implement EthereumLikeParser fails safe rather than serving labels. Naming matches UseEnsReverseAliases so the key cannot be misread as gating the forward path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(eth,tron,api): cover both ENS reverse alias gates and the config wiring The opt-in had no test on either gate, and the Tron test asserted the parser predicate rather than the wiring that was the actual regression - it still passed with the re-apply in NewTronRPC deleted. - TestGetAddressAliases_EnsReverseGate indexes the two EthereumType test blocks (one ENS label, one named ERC20) and pins the read path: closing the gate drops only the ENS label, never the contract name. Asserting the whole map also pins where the continue sits, so hoisting it above the contract lookup fails. - TestProcessEventsForBlock_EnsReverseGate feeds a NameRegistered log through a stubbed eth_getLogs and pins the write path, including that the tx logs themselves are returned either way. - TestNewTronRPC_PropagatesEnsReverseOptIn builds a real config, calls NewTronRPC and reads the flag back through EthereumLikeParser, the way api.NewWorker does. Saves and restores the globals NewEthereumRPC writes so the test stays order-independent. - TestNewEthereumRPC_PropagatesEnsReverseOptIn does the same for the non-Tron chains; TestTronParser_UseEnsReverseAliases becomes the predicate truth table, adding the opt-in-without-address_aliases case. Each gate test was verified by mutation: removing the gate, hoisting it above the contract lookup, forcing recordEns true, and deleting the Tron re-apply each make the matching test fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): opt the EthereumType public server test into ENS reverse aliases httpTestsEthereumType asserts the ENS reverse labels the test blocks record (address7b.eth in two explorer pages and two api responses). Making the labels opt-in turned them off for this server, so all four subtests failed. Setting the flag on the test's parser keeps the alias rendering path covered for chains that do opt in, rather than deleting the expectations. Adds TestNewWorker_ReadsEnsReverseOptInFromParser to cover the parser->worker hop this test depends on and that nothing else exercised: the server test configures only the parser and expects the labels to reach the rendered pages, so a worker that stopped consulting the parser would surface as a server-test failure instead of an api one. It pins the wire format too (address7b.eth), so the api package now fails before the server package does. Verified by mutation: dropping the parser lookup in NewWorker fails the new test. The server package itself cannot run in the authoring sandbox - binding any local port is denied, which is why the original break reached CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(eth): address review on the ENS reverse alias opt-in Three findings from @pragmaxim's review: Declare UseEnsReverseAliases on bchain.BlockChainParser with a false default on BaseParser, matching how UseAddressAliases already works. The api layer no longer type-asserts eth.EthereumLikeParser, so an EthereumType parser that does not implement that interface (a decorating parser, a future EVM-like coin) can no longer have its opt-in silently swallowed while the write path still records labels. Removes the assertion, its fallback comment, and the redundant declaration on EthereumLikeParser. Extract the config->parser application into Configuration.ApplyToParser, called by both NewEthereumRPC and NewTronRPC. NewTronParser discards the parser NewEthereumRPC configured, so every config-derived field had to be re-applied by hand in tronrpc.go - a 7-line list nothing kept in sync, and the exact reason Tron dropped the ENS flag in the first place. New config-derived parser fields now have a single point of addition. The Tron wiring test asserts two more config-derived fields so it fails if the helper call is dropped, not just if the ENS flag is. Document that enabling the flag later does not backfill: labels are only recorded as blocks are synced, so a chain that opts in afterwards serves a permanent gap for the off-window until it is reindexed. Noted on both the config field and UseEnsReverseAliases, since "a one-line config flip" reads as recovering the labels. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…and reorgs (#1703) * fix(db): keep contract registry consistent for same-block lifecycles and reorgs - storeContractInfo resolved a destruction only against committed state, so an ephemeral contract created and destroyed in one block lost its destruction: the creation sat in the uncommitted write batch and its cache entry was deleted. Registry writes of a block now share an in-batch map that destructions merge into. The resolved record is also copied before mutation so the shared LRU value cannot expose an uncommitted DestructedInBlock. - Nothing removed cfContracts rows on disconnect: a reorged-away creation left a stale row that permanently disables GetBalanceHistory for that address. DisconnectBlockRangeEthereumType now deletes rows whose creation was disconnected and resets disconnected destructions back to alive, composing across the disconnected range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(rocksdb-ethereum): notify about the contract destruction branch * fix(db): keep the row of a contract recreated in a disconnected block The connect path cannot represent a destroy-then-recreate pair inside one block: the sparse creation record overwrites the merged row, so the row of a metamorphic redeployment reads CreatedInBlock == height for a contract that was created before the disconnected range. The creation rollback matched on that and deleted the row, dropping a contract that lives on the canonical chain - and since GetBalanceHistory gates purely on row existence, the address started serving a balance history instead of being refused, the mirror image of the stale rows this branch fixes. A destruction frame already rolled back for the key at that height is a sufficient discriminator - only an existing contract can be destroyed - so the row is now kept rather than deleted. Its CreatedInBlock stays stale, which no rollback can repair, but that is the less harmful wrong state. The marker is height-scoped so it cannot suppress a legitimate delete in another block of the range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(db): cover the cross-block contract registry rollback composition The range-scoped rollback map exists so that a destruction reset in a later block composes with the creation removal in an earlier one, but every existing case disconnected a single height, so the pending-and-non-nil arm of rollbackContractRegistryRow had no coverage anywhere - the composition is unreachable inside one block, where a CREATE frame always precedes its SELFDESTRUCT in disconnect order. The behaviour was already correct, so this is regression cover only. The test needs BlockAddressesToKeep >= 2, otherwise cleanupBlockTxs prunes the older cfBlockTxs row and the multi-block disconnect aborts before any rollback runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(eth)!: heal the internal data error queue automatically When a block's internal data fetch fails during sync, the block is indexed without its internal transfers and parked in the blockInternalDataErrors queue. Draining that queue required an operator to notice it and press "Retry fetch" on /admin/internal-data-errors, so a transient backend hiccup left a permanent hole in the index: missing internal transfers on getTransaction, missing internal value flows in balance history, and a contract stuck at createdInBlock 0 once the API back-filled it on demand. healInternalDataLoop now drains the queue by itself - a first pass a minute after the initial sync, while the backend still has the failed traces cached, hourly after that. main joins it before closing RocksDB, since a pass writes to the database from its own goroutine. The retry cap of 25 is replaced by a backoff: a block that has failed n times is attempted every 2^min(n,6) passes, between an hour and about 2.7 days, so nothing is abandoned and no operator action is ever needed. A failure is charged only when it is about the block itself, so a backend outage cannot back off the whole queue. Pass 0 matches every schedule, so a restart retries the queue once. Reconnecting internal data to an indexed block gains the guards that make it safe to run unattended: - it refuses to write when the height no longer holds the fetched hash. Previously it wrote address rows at a disconnected height that no rollback could reach, and dropped the queue entry of whatever block had taken over that height. - it skips transactions that already hold internal data, so the cumulative InternalTxs/NonContractTxs counters cannot be inflated by a repeat store that a later disconnect only decrements once. - it stores the contracts the block created or destroyed, which the failed fetch never produced. storeHealedContractInfos overlays only the lifecycle fields, so it cannot strip the name/symbol/standard of a contract queried before its creating block was healed, and it records a destruction whose creation was never indexed. - charging a retry is conditional on the entry still being queued under the same hash, so a pass reporting from its pre-start snapshot cannot resurrect an entry a rollback removed. A db test asserts the healed index matches one that synced with its internal data present - every column family byte for byte, address rows compared as index sets because the heal appends to rows the failed sync already wrote - and that healing twice changes nothing. BREAKING CHANGE: POST /admin/internal-data-errors no longer triggers a refetch and the page is read-only; api.Worker.RefetchInternalData and IsRefetchingInternalData are removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eth): guard the healed contract lifecycle against address reuse storeHealedContractInfos overlaid CreatedInBlock and DestructedInBlock on the stored record with no height comparison. Sequential sync gets away with that, healing does not: it applies a block's lifecycle events after the blocks around it are already indexed, and the backoff can even invert the order of two queued blocks. On an address reused through SELFDESTRUCT plus CREATE2 that could rewind createdInBlock to a dead incarnation, or record a destruction preceding the creation - a contract reported destroyed before it existed, which no sync can produce. The records are now applied in trace order through two guards that reproduce what a sequential sync leaves behind: a creation older than the stored one is dropped, a creation clears only a destruction no younger than itself, and a destruction is dropped unless it follows the stored creation and is newer than the stored destruction. Applying them in order instead of merging them per contract first is what makes a create and a destruct in the same block independent of which value a map happened to keep. Every drop is logged with both heights, because a row left stale by a reorg (the disconnect does not revert cfContracts) is the one case where a guard can skip a legitimate heal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(eth): meter internal data healing and pace the passes Every pass now refreshes the blockbook_internal_data_error_queue gauge and counts outcomes in blockbook_internal_data_heals_total (healed/failed/reconnect_error), so a queue that never drains - a backend that cannot serve the traces - raises an alert instead of waiting for someone to open the admin page. Heal attempts are spaced a second apart: after a long backend outage the queue can hold tens of thousands of blocks and pass 0 retries all of them on restart, so back-to-back traces would compete with live sync for the backend. The sleep yields to shutdown. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eth): count uncharged fetch errors in the healing metrics A transport failure spends no backoff budget, so it previously landed in no counter - a pass lost to a backend outage was indistinguishable in blockbook_internal_data_heals_total from a pass that attempted nothing. Such attempts now count as result="fetch_error", making the outcome labels total: every attempted block lands in exactly one of healed, failed, fetch_error or reconnect_error. The failure classification - charge on ErrBlockNotFound and on a repeated trace error, never on a transport error - lived only in the untested loop body; TestWorker_HealInternalData pins it with a stubbed chain against a real database, including the metric arithmetic and the backoff skip. Also documents that the backoff is aligned to the pass counter, so the wait after a failure is up to, not exactly, 2^retryCount passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(db): reset the package-level contract cache between test databases cachedContracts outlives a test's database and is keyed only by address descriptor plus generations that every fresh database starts at, so a row cached by one test leaked into the next one using the same test address - TestStoreContractInfo_MergesSameBatchCreateAndDestroy's EthAddr20 row broke TestRocksDB_storeHealedContractInfos's expectation of an unknown contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Litecoin Core 0.21.5.6 is an urgent maintenance release hardening MWEB transaction, block and P2P-service validation. It also adds a soft-forking rule at mainnet height 3,154,440 rejecting MWEB kernels that signal a pegout while carrying an empty pegout list, so nodes should be upgraded before activation. No reindex or Blockbook resync is required: there is no on-disk format change between 0.21.4 and 0.21.5.6, and Blockbook already bypasses the raw-block wire path for Litecoin (getblock verbosity=1 + per-txid getrawtransaction), so the MWEB serialization work does not affect parsing. ZMQ notifications are unchanged. Artifacts verified: x86_64 tarball sha256 matches the release notes and the detached signature checks out against the bundled litecoin-releases.asc key (D35621D53A1CC6A3456758D03620E9D387E55666), so verification_type "gpg" stays as is. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore(tron): update java-tron 4.7.7->4.8.2.1 * chore(tron): saveInternalTx = true
…n) (#1700) * feat(eth): add per-chain Multicall3 address override The Multicall3 contract address was a single hardcoded canonical const, so chains that deploy Multicall3 at a non-canonical address could not use the aggregate3 path. Add a code-set override field EthereumRPC.Multicall3AddressOverride (empty = canonical); the resolver multicall3ContractAddress() reads it for both the eth_getCode deployment probe and the aggregate3 call, so the cached deployment verdict refers to the address actually called. The override is set in code at construction (see the Tron commit), not via config. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eth): fetch ERC-20 balances via Multicall3 aggregate3 EthereumTypeGetErc20ContractBalancesAtBlock issued one billable eth_call (balanceOf) per token via a JSON-RPC batch; on metered providers that is ~20xN credits and each call runs at an independent block. Route each chunk through Multicall3 aggregate3 instead (one eth_call, all balances pinned to the same block), with AllowFailure=true so a reverting balanceOf yields nil rather than failing the batch. Fall back per chunk to the existing JSON-RPC batch path on any error (not deployed, transient probe/RPC failure, decode error, or result-count mismatch), preserving the same-length/order and nil-on-failure contract callers depend on. errMulticall3NotDeployed is the expected steady state on non-multicall chains and is excluded from the erc20_multicall fallback metric to avoid alert noise. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(tron): fetch ERC-20 balances via Tron's Multicall3 Tron mainnet deploys Multicall3 at a non-canonical address (0x32a4f47a74a6810bd0bf861cabab99656a75de9e, base58 TEazPvZwDjDtFeJupyo7QunvnrnUjPH8ED), not the canonical one. Apply it in NewTronRPC via EthereumRPC.Multicall3AddressOverride as a Tron protocol constant, alongside the other Tron protocol facts (chain id, token standards), rather than in tron.json. This enables the aggregate3 ERC-20 balance path on Tron; it must be code-driven rather than chain-id-detected because Tron's runtime net_version is not the real chain id (728126428). Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(eth): label Multicall3 aggregate3 calls as mode=multicall aggregate3 calls flowed through EthereumTypeRpcCallAtBlock and were counted as eth_call_requests{mode=single}, indistinguishable from direct reads. Extract the shared eth_call primitive as ethCallAtBlock(mode) and label aggregate3 batches mode=multicall so the Multicall3 fast path's call volume (and thus the credit savings) is measurable. Also reclassifies the erc4626 enrichment's aggregate3 calls from single to multicall. Label value only — no metrics.yaml change (eth_call_requests mode is dynamic); fallback rate stays on ChainDataFallbacks{component=erc20_multicall}. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: shorten comments to brief clean-code notes Trim the multi-line explanatory comments added for the Multicall3 balance work down to brief why-focused notes; no code changes. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eth): re-resolve gas-starved aggregate3 balances via independent calls aggregate3 runs every balanceOf under one shared eth_call gas budget, so an expensive or late sub-call can hit out-of-gas and return Success=false while the batch/single path (an independent eth_call with the full node gas cap) returns the real balance. Success=false was mapped to nil and taken as final, silently dropping a real balance on multicall-enabled chains (notably Tron / low-gas-cap providers). erc20BalancesMulticall3 now reports the Success=false indices, and EthereumTypeGetErc20ContractBalancesAtBlock re-resolves just those via the JSON-RPC batch; a genuine revert still stays nil. Success=true-but-unparseable results (the common dead/airdropped-token case) are left nil without a retry, so the credit win is preserved. Reported in review. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eth): refine Multicall3 balance path per review - Bound the aggregate3 chunk at multicall3MaxCallsPerAggregate (min with erc20BatchSize) so it stays within the node eth_call gas cap: erc20_batch_size sizes a JSON-RPC batch by request count, not gas, and a raised value must not make aggregate3 fail-and-fall-back every request. - Gate the multicall attempt on the cached deployment probe, so non-Multicall3 chains no longer build+hex-encode a call slice on every balances request just to discard it when the probe reports not-deployed. - Count aggregate3 as len(calls) reads under mode="multicall" and feed the eth_call batch-size histogram (like the JSON-RPC batch path), so per-token read volume and batch-size percentiles stay continuous when a chain moves balances onto aggregate3 (the mode="batch" drop is inherent). - Fix stale doc on EthereumTypeGetErc20ContractBalances (still claimed RPC batch) and reword errMulticall3NotDeployed, which hardcoded "canonical address" (misleading for override chains like Tron). Reported in review. Refs: #1683 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(eth): address Multicall3 balance review comments - probe latch: after multicall3MaxProbeFailures consecutive transient eth_getCode errors, cache not-deployed so an eth_getCode-restricting provider stops paying a probe per request; surface probe_error fallback - re-resolve only empty-returndata failures: a Success=false element with revert returndata (Error/Panic) is a genuine revert, never gas-starved, so it stays nil instead of being re-resolved every request - per-element multicall metrics: observeEthCallError("multicall","elem"| "invalid") mirroring the batch path, keeping error continuity on a chain that moves onto aggregate3 - elem_fallback now counts elements re-resolved (Add(len(failed))) not requests, so the re-resolve volume is measurable - document the probe-at-latest vs aggregate3-at-block trap and fix the stale "canonical address" probe doc Tests: genuine-revert-not-re-resolved, empty-revert-without-batcher stays nil, probe latches after repeated failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(eth): fall back per aggregate3 chunk, not for the whole list erc20BalancesMulticall3 aborted on the first failing chunk, so the caller re-fetched every contract through the JSON-RPC batch. For a 300-token address whose third chunk failed that cost 200 metered sub-calls plus 300 batch calls against a 300-call baseline - worse than never using multicall at all. Return the resolved balances plus two index sets instead of an error: the Success=false elements to re-resolve, and the elements whose own chunk failed. The caller settles both in one batch, so a chunk failure only ever costs the contracts in that chunk. Holes that no batcher can fill still discard the multicall result, so an unknown balance is never reported as nil to callers that treat a present entry as authoritative. rpc_fallback_calls{component="erc20_multicall",reason="error"} now counts failed chunks rather than requests, and the warning names the index range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eth): stop aggregate3 at the first failing chunk, fix fallback metrics Review follow-up to the previous commit. A failing chunk only continued the loop, so a chain whose eth_call gas cap cannot fit aggregate3 paid one doomed call per chunk and then re-fetched the whole list anyway: 10 failing calls plus a 1000-contract batch for a 1000-token address, where before there was one. Chunk failures are systemic far more often than not, so stop at the first one and mark the remainder unresolved. Partial results from earlier chunks are still kept, which is what the previous commit set out to fix. elem_fallback counted len(reresolve) while the batch it describes covers reresolve+unresolved, so a chunk-failure fallback recorded zero - the count hit observeChainDataFallback's count<=0 guard - and the extra round trip was invisible on cost dashboards. Count the whole subset. Stopping at the first failure also restores one "error" fallback event and one log line per request, rather than one per failing chunk, and the log no longer claims only that chunk's contracts fall back when the whole list does. Tests, each verified to fail without its fix: systemic failure stops after one aggregate3; a reresolve hole and a failed chunk in one request map every result back to its own contract; a chunk failure with no batcher errors instead of returning unknown balances as nil entries; and the two fallback counters record the whole subset and one event per request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(eth): isolate the prometheus registry in multicall metrics tests common.GetMetrics registers with the global default registerer, so a repeated in-binary run (go test -count=2) hit duplicate-registration errors. Swap in a fresh registry per test, restored via t.Cleanup, mirroring common/metrics_test.go. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(eth): use a neutral Multicall3 override address in eth tests The eth test const duplicated Tron's real Multicall3 address under the same name, implying it locks the Tron wiring — it cannot; only the tron package test (against the tronrpc.go const) does. Rename it and use an obviously arbitrary address so the override mechanism test reads as what it is. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth): reject out-of-range words when decoding aggregate3 results decodeAggregate3Result narrowed node-supplied 32-byte offset, length and count words with int(...) after only an IsUint64 check; a word >= 2^63 wraps negative, slips past the later bounds checks and panics on slicing (or a negative make). Bound every word by len(raw) before narrowing — nothing larger can address data inside the response — so a malformed response errors into the batch fallback instead of crashing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth): verify the aggregate3 result count inside the API call The len(results) != len(calls) sanity check lived only at the erc20 balance call site, leaving the erc4626 enrichment call sites exposed to a silent result misalignment. Enforce it inside EthereumTypeMulticallAggregate3 so every caller gets the guarantee, and drop the call-site copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth): suspend Multicall3 probing instead of latching not-deployed Five consecutive transient eth_getCode failures — e.g. a node briefly down at startup — permanently stored multicall3NotDeployed, disabling Multicall3 (including the unrelated ERC-4626 enrichment) for the process lifetime and collapsing the transient/permanent distinction into a false "not deployed on this chain". Suspend probing for a 10-minute window instead: an eth_getCode-restricting provider still stops paying a probe per request, and probing self-heals once the window elapses. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eth): count physical aggregate3 eth_call requests The mode-labeled eth_call counter reports one request per sub-call (deliberately, to keep per-token read volume continuous with the batch path), so the many-sub-calls-per-one-metered-eth_call reduction that motivates aggregate3 was invisible, and no metric counted actual aggregate3 requests. Add a blockbook_eth_call_multicall_requests counter incremented once per physical aggregate3 eth_call; it also pairs with eth_call_errors{mode="multicall",type="rpc"} as a matching denominator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(eth): decouple the aggregate3 chunk size from erc20_batch_size min(erc20BatchSize, multicall3MaxCallsPerAggregate) coupled a gas budget to a knob that exists for provider JSON-RPC batch-count limits; with the default of 100 the min could never fire, and a chain configured with a smaller batch size would pointlessly shrink a single-request aggregate3. Chunk at the gas-bound const directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth): exclude malformed contract descriptors from aggregate3 encodeAggregate3 fails whole-chunk on a single non-address descriptor, and the chunk loop treats any error as systemic: the chunk and every later one were marked unresolved, disabling multicall for that address on every request. Filter non-20-byte descriptors up front instead — they stay nil, exactly what the batch path produces for a bogus `to` — and chunk over the valid indexes only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth): keep unknown aggregate3 balances out of nil results Two paths could surface an unknown balance as an authoritative nil entry, which callers treat as "no balance": - without a batcher, a possibly gas-starved (empty-returndata) Success=false element was silently left nil. Single eth_calls need no batcher, so settle those holes with individual calls, mirroring the batch path's single-call fallback semantics. - if the merge fallback batch errored, mcBalances was returned anyway with chunk-failure holes as nil — exactly what the sibling no-batcher branch refuses. Propagate the error instead (unreachable today, kept defensive). Also name the real failure when a deployed Multicall3 leaves holes no batcher can fill, instead of the misleading "BatchCallContext not supported". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * perf(eth): circuit-break the erc20 multicall path on repeated failures A deployed Multicall3 whose aggregate3 eth_call always fails (eth_call gas cap, node quirks) cost every address request a doomed eth_call plus a warning log, forever. After 5 consecutive requests in which aggregate3 resolved nothing, suspend the erc20 multicall balance path for 10 minutes (counted under the "suspended" fallback reason) and let the JSON-RPC batch serve balances; any resolving request closes the breaker. Scoped to the balance path only — the deployment probe and other Multicall3 users such as the ERC-4626 enrichment are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: shorten comments to brief clean-code notes Trim the multi-line explanatory comments added for the aggregate3 re-resolve, breaker and probe-suspension work down to brief why-focused notes; no code changes. Refs: #1683 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(eth): settle aggregate3 holes as one set, drop unreachable paths The erc20 balance path grew three structures that guard states it cannot reach: - the reresolve/unresolved split, canFillHoles and the error return on a failed merge batch all rest on erc20BalancesBatchAtBlock returning an error, which it never does: every path returns (balances, nil), absorbing a batch-level RPC failure into per-element single calls. So no unknown hole could ever have leaked as nil through that branch. One hole set, settled by the batch, with plain error propagation. - every !hasBatcher branch inside the multicall path was dead. Each EVMRPCClient either embeds *rpc.Client or forwards BatchCallContext, so only test mocks lack it. Gate the whole function on the batcher up front, as before the multicall path existed, and drop the per-contract single-call loop and its bespoke error. - the erc20 breaker needed a deployed Multicall3 whose aggregate3 fails on five consecutive requests, to save one eth_call per request while results stayed correct throughout. Its anyResolved signal also misread a fully failed chunk as resolved whenever a malformed descriptor was present. Removed; the probe breaker is untouched. elem_fallback now counts element-level holes only, emitted beside the "error" reason it belongs with, so a systemic chunk failure is no longer counted twice — once per request as "error" and once per contract. Behaviour is unchanged on every reachable path. Gas starvation, the reason holes exist at all, stays guarded: eth_call carries no gas, so a node's RPCGasCap (geth default 50M) covers a 100-call chunk many times over, leaving Tron's constant-call energy cap as the open question. Refs: #1683 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(eth): merge the two prometheus counter readers into one counterValue hardcoded the label name to "action" while counterVecValue took it as a parameter, so the package carried two readers doing the same job. Keep the general one and move it to an untagged file, since only the unittest-tagged tests could reach it before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(eth): tighten the erc20 Multicall3 balance path Cleanup pass over the aggregate3 balance code, no behaviour change. Instrumentation: ethCallAtBlock took a mode="" sentinel so the multicall caller could skip it and count len(calls) itself. It now takes subCalls, so the primitive owns the mode label and the rpc-error label and no caller can opt out; anything added there cannot silently miss the largest calls. State: the four loose multicall3Probe* fields on the shared EthereumRPC struct become one multicall3Gate value in multicall.go, and the aggregate3 chunk bound moves next to the primitive whose gas budget it describes. Allocations, per 100-call chunk: encode 609 -> 205, decode 605 -> 202. One wordAsIndex helper replaces four copy-pasted bounds checks and drops the big.Int per ABI word; the per-element error counters are tallied and emitted once instead of rebuilding a label map per held token. Tests: drop the byte-for-byte copy of common/metrics_test.go's registry swap and build the collectors under test by hand; two of the four aggregate3 mocks were subsumed by the others. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eth): bound aggregate3 decoding to the response size decodeAggregate3Result checked every offset and length against len(raw) but never their total, and nothing required head offsets to be distinct. A response could therefore declare len(raw)/32 elements whose heads all alias one large tuple, and each element passed its own bounds check while the set of them decoded far past the response. The result count was only compared with the request after the full decode had already been materialized. Measured on the previous commit: a 0.16 MB response decodes into 132.91 MB of hex strings with no error. Output grows with the square of the input, and the eth_call response itself is uncapped (geth's HTTP client hands back resp.Body unwrapped), so one reply from a hostile or buggy node is enough to OOM the process. This PR put the per-address balance path through it. The decoder now takes the number of sub-calls sent and rejects any other count before touching the tails, and it tracks cumulative returndata against len(raw) so aliased heads cannot amplify. A canonical response lays its tuples out disjointly, so neither guard can reject one. The same 20k-head aliasing payload that would decode 7.63 GB is now refused after 4.46 MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf(eth): re-resolve aggregate3 holes only on a starved tail aggregate3 reports a sub-call that reverts with no returndata identically to one starved of gas, and every such element was re-read through the JSON-RPC batch. Contracts without balanceOf fall into that class and are common in holders' lists, so each one cost a metered call on every request -- the cost this path exists to avoid. Gas starvation is necessarily a trailing run, since aggregate3 forwards all remaining gas to each sub-call: once one exhausts it, the calls after it fail too. A chunk whose last element did not fail empty was never starved, so its empty failures are bare reverts and stay nil, matching the batch path. When the tail is starved, earlier empty failures are re-read as before. Measured against live Tron and Ethereum mainnet with a real contract lacking balanceOf: a 26-contract request with 25% such tokens drops from 6 metered calls to 1 (the batch path bills 26). A dead token landing last in a chunk is still re-read, which the signal cannot distinguish. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(contract.go): update docs startvation heuristic comment * chore(eth): remove unused imports * feat(multicall): make multicall batch size configurable * chore(tron-backend): increase timeout to allow 100 multicall batch sizes * chore(tron): normalize addresses in multicall --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: pragmaxim <pragmaxim@gmail.com>
* fix(evm): pool JSON-RPC connections to prevent port exhaustion EVM dials passed no HTTP client to rpc.DialOptions, so go-ethereum used http.DefaultTransport with its default of 2 idle connections per host. With -workers=16 and ~4 calls per block, a sub-second chain closed a socket per call: on blockbook-dev2 that left 20326 TIME-WAIT entries against 11 ESTAB toward one backend, exhausted the host's ephemeral ports and failed dials with EADDRNOTAVAIL. Sync then stalled silently for 10 days, because the resync retry, the tip watchdog poll and reconnectRPC all need a new socket to recover, while the API kept serving stale reads from RocksDB. Add eth.RPCDialOptions as the single dial-option source for EVM coins: a shared pooled client for http(s) endpoints and the unlimited message size for ws(s) ones. Mirrors the tuning btc, nuls, dcr and the Tron REST client already carry for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(eth): brief description --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ient errors (#1716) * fix(eth): log alternative provider send rejections with the error When every alternative provider rejected eth_sendRawTransaction, the rejection reason was only returned to the client and never logged - the per-URL Info line printed an empty txid and dropped the error. The fetch-back then ran with the empty txid, querying the zero hash and logging a confusing 'did not find txid' error with no txid. Log the provider error at Error level and skip the fetch-back when no provider accepted the send. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sendtx): make successful and failed transaction sends observable Sends were only visible as blockbook_rpc_latency{method="SendRawTransaction"}: it counted failures but never said why or where. The primary RPC send path logged nothing at all, the alternative -> primary fallback was invisible, and the post-send fetch-back that makes a private tx show as pending was log-only. Add a backend-agnostic rejection classifier and four metrics: - sendtx_requests_total{result,reason} - every client submission, all coins and all APIs, with the rejection classified so caller-side classes (underpriced, nonce_too_low, already_known) are separable from backend ones (rate_limited, timeout, unavailable), which is what decides whether anyone is paged - eth_sendtx_path_total{path,result} - alternative, primary or primary_fallback, so transactions leaking to the public mempool are visible - eth_alternative_sendtx_total{provider,result,reason} and eth_alternative_sendtx_duration_seconds{provider,result} - a broadcast succeeds if any provider accepts, so a degrading relay shows up only here Fetch-back outcomes join the reconciliation counter as fetchback_cached, fetchback_missing and fetchback_error. Every send now logs the sender and nonce decoded from the raw transaction, plus txid and duration, on both paths - that is the record a later missing-tx or nonce-gap report has to be reconciled against. A per-provider rejection drops to Warning (a later provider may still accept) with a single Error when all providers reject. Provider URLs are reduced to their host in logs and metric labels, including the startup line that printed *_ALTERNATIVE_SENDTX_URLS verbatim - those URLs routinely carry an API key. Also stop feeding an empty txid to the mempool when a primary-path send fails with DisableMempoolSync set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(eth): scrub provider urls from errors before logging or returning them Reducing the URL to its host in log arguments was not enough: the URL also travels inside the error message. It gets there from our own annotations (errors.New(url + ...)) and from the http client, whose *url.Error reads Post "<url>": dial tcp ... on any dial failure or timeout - exactly the failures the new send log lines exist to surface. With a key in the URL the reproduction was: eth_sendRawTransaction rejected by all 1 alternative providers, tx from 0x7156... nonce 1: Post "http://127.0.0.1:1/v3/SECRET_API_KEY_ABCDEF": ... Scrub at the provider call boundary, so the redaction also covers the error handed back to the API client, not just the log line: every URL in the message is replaced by its host. The batched nonce read dials the url itself and is redacted too. The log sites scrub again rather than trust every current and future error path, and the primary RPC send lines scrub as well - that url can carry an API key just the same. A plain JSON-RPC rejection carries no url and passes through untouched, so error classification keeps seeing the backend's own wording. Corrects the docs claim from the previous commit, which promised more than the host-only labels delivered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(grafana): add panels for the send-transaction metrics AGENTS.md pairs "Add a metric" with "Add a panel", and render_grafana.py --check only validates panels -> metrics, so the reverse omission is invisible to CI: the four metrics added for send observability were the only ones in configs/metrics.yaml without a panel, apart from initial_sync. Add four timeseries panels to the RPC row - overall send rate with the failure classes broken out, the EVM send path, per-provider outcomes and per-provider latency quantiles - each describing how to read it against the others, since the whole point of the split is that a failing relay and a failing send are different events. Also refresh the alternative mempool reconciliation panel prose, which still enumerated the pre-fetchback action list while the query aggregates by (action), so the three new series would have plotted undescribed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * revert(eth): keep provider urls in error messages, correct the docs instead Reverts b8b6b72. Logging a provider URL that carries an API key is acceptable in this deployment, so the scrubbing at the call boundary and at the log sites goes away along with its tests. What stays wrong in b8b6b72 is only the documentation: host-only metric labels do not stop a URL from reaching the logs, because the http client renders Post "<url>": ... into the error message on any dial failure or timeout, and that message is logged and returned verbatim. Say that instead of claiming a key is never exported. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): redact urls from error messages returned to clients A provider URL travels inside the error message, not just in the log arguments: it gets there from our own annotations (errors.New(url + ...)) and from the http client, whose *url.Error reads Post "<url>": dial tcp ... on any dial failure or timeout. Every API surface then hands that message to the caller verbatim, so on an only-alternative deployment a relay URL with a key in it was served to anyone who could reach the API: Post "https://relay.example.com/v3/SECRET_API_KEY": context deadline exceeded The exposure was wider than the sendtx path that surfaced it. jsonHandler renders the text of an internal APIError too, so Public was never the gate it looks like; the ws protocol answers every error with its message, which puts the alternative provider's nonce read (a getAccountInfo away) on the list; and the same shape carries the Infura, 1inch and CoinGecko keys out through estimateFee and the fiat rates. explorerSendTx builds its own APIError and returns no error at all, so it bypasses the handler entirely. Redact where a message is handed to a client rather than where the error is made: one place per surface covers every backend and every future error path, and logging keeps the URL an operator needs. Every URL becomes scheme://host, which drops the userinfo, path and query a key is carried in while leaving the backend's own wording intact - wallets classify sends on that text. The regression test drives the real path (unreachable provider, real dial failure, key in the URL) and asserts both halves: the returned error still carries the key, the redacted form does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(metrics): align the send-tx labels with the registry's conventions Three label names on the new send metrics conflicted with established ones: - provider already means a symbolic fee-provider implementation (infura, 1inch) on two metrics released in v0.6.0; the send providers are configured as a list of URLs and a host is the only stable identity they have. Two value domains must not share a label name - that name is the join surface - so the send metrics use provider_host. - path already means an HTTP endpoint path (address, xpub) on five balance-history metrics, while here it meant the broadcast route: send_path. - status is the registry's word for an outcome (10 metrics) and result was used by exactly one; the new metrics used result. They now use status, with success/failure values to match the domain the other status labels use. Free to do now because all four metrics are unreleased, added in this branch. Deliberately not touched: eth_alternative_nonce_requests_total{result}, the one pre-existing outlier - it shipped in v0.6.0 and renaming its label would break any alert or saved query built on it, for consistency alone. No dashboard query joined on these labels, so this is a rename rather than a fix, and the panels and their prose move with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(eth): decode a raw transaction once per send describeRawTx recovered the sender with types.Sender purely to build a log line, and registerSuccessfulSend then decoded the same hex again a few lines later. Recovering a sender is an elliptic curve operation - measured here at ~800us against ~1us for the RLP unmarshal alone, and still ~50-100us on a Linux build with libsecp256k1 - so the accepted path paid for two of them on every broadcast, on the request a wallet is blocked on. decodeRawTx now returns a decodedTx value carrying the transaction, the sender and the decode error, with String() rendering the log form. The send decodes once at the top and hands the value to registerSuccessfulSend. The rare path where every provider rejects still decodes in both functions; it is already dominated by N provider timeouts, and threading the value through the method signature would only complicate the callers. Three simplifications fall out of touching the same code: - providerLabel ran twice per provider per send (once for the log line, once inside observeSendTx). The host is reduced once per iteration and passed down. Not precomputed at construction: a hosts field parallel to urls is an invariant the tests that build the struct literally already break, for ~1us. - acceptedURL was tested three times in a row, which forced gen to be declared ahead of its use. One early return replaces all three. - EthereumRPC.SendRawTransaction assigned "primary" to path, reassigned it, and then recovered the same state by comparing the string back against a literal. retErr is only ever set by the alternative providers above and both other outcomes return early, so path is derived from it once. On the test side, newSendTxMetrics had re-declared EthAlternativeMempoolEvents under the same collector name as newReconcileTestMetrics; it now extends that constructor, leaving the counter one owner. gatherMetric also generalizes the three readers already in the package, so gaugeValue, counterValue and residenceSampleCount become one-liners over it instead of three more copies of the register-gather-match body. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(server): redact the debug messages built in the panic handlers The redaction added earlier in this branch covered the error branches of both handlers but not their panic recovery, which builds a client-visible message the same way: jsonError{fmt.Sprint("Internal server error: recovered from panic ", e), ...} A panic value is routinely an error, and on the send path that error is the one carrying Post "<url>": ... with the provider key in it - so in debug mode the key went straight out to the caller through the one exit the redaction had not reached. Rather than add a fourth hand-placed call, every debug-mode message now goes through debugErrorText, which formats and redacts in one step. That covers the two panic branches and any branch added later by construction, and replaces the three copies of the RedactURLs(fmt.Sprintf(...)) pair. jsonHandler also redacted an *api.APIError by hand while htmlTemplateHandler, twenty lines below, used api.RedactAPIError on the same type. Both use the helper now, so the file states the rule once. The bespoke nonAPIError test type goes with it: the handler branch it exercises is a type assertion to *api.APIError, which errors.New already fails. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(bchain): declare the reason labels above the table that uses them Every class value in sendTxErrorClasses was a forward reference resolved fifty lines further down, so the table could not be read top to bottom. Moving the const block ahead of it is a pure reordering. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(env): correct the send-tx metric labels and narrow the redaction claim The label rename in 3466a2e (result -> status, path -> send_path, provider -> provider_host) did not reach this paragraph, leaving all four documented label sets wrong. A selector copied out of the docs matched nothing. The same paragraph claimed every error message handed to a client has its URLs reduced. That holds for the error responses - REST, WebSocket and the explorer - but not for the backend error text Blockbook ships inside successful ones: BackendInfo.BackendError on /api/ and Erc4626Token.Error both carry a live RPC error into an HTTP 200 body, where no handler error branch runs. Redacting those belongs where the URL enters the error rather than at another exit, so the claim is narrowed to what the code actually guarantees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(bchain): classify broadcast failures that carry only an HTTP status A revoked provider key or a failing provider surfaces as geth's rpc.HTTPError, whose text is "<status>: <body>" with no backend wording the classifier table could match, so every one of them landed in "other" - the single class both the metric help and the dashboard tell the operator to read as a gap in the table rather than as an incident. A total broadcast outage was therefore reported in the one bucket nobody alerts on. Match the standard reason phrases, and fall back to the status code itself for the deployments where a proxy replaced them. 401/402/403/407 get their own unauthorized class because a rejected or exhausted key calls for a rotation rather than a provider investigation; 5xx joins unavailable next to the bad gateway and service unavailable phrases already there. 400 and 404 stay in "other" on purpose, their meaning depends on the request. The status is read only after the table, so a backend that wraps its own wording in a 400 still classifies on the wording. The same pass anchors the "rlp" fragment on the colon geth's rlp errors always carry. At three characters it also matched the provider URL that travels inside a failed call's message, so any deployment whose relay path contains those letters reported every timeout of that provider as a caller-side rejection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.