fix(reputation,qos): stop health-check probes benching endpoints, plus cardinality/solana/archival follow-ups

[oten91]

Follow-up to #527. The branch grew past its original scope: it now carries the F5/F6
cardinality follow-up plus four other lines of work that were deployed and validated
together over the last two weeks.

Every commit here is live on canary and mainnet as sha-064bc62-rc and has been
validated in production. That is the main argument for reviewing it as one unit —
these commits were never exercised separately, so splitting them would put untested
combinations into main.

---

1. Metric cardinality (the branch's original purpose)

e3f5c7e2 — bound histogram labels, remove supplier from aggregate metrics.

Two rounds of a production cardinality incident converged on one rule: only a label's
value set bounds it. A sanitizer bounds a value's shape, never the set. A cardinality
guard bounds the live registry, never the number of distinct series Prometheus retains —
path_supplier_signal_total sat at 26% of its cap while being one of the two largest
series sources in the job (6,523 tuples live in 10 minutes against 60,674 distinct over one
pod's 7.7h life).

A label on a histogram costs roughly 12× what it costs on the counter beside it.
path_relay_latency_seconds_bucket was 31.6% of all gateway series because it carried
status_code × reputation_signal, a 20× pair no dashboard ever queried from the
histogram. Outcome taxonomy now lives on the counter; the histogram keeps topology labels.

Per-supplier questions are served by GET /ready/<service>?detailed=true — a point lookup
instead of ~74K retained timeseries. Three metrics keep supplier deliberately, where the
address is the actionable payload rather than a way of naming an operator.
Test_SupplierLabelIsGone enforces the rest.

2. Solana sync allowance and health checks

2304bc9b 2d49d8e5 ad729c3f 5c1c8d60 0e856cc4 068ff990

Solana's ValidateEndpoint had no sync allowance at all — sync_allowance: 750 was
configured but never implemented. During the 2026-08-18 surge (200 → 2500 rps, ~2.5
blocks/s) only the freshest endpoint stayed valid, producing near-total lock-in on one
operator while others sat at score 100 with no traffic. Health checks cannot correct this;
they run at the same per-endpoint rate on every operator.

Also here: kava CometBFT checks routed to comet_bft, slower xrplevm websocket probes, a
missing solana health observation no longer scored as a fault, and relays that never
received an HTTP status no longer counted as successes.

3. Go, dependencies, CI

d3e0273c e6e5530e fa146acd c10e5be6 — poktroll v0.1.35, Go directive to 1.26.6 for
the stdlib security fixes, drop the libsecp CGO variant, build release platforms
concurrently.

The Go bump takes govulncheck from 12 reachable findings to 6, all of which are
Fixed in: N/A (three withdrawn lib/pq advisories, openpgp, two cosmos x/crisis init-only).
The net/http CVE was genuinely reachable via router.go ListenAndServe. The Dockerfile
Go patch is deliberately left floating so future patches are picked up; the directive sets
the floor.

4. Heuristic and reputation detectors

dd710d38 4931c6f4 6ef6ca1a 22e9dff1

An endpoint returning zero-length payloads at ~0.2% of its traffic held a reputation score
of 100 all day, and neither existing mechanism could reach it:

• Additive scoring is outvoted by volume. At one violation per 1000 requests the endpoint
  earns +998 and loses −25, so the score returns to its ceiling however long the behaviour
  continues. Raising the per-event penalty to FATAL (−50) does not change the sign.
• The critical-rate detector is tuned for "unambiguously broken" (30%), and more
  fundamentally CriticalRateEWMAAlpha's ~20-request memory cannot represent a sub-1%
  rate at all — the EWMA can only be 0 or ~0.05 there. No threshold change to that
  detector could have worked.

So 22e9dff1 adds a second detector rather than retuning the first, with a much longer
window (alpha 0.001, ≈1000 requests) and a threshold three orders of magnitude lower.

Two classifier bugs fixed alongside it: a "(method=X)" suffix made every exact-match case
in classifyHeuristicErrorAsSignal unreachable, so an empty payload degraded to
unknown_payload_error (MINOR) — only the HasPrefix("error_indicator_") case survived,
which is why it hid. And getProgramAccounts returning an empty array is a valid success,
not a fault.

5. Archival promotion and health-check contamination (today)

54659fb6 — geth PBSS pruned state

Geth's path-based state scheme reports metadata is not found, <block>. Every archival
pattern in PATH used hash-based-scheme wording (missing trie node, state has been pruned), so a PBSS node's honest "I do not retain that state" matched nothing, at four
separate sites.

The deeper defect: IsArchival returned true for any successful eth_getBalance /
eth_call / eth_getCode / eth_getStorageAt / eth_getTransactionCount without
reading the block parameter. Those are also the ordinary way to read current state, and a
pruned node answers them perfectly — so the archival pool was polluted by construction and
marked archival for 8h. targetsHistoricalBlock now gates the promotion.

Known residual: the DataExtractor interface carries no perceived chain tip, so a numeric
block a few blocks back still reads as archival. Closing that needs an interface change
across all four extractors.

d8f4c3c1 — health-check probes were feeding both rate detectors

Both volume-independent rate detectors are wrapped in if !signal.IsHealthCheck, so a
probe cannot bench an endpoint on its own — a strict or flaky check must not cool an
endpoint that serves user reads perfectly. That guard was intact. The stamp was not.

IsHealthCheck was set at only three call sites, all in the health-check executor. One
probe also reaches reputation through the protocol layer twice more — the relay itself via
requestContext, and Apply{HTTP,WebSocket}Observations on that same relay's observations
— and neither stamped it. The field doc on requestContext.isHealthCheck stated outright
that it "does not affect reputation signals or observations", which is why the omission
read as deliberate.

The result is a self-sustaining loop rather than a one-off penalty: a benched endpoint
receives no user traffic, so probes become its only signal, so its rate EWMAs are entirely
probe-derived, so it re-benches itself on the next probe failure.

This predates the new detector. The critical-rate detector has been contaminated since it
shipped; 22e9dff1 only made it visible by tripping at a much lower threshold.

Apply{HTTP,WebSocket}Observations now take isHealthCheck as a required parameter rather
than defaulting it — the receiving protocol layer cannot distinguish synthetic observations
from real ones, so each caller states it at compile time.

Scope is narrow on purpose: only the rate detectors exclude probes. A probe still moves the
additive score — that is how a benched endpoint recovers when it receives no user traffic —
and still increments the counters, so no rate's denominator changes shape.
TestHealthCheckSignals_StillMoveTheAdditiveScore guards that.

064bc628 — each rate cooldown escalates against its own history

Score.InvalidRateCooldownCount is documented as kept separate from RateCooldownCount so
the two detectors escalate independently. The counters were separate; the timestamp they
escalated against was not — both compared against the shared Score.CooldownUntil, which
the strike system also writes.

Note the sign: time.Since() on a cooldown still in force is negative, hence always below
DefaultMaxCooldown. Any bench in force, from any mechanism, made the next trip of either
detector read as consecutive.

Each detector now records the end of the cooldown it set and escalates against that.
CooldownUntil is unchanged and remains the only field selection reads.

b395d183 — the "historical state" pruned-state wordings

Found by probing rather than by reading. Sending a block 27M deep to endpoints PATH had
marked archival returned two wordings that missed every pattern in
archivalErrorIndicators by a single word:

gnosis: "historical state is not available"   -- "state not available" misses on "state IS not"
poly:   "historical state <hash>"             -- "historical data" misses on "historical STATE"

Both fell through to the "some other error" branch, which returns an error rather than
false, so an endpoint that had just failed an archival query was never demoted out of the
archival pool. The bare "historical state" prefix covers both, and is already present in
qos/heuristic/indicators.go — the two catalogues had drifted, so this realigns them.

31617122 — an unverified archival mark no longer outlives a verified one

The two sources of archival status had drifted 16× apart, in the damaging direction:

health-check mark   30m
user-traffic mark    8h

The health-check path pins an exact expected historical value in the rules file, so a node
that ignores the block parameter and answers from current state fails it. The user-traffic
path cannot pin a value — the query is whatever a client sent — so it grants archival status
on any successful archival-method call, and 54659fb6 notwithstanding it still trusts a
successful response, which is what a fabricating node always produces.

Measured in production: four endpoints on one operator were marked archival while returning
current state for every block asked, including one 256× past the chain tip. The archival
health-check rule for the service they served had been deleted for failing every endpoint —
which was the rule working correctly, that service has no archival nodes — leaving only the
unverified 8h path to promote them.

Both paths now share gateway.ArchivalStatusTTL. The old comment on the 8h constant claimed
it "matches health check archival TTL"; it did not, and the false comment is probably why
the drift went unnoticed. The test reads the stored expiry back through
UpdateFromExtractedData rather than comparing constants, so re-hardcoding a duration at
the call site fails it.

Bootstrapping is unaffected: promotion still happens via requests naming a numeric block
within the archival-required threshold, which route freely rather than being filtered to
already-archival endpoints.

---

Production validation

Deployed to canary at 07:18 UTC and mainnet at 08:57 UTC on 2026-08-20. Because the two
environments flipped at different times, the same metric collapsing twice — each time
following the build — rules out pod age and traffic composition.

	before	after
mainnet rate_cooldown_total	0.178/s	0.0006/s (297×)
canary rate_cooldown_total	0.005/s	0.0012/s
pool_collapse_guard{solana} on canary	143,090 / 14h	~35 / 85m
invalid-rate trips	10 services	solana only, both envs

Guardrails flat throughout, judged against 6h ranges rather than point readings: fleet
success 95.7% mean (min 92.5, max 97.3), solana pool size unchanged, no cooldown spike.

Redis confirms the escalation fix directly. Mainnet DB2 went from 0 of 5,901 keys
carrying rate_cooldown_until to 292 of 297 sampled within five minutes of the flip, and
two solana endpoints that tripped post-deploy each recorded invalid_rate_cooldown_count = 1 with their own timestamp — a first offence, unescalated. For contrast, the pre-flip
distribution had 84 keys above zero with 35 at ≥ 6, i.e. benched the full hour every
time, topping out at 98.

Archival counts from /ready/<svc>?detailed=true moved as intended. The old build pinned
five services at exactly 100% — every endpoint of every operator archival, which is not a
plausible ground truth. The fixed build discriminates: on gnosis it keeps qspider.com at
12/12 while dropping two other operators to 0/17 and 4/17, where the old build had all
three at 100%.

Testing

go build, go vet and golangci-lint clean. Unit tests pass; reputation/storage needs
Docker for testcontainers-Redis and fails without it.

Every call site in the two reputation fixes was revert-checked individually — seven reverts,
seven confirmed test failures. Two traps found while writing those tests, both recorded in
comments:

• The first escalation test passed against the revert. A fresh key plus a foreign bench
  in force does not discriminate, because incrementing a zero counter yields 1 — exactly
  what a correct reset yields. The discriminating case needs stale non-zero history and a
  foreign bench.
• runAtRate(4000, 100) trips the detector five times, not once: a trip resets the EWMA and
  the loop continues. Any first-offence assertion has to drive one signal at a time and stop
  at the first trip. The pre-existing test never noticed because it only asserted
  IsInCooldown().

Tests assert on the signal reputation receives, from the production caller, rather than on
the flag the caller set — the flag was already true and proved nothing.

CI note: the xrplevm HTTP E2E check has been failing on this repo independently of this
branch. Please confirm it also fails on main before treating it as a blocker here.

Known open items

1. moonbeam genuinely has no archival nodes. Answered by direct probing:
   moonbeam_archival was deleted from the rules file on 2026-08-07 for failing 100% of
   runs — which was the rule working correctly — and the single endpoint still marked
   archival there is one that ignores the block parameter. Impact is small: moonbeam draws
   1.7–3.5 archival_required rejections/s against poly's 761/s.
2. Endpoints that ignore the block parameter cannot be detected by any success-based
   check, including the targetsHistoricalBlock gate added in 54659fb6, which verifies
   the request named a historical block and then trusts a successful response. Measured on
   one operator across two services: identical balances at block 1, block 5,000,000 and
   block 4,294,967,295. A design for detecting this is written up in
   DESIGN_NEGATIVE_HEALTH_CHECK.md and deliberately parked — the two commits above
   cover most of the exposure without a new check type. The cheapest future detector is
   eth_getBlockByNumber(H) asserting result.number == H, which is self-verifying and
   needs no external reference.
3. Only 22 of 69 services have an archival health-check rule. On the rest, archival
   status comes exclusively from the unverified user-traffic path.
4. poly_archival and xrplevm_archival assert expected_response_contains: "0x0", a
   substring matching a large share of hex values. Weak, and in the external rules file
   rather than this repo.
5. InvalidRateThreshold = 0.005 was sized from one hour of data with a contaminated
   denominator. Now that the denominator is honest it fires ~31/hour on solana, correctly
   confined but not near-silent. Worth re-deriving from the real per-key rate distribution.
6. User relays record a reputation signal twice — once in context.go and once via the
   observation path. Pre-existing double-count, not addressed here.