Cap timer histogram buckets at 13 instead of 69

[mxssl]

Problem

publishPercentileHistogram() makes Micrometer expand each timer into a 69-step exponential le ladder (0.001s → 30s). Measured on a live public-multiregion pod (image 6a2fb45, container age 10d22h), that ladder is almost the entire /metrics response:

Metric family	Bytes	Series	% of body
dshackle_upstream_rpc_conn_seconds_bucket	43.41 MB	300,840	58.3%
dshackle_request_grpc_native_response_seconds_bucket	24.46 MB	195,891	32.8%
dshackle_request_grpc_response_seconds_bucket	2.28 MB	21,252	3.1%

Body is 74,507,051 bytes. Every *_seconds_bucket line together is 94.7% of it. Label cardinality is not the issue — upstream.rpc.conn has only 4,360 label sets. 4,360 × 69 = 300,840 series exactly. The multiplier is the ladder: 10.10 KB of response body per label set.

Our metrics agent has a hard 64 MB cap on the raw response body, applied before parsing, with no partial ingestion. Over the cap it drops the entire scrape. So two oversized families take out every dshackle_* metric from the pod — upstreams_availability, current_head, stuck_head, upstream_blocks, the JVM metrics, all of it. Two of our three public-multiregion clusters have been dark for 5-6 days each. dshackle itself is healthy throughout; it served 9.27M requests in the 30 minutes before this was measured.

Restarting is not a fix. A fresh pod starts at 42-51 MB — already 66-80% of the cap — and crosses 64 MB in 2-3 days, because the two ConcurrentHashMap timer caches (RequestMetrics.kt:27-33, BlockchainRpc.kt:216-219) never evict and the method tag keeps growing. 24.7% of label sets have recorded exactly one observation in ten days, and each still costs 10.10 KB on every scrape.

For context on why this got acute recently: #863 added the method tag to upstream.rpc.conn. That is a genuinely useful label, but it took that family from ~199 label sets (upstream × chain) to 4,360, multiplying a 69-bucket histogram by ~22×, which is roughly +43 MB of body. This PR keeps the label and cuts the buckets instead.

What this changes

A MeterFilter in MonitoringSetup replaces the percentile histogram with 12 explicit boundaries — 1ms, 2.5ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s, 30s — so 13 buckets including +Inf. One filter covers all six publishPercentileHistogram() call sites (BasicHttpFactory.kt:43, BlockchainRpc.kt:204/213/230, WsConnectionFactory.kt:40, GrpcUpstreams.kt:292), and a new call site cannot forget it.

Effect on the same pod: 74.5 MB → ~16 MB, a 5.3× cut on 94.7% of the body.

The filter is deliberately narrow. It returns the incoming config untouched unless the meter is a TIMER and already asked for a percentile histogram, so:

• no timer ever gains buckets (executor.seconds, request.jsonrpc.call and friends stay bucket-free);
• non-timer distributions are skipped, because their recorded values are not nanoseconds and a latency ladder would be meaningless on them.

Values are set in nanoseconds, which is what DistributionStatisticConfig expects for timers — the same conversion Timer.Builder.serviceLevelObjectives(Duration...) does internally via Duration::toNanos. merge(config) is ordered so the builder's values win and everything else is inherited.

Why 13 buckets and not 20

Sizing against today's 74.5 MB gives the wrong answer. The ceiling matters: dshackle has no per-user label anywhere (all 22 label names checked), and upstream is strictly 1:1 with chain, so the worst case is Σ over upstreams of allowlist(chain) — 203 upstreams × 52 upstream-callable methods for a plain EVM chain, plus 788 chain-specific extras. That is 11,344 label sets on upstream.rpc.conn and 13,199 on request.grpc.native.response, i.e. a ~240 MB ceiling at 69 buckets.

Ladder	Body at that ceiling	Under 64 MB?
69 (today)	240.0 MB	no
20	77.5 MB	no
15	60.9 MB	yes
13 (this PR)	54.3 MB	yes
12	51.0 MB	yes

A 20-bucket ladder would fix today and break again later. 13 leaves real headroom.

What this does not change

• No metric is added or removed, and no label changes. _sum, _count, _max are untouched.
• histogram_quantile consumers keep working at coarser resolution. Eight panels across two dashboards depend on these buckets, which is why dropping the bucket families outright was rejected.
• monitoring.extended is unrelated. It gates four small extras (no_matching_upstream, native_call_failure, the FilteredApis summaries, ProxyServer.ExtendedRequestMetrics) and never touched publishPercentileHistogram().

Still worth doing separately

This PR caps the per-label-set cost. It does not stop label sets accumulating, so the body still grows — about 4.6× slower. Two follow-ups, deliberately out of scope here:

1. Bound the two timer caches — evict a method's timer after an idle period, or skip creating one below a call-count threshold. That would remove the 24.7% of label sets carrying a single observation.
2. BlockchainRpc.kt:78-81 builds a timer from the caller-supplied item.method on the request path with no validation, and that cache never evicts. Observed values are all real method names today, so there is no evidence of a problem in practice, but nothing in dshackle bounds it.

Testing

MonitoringSetupSpec covers the filter against a local PrometheusMeterRegistry, so it asserts on real scrape output rather than on config objects:

• a timer with a percentile histogram exposes exactly 13 buckets, not 69;
• the boundaries come out in seconds (min 0.001, max 30.0) — this is the regression test for the nanosecond conversion, which silently produces sub-nanosecond buckets if you get it backwards;
• a timer without a percentile histogram exposes no buckets;
• a DistributionSummary with a percentile histogram does not get the latency ladder.

I could not compile or run this locally — there is no JDK 21 on the machine I worked from, so I am relying on CI (make test, which runs ./gradlew check including ktlint) for compile and test verification. Every Micrometer API used here was checked against the published micrometer-core:1.16.5 sources rather than from memory: MeterFilter.configure receives the post-map id, serviceLevelObjectives takes double... in nanoseconds for timers, merge lets the receiver win, and with percentilesHistogram(false) only the SLO boundaries become buckets with no min/max clamping.

Verification after deploy

# should read 13, not 69
count by (job) (count by (job, le) (dshackle_upstream_rpc_conn_seconds_bucket))

# should drop well under 54,400,000
max by (drpc_cluster, job) (scrape_response_size_bytes{job=~"dshackle.*"})

---