Skip to content

metrics: replace Resource Control RU Max with natural-second peaks - #71408

Draft
JmPotato wants to merge 1 commit into
pingcap:masterfrom
JmPotato:codex/ru-minute-peak
Draft

JmPotato wants to merge 1 commit into
pingcap:masterfrom
JmPotato:codex/ru-minute-peak

Conversation

@JmPotato

@JmPotato JmPotato commented Sep 20, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #71407, ref tikv/pd#11256

Problem Summary:

The legacy Resource Control Max RU panel groups consumption by report arrival and displays separate read/write maxima over 20 seconds. Those values cannot reliably represent the busiest accounting second of the resource group across clients.

What changed and how does it work?

Replace panel 5 in both classic and NextGen Resource Control dashboards with RU Max - 1s, preserving its ID, position beside RU and existing {{resource_group}} legend convention. Plot the combined RRU+WRU peak supplied by PD in RU/s; keep replica/keyspace labels intact and do not sum already-aggregated peaks. One-minute minimum resolution and max_over_time preserve peaks at longer display intervals. Availability and sample-count checks leave incomplete intervals as gaps.

Upgrade PD client and KVProto dependencies and regenerate DEPS.bzl. The latest PD client requires Go 1.26.8; align the module, Bazel SDK, CI setup and builder images with that requirement. The new builder image digest was resolved and verified against the registry manifest. No audit hooks, workload generators or legacy comparison panels are included in the production diff.

Dependency / merge gate: this draft depends on KVProto #1539 → PD #11293. Immutable, checksum-verified fork replacements make the stack buildable for review. Before merging, change both to official merged module versions and regenerate module sums and Bazel metadata. The peak metric remains opt-in through PD's enable-ru-minute-peak setting; deployments must upgrade all participating producers before enabling it.

The panel describes the preceding completed minute, the 30-second publication delay and excluded accounting models. The existing RU panel continues to show average rates. The new view has a single total series per metric identity, replacing the old independent read/write peak curves.

Check List

Tests

  • Unit test

  • Integration test

  • Manual test (add detailed scripts or steps below)

    Ready validation: make bazel_prepare, make lint, GOTOOLCHAIN=go1.26.8 go build -tags nextgen ./cmd/tidb-server, and the Classic build with the same toolchain and no build tags. Dashboard JSON was regenerated with the repository's grafonnet dependency, compared byte-for-byte, and checked to change only panel 5 while preserving its position and resource-group legend. Local Bazel preparation used JVM proxy settings and an officially checksum-verified SDK archive to accommodate this host's network.

    Final real-cluster evidence, screenshot and offline verifier: after the PD leadership-handoff fix, 5,244 confirmed accounting events and 2/2 complete minute peaks matched the logs, exporter, Prometheus and actual Mac mini Grafana datasource. Maximum absolute error: 2.114575181622058e-11 RU. Publication was observed 30.465–30.607 seconds after minute end, with zero capture errors. The earlier five-minute random run is retained with separate identities. The final server-only PD fix does not change the pinned PD client module. Full methodology and limits are described in PD #11293.

    Docker images and the full TiDB regression suite were not run locally; the builder manifest was verified, and standard build/CI configuration is aligned with the dependency's minimum Go version.

  • No need to test

    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Replace the Resource Control 20-second Max RU panel with natural-second RU peaks, preserving peak values at longer display intervals and leaving incomplete data as gaps. Requires enabling RU minute-peak observation in PD.

Signed-off-by: JmPotato <github@ipotato.me>
@ti-chi-bot

ti-chi-bot Bot commented Sep 20, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Sep 20, 2026
@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot

ti-chi-bot Bot commented Sep 20, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cfzjywxk, nolouch for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metrics: replace arrival-based RU Max with natural-second peaks

1 participant