feat: collector-based rule_matched_nodes and typed failure reasons by rawadhossain · Pull Request #286 · kubernetes-sigs/node-readiness-controller

rawadhossain · 2026-07-02T16:07:42Z

Description

Adds node_readiness_rule_matched_nodes metric to the scrape-time collector pattern introduced in #398.

The metric is computed on each scrape from the cached Node list, sharing the same snapshot as node_readiness_rule_nodes. Dry-run and deleting rules are also included.

(exisitng) `node_readiness_failures_total` (new reason: `AnnotationPatchFailed`)

Also adds AnnotationPatchFailed as a new reason for the existing node_readiness_failures_total metric. Tracks failures when writing the bootstrap-completion annotation, keeping all failure reporting under the existing failures_total counter.

Changes

Moved node_readiness_rule_matched_nodes to the scrape-time ReadinessCollector.
Shares the Node list with node_readiness_rule_nodes.
Includes dry-run and deleting rules.
Added AnnotationPatchFailed to node_readiness_failures_total.
Failure reasons (EvaluationError, AddTaintError, RemoveTaintError, AnnotationPatchFailed) are now defined as a type with named constants.

Related to #397

Type of Change

/kind feature

Testing

make test, make lint, go test ./... -race all pass
Verified the metric against a live cluster and confirmed the counts match the current Node selectors.

Checklist

make test passes
make lint passes

netlify · 2026-07-02T16:07:47Z

✅ Deploy Preview for node-readiness-controller ready!

Name	Link
🔨 Latest commit	`5278d1d`
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a98f2fd76f4f200089e568c
😎 Deploy Preview	https://deploy-preview-286--node-readiness-controller.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

kubernetes-prow · 2026-07-02T16:07:52Z

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

AvineshTripathi

Thanks for the PR! Left a few comments.

rawadhossain · 2026-07-05T07:39:37Z

@AvineshTripathi PTAL. I'll commit the changes once you approve.
Thanks.

AvineshTripathi · 2026-07-07T18:24:50Z

@rawadhossain I left one comment PHAL and maybe remove the Bootstrap duration metrics till we merge #224

rawadhossain · 2026-07-08T12:58:26Z

@AvineshTripathi I have made the changes. PTAL.
Thanks.

AvineshTripathi · 2026-07-20T05:35:04Z

/ok-to-test

AvineshTripathi · 2026-07-23T18:19:02Z

/lgtm

cc @ajaysundark for final look

rawadhossain · 2026-08-09T07:29:55Z

updated the description. Thanks.

AvineshTripathi · 2026-08-09T17:16:23Z

/lgtm

/assign @ajaysundark
for review and approve

kubernetes-prow · 2026-08-15T10:39:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please ask for approval from ajaysundark. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

AvineshTripathi · 2026-08-17T17:44:28Z

@rawadhossain can you please resolve the conflicts?

rawadhossain · 2026-08-17T18:27:24Z

@AvineshTripathi resolved the conflicts. GitHub was down for quite a while 😥
PTAL.

rawadhossain · 2026-08-19T05:28:00Z

@ajaysundark if everything looks right here, can we get it merged?
Will do the collector migration for rule_matched_nodes as a follow-up PR once this land.

rawadhossain · 2026-08-19T20:39:24Z

@ajaysundark made the changes based on scrape time collector and updated the description.

Could we get #431 (blocked_nodes) reviewed and merged first? This PR has a small adjustment that depends on the changes in #431. Once it's merged, I'll make that change here and push the final commit.

rawadhossain · 2026-08-31T19:12:54Z

@ajaysundark made the adjustments here after 431 landed. I think this should be good to merge.
PTAL. Thanks!

AvineshTripathi · 2026-09-02T19:38:59Z

/lgtm

kubernetes-prow · 2026-09-03T04:09:34Z

New changes are detected. LGTM label has been removed.

ajaysundark · 2026-09-03T08:28:31Z

 		}
 	}
+
+	matched, err := c.lister.ListRuleMatchedNodes(ctx, nodes, rules)


@rawadhossain Could you check if we are scraping these metrics as a single-pass instead of scraping the cache individually for each of these scrape time metrics?

Yeah, checked it. All three scrape time metrics scrape the node and rule cache only once, not three individual cache reads. Also tested it on a live cluster to double check, and everything matched up correctly.

kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 2, 2026

kubernetes-prow Bot requested review from dchen1107 and haircommander July 2, 2026 16:07

rawadhossain force-pushed the feat/add-observability-metrics branch from 406dde6 to 11900ae Compare July 3, 2026 15:07

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 3, 2026

ajaysundark self-requested a review July 3, 2026 21:22

AvineshTripathi reviewed Jul 4, 2026

View reviewed changes

Comment thread internal/controller/nodereadinessrule_controller.go Outdated

Comment thread internal/metrics/metrics.go Outdated

Comment thread internal/metrics/metrics.go Outdated

Comment thread internal/controller/nodereadinessrule_controller.go Outdated

rawadhossain force-pushed the feat/add-observability-metrics branch from 11900ae to 12ab0d9 Compare July 5, 2026 07:32

kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2026

ajaysundark requested review from AvineshTripathi and removed request for dchen1107 and haircommander July 7, 2026 21:33

rawadhossain force-pushed the feat/add-observability-metrics branch from 12ab0d9 to 8fcbeca Compare July 8, 2026 10:01

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 8, 2026

rawadhossain force-pushed the feat/add-observability-metrics branch from 8fcbeca to 3051634 Compare July 8, 2026 10:20

kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 20, 2026

kubernetes-prow Bot assigned AvineshTripathi Jul 23, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 23, 2026

kubernetes-prow Bot assigned ajaysundark Aug 9, 2026

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 9, 2026

ajaysundark reviewed Aug 14, 2026

View reviewed changes

Comment thread internal/metrics/metrics.go Outdated

rawadhossain force-pushed the feat/add-observability-metrics branch from db7d7e2 to 4bc164e Compare August 15, 2026 10:39

kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 15, 2026

rawadhossain force-pushed the feat/add-observability-metrics branch from 4bc164e to 0034c33 Compare August 15, 2026 10:50

kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2026

rawadhossain force-pushed the feat/add-observability-metrics branch from 0034c33 to aa9c0a4 Compare August 17, 2026 18:25

kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 17, 2026

rawadhossain mentioned this pull request Aug 19, 2026

[FEATURE] Scrape-Time Metrics Collector #397

Closed

3 tasks

rawadhossain force-pushed the feat/add-observability-metrics branch from aa9c0a4 to 4ea706d Compare August 19, 2026 20:12

rawadhossain mentioned this pull request Aug 23, 2026

REQUEST: New membership for rawadhossain kubernetes/org#6576

Closed

11 tasks

rawadhossain changed the title ~~feat: Add selector and bootstrap observability metrics~~ feat: collector-based rule_matched_nodes and typed failure reasons Aug 31, 2026

rawadhossain mentioned this pull request Sep 1, 2026

[BUG] ListBlockedNodes incorrectly reports satisfied anyOf nodes as "blocked" in metrics #458

Open

kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 2, 2026

rawadhossain added 2 commits September 3, 2026 09:28

bootstrap failures and use reason constants

c6080ad

rule matched nodes to scrape-time collector

5278d1d

rawadhossain force-pushed the feat/add-observability-metrics branch from 4ab42a3 to 5278d1d Compare September 3, 2026 04:09

kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 3, 2026

ajaysundark reviewed Sep 3, 2026

View reviewed changes

Uh oh!

Conversation

rawadhossain commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

(exisitng) node_readiness_failures_total (new reason: AnnotationPatchFailed)

Changes

Related to #397

Type of Change

Testing

Checklist

Uh oh!

netlify Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for node-readiness-controller ready!

Uh oh!

kubernetes-prow Bot commented Jul 2, 2026

Uh oh!

AvineshTripathi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rawadhossain commented Jul 5, 2026

Uh oh!

AvineshTripathi commented Jul 7, 2026

Uh oh!

rawadhossain commented Jul 8, 2026

Uh oh!

AvineshTripathi commented Jul 20, 2026

Uh oh!

AvineshTripathi commented Jul 23, 2026

Uh oh!

rawadhossain commented Aug 9, 2026

Uh oh!

AvineshTripathi commented Aug 9, 2026

Uh oh!

Uh oh!

kubernetes-prow Bot commented Aug 15, 2026

Uh oh!

AvineshTripathi commented Aug 17, 2026

Uh oh!

rawadhossain commented Aug 17, 2026

Uh oh!

rawadhossain commented Aug 19, 2026

Uh oh!

rawadhossain commented Aug 19, 2026

Uh oh!

rawadhossain commented Aug 31, 2026

Uh oh!

AvineshTripathi commented Sep 2, 2026

Uh oh!

kubernetes-prow Bot commented Sep 3, 2026

Uh oh!

ajaysundark Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rawadhossain Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rawadhossain commented Jul 2, 2026 •

edited

Loading

(exisitng) `node_readiness_failures_total` (new reason: `AnnotationPatchFailed`)

netlify Bot commented Jul 2, 2026 •

edited

Loading

ajaysundark Sep 3, 2026 •

edited

Loading