Skip to content

feat: add enforcement latency metric with operation labels - #450

Open
rawadhossain wants to merge 1 commit into
kubernetes-sigs:mainfrom
rawadhossain:enforcement-latency-seconds
Open

feat: add enforcement latency metric with operation labels#450
rawadhossain wants to merge 1 commit into
kubernetes-sigs:mainfrom
rawadhossain:enforcement-latency-seconds

Conversation

@rawadhossain

@rawadhossain rawadhossain commented Aug 24, 2026

Copy link
Copy Markdown
Member

Description

  • Adds node_readiness_enforcement_latency_seconds with add/remove operation labels, keeping node_readiness_reconciliation_latency_seconds unchanged.
  • Updates scale PromQL queries and the scalability report.
  • Updates monitoring.md and docs/TEST_README.md.
  • Adds related tests.

from the design doc:

image image

Related to #446

Type of Change

/kind feature

Testing

make test, make lint, go vet, gofmt -l all pass.

Checklist

  • make test passes
  • make lint passes

@kubernetes-prow kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 24, 2026
@netlify

netlify Bot commented Aug 24, 2026

Copy link
Copy Markdown

Deploy Preview for node-readiness-controller canceled.

Name Link
🔨 Latest commit d959fad
🔍 Latest deploy log https://app.netlify.com/projects/node-readiness-controller/deploys/6a968d8d4cd10f0009078c8a

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 24, 2026
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please assign haircommander for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow
kubernetes-prow Bot requested review from mrunalp and tallclair August 24, 2026 10:17
@kubernetes-prow

Copy link
Copy Markdown

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 24, 2026
@rawadhossain

Copy link
Copy Markdown
Member Author

/cc @ajaysundark

@kubernetes-prow
kubernetes-prow Bot requested a review from ajaysundark August 26, 2026 08:20
@rawadhossain

Copy link
Copy Markdown
Member Author

/cc @AvineshTripathi

@AvineshTripathi AvineshTripathi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

left some minor question

Comment thread internal/controller/nodereadinessrule_controller.go
Comment thread internal/metrics/metrics_test.go Outdated
labelNames[l.GetName()] = true
}
}
for _, want := range []string{"rule", "operation"} {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this check? CollectAndCompare should already check this right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah fair, thought to have a cleaner failure message thats why put it there, but you're right its redundant. Removed it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: another reason for #447 (review) comment

@kubernetes-prow kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 29, 2026
@AvineshTripathi

Copy link
Copy Markdown
Contributor

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 29, 2026
@rawadhossain
rawadhossain force-pushed the enforcement-latency-seconds branch from 9b1a813 to 68de79c Compare August 29, 2026 14:09
@kubernetes-prow kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 29, 2026
@rawadhossain
rawadhossain force-pushed the enforcement-latency-seconds branch from 68de79c to bd881be Compare August 29, 2026 14:10
@rawadhossain

Copy link
Copy Markdown
Member Author

@AvineshTripathi i dont know why the CI check is failing in every PR. From my end everything looks fine.

@AvineshTripathi

Copy link
Copy Markdown
Contributor

It's seems slack invite redirection is down. I have raised this in their channel. Will ping you there

@AvineshTripathi AvineshTripathi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

left a minor suggestion
Tested the metrics part locally. However, didn't run the scale test though

}

recordLatency := func(operation string) {
recordLatency := func(operation string, enforcementOperation metrics.EnforcementOperation) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not both metrics.<metrics-name>. inconsistency here

@kubernetes-prow kubernetes-prow Bot added lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 30, 2026
@AvineshTripathi

Copy link
Copy Markdown
Contributor

@rawadhossain please resolve the conflicts too. Thanks!

@rawadhossain
rawadhossain force-pushed the enforcement-latency-seconds branch from bd881be to 3ce515a Compare August 30, 2026 16:13
@kubernetes-prow kubernetes-prow Bot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 30, 2026
@rawadhossain
rawadhossain force-pushed the enforcement-latency-seconds branch from 3ce515a to b07d00d Compare August 30, 2026 16:32
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 30, 2026
Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>
@rawadhossain
rawadhossain force-pushed the enforcement-latency-seconds branch from b07d00d to d959fad Compare September 1, 2026 08:32
@rawadhossain

Copy link
Copy Markdown
Member Author

@AvineshTripathi resolved the conflicts and inconsistency.
Also applied the same test refactor here as in #447 (review). lmk if this looks good. Thanks.

@AvineshTripathi AvineshTripathi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

wouldn't we get conflicts when one of the PR gets merged regarding the util func

@kubernetes-prow kubernetes-prow Bot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 5, 2026
@rawadhossain

Copy link
Copy Markdown
Member Author

Yes probably we would. Will have to resolve the conflict afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants