fix: record conflict failures and fix missing status patch metric

[rawadhossain]

Description

Adds node_readiness_api_conflicts_total, tracking every conflict and retry across 7 places:

• adding/removing a taint
• marking bootstrap done
• adding/removing the rule finalizer
• updating a node's status or the whole rule's status.

Earlier it was just logs and most of these conflicts couldn't actually happen. Now that #343 is in, conflicts are real, so bringing the metric to handle them.

What's added

• node_readiness_api_conflicts_total with all 7 operations, each tested with a real forced conflict.
• New failure reason RuleStatusRuleSweepConflictExhausted: full rule status update can exhaust its retries under load, and that wasn't tracked before.
• monitoring.md updated.

Related to Issue #182

Type of Change

/kind feature

Checklist

• make test passes
• make lint passes

[kubernetes-prow]

@rawadhossain: The label(s) kind/improvement cannot be applied, because the repository doesn't have them.

Details

In response to this:

Description

This change improves failure observability by recording conflict-specific failure reasons in node_readiness_failures_total and fixing a missing failure metric in the node reconciler.

Previously, conflict exhausted taint operations were recorded as generic taint errors, so it wasn't possible to distinguish them from other patch errors. The rule status patch error path in the node reconciler also logged failures without recording a failure metric.

To fix this:

• add conflict-specific failure reasons:
• AddTaintConflictExhausted
• RemoveTaintConflictExhausted
• StatusPatchConflictExhausted
• record status patch failures in the node reconciler
• add a structured operation field to evaluateRuleForNode error logs
• update monitoring.md with the new reason label values

Related to Issue #182

Type of Change

/kind improvement

Checklist

• make test passes
• make lint passes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please assign mrunalp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for node-readiness-controller ready!

Name	Link
🔨 Latest commit	4d3181b
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a9d2a61b298e50008867005
😎 Deploy Preview	https://deploy-preview-320--node-readiness-controller.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[kubernetes-prow]

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[AvineshTripathi]

left my initial thoughts here!

[AvineshTripathi]

cc @Priyankasaggu11929 I remember you had some thoughts on this

[rawadhossain]

Hi @AvineshTripathi
gave this a fresh look. Actually built api_conflicts_total locally and live tested it to see if it was worth keeping. Most of what it would show either duplicates the ConflictExhausted reasons already in this PR, or just stay at zero where conflicts don't happen.
So I reworked this PR to go with the logs approach instead, which also addresses your earlier comments. PTAL.

[ajaysundark]

The largest source of api conflicts (409s) don't rise up because they are handled as merge patch. We should check this after #343 is merged, only after when status conflicts will 'surface'

[rawadhossain]

cc @ajaysundark for some comments to get holistic context.

[rawadhossain]

@ajaysundark
added the metric and possible conflict operations. Tested every scenario, and working as expected.
Lmk if any changes needed. Thanks!