Skip to content

fix: record conflict failures and fix missing status patch metric #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rawadhossain wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix: record conflict failures and fix missing status patch metric #320

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cc7e8c6
Improve failure metrics
rawadhossain Jul 22, 2026
f1bc875
record errors and logs
rawadhossain Aug 22, 2026
4d3181b
add api conflicts metric
rawadhossain Sep 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion docs/book/src/operations/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Total number of failure events recorded by the controller.
| Label | Description | Values |
| --- | --- | --- |
| `rule` | `NodeReadinessRule` name | Any rule name |
| `reason` | Failure label recorded by the controller | `EvaluationError`, `AddTaintError`, `RemoveTaintError` |
| `reason` | Failure label recorded by the controller | `EvaluationError`, `AddTaintError`, `RemoveTaintError`, `AddTaintConflictExhausted`, `RemoveTaintConflictExhausted`, `StatusPatchError`, `StatusPatchConflictExhausted`, `RuleStatusRuleSweepConflictExhausted` |

### `node_readiness_build_info`

Expand Down Expand Up @@ -141,6 +141,35 @@ Total number of nodes that have completed bootstrap.
| --- | --- | --- |
| `rule` | `NodeReadinessRule` name | Any rule name |

### `node_readiness_api_conflicts_total`

Total number of conflicts encountered on API writes, counted once per failed attempt, including attempts that later succeed on retry.

| Property | Value |
| --- | --- |
| Type | `counter` |
| Labels | `rule`, `operation` |
| Recorded when | Controller retries an API write after a conflicting update from another writer |

#### Labels

| Label | Description | Values |
| --- | --- | --- |
| `rule` | `NodeReadinessRule` name | Any rule name |
| `operation` | Which write conflicted | See table below |

#### Operation values

| `operation` | Triggered by |
| --- | --- |
| `add_taint` | Adding a rule's taint to a Node (`addTaintBySpec`). |
| `remove_taint` | Removing a rule's taint from a Node (`removeTaint`). |
| `mark_bootstrap_completed` | Marking bootstrap as completed when no taint needs to be removed (`markBootstrapCompleted`). |
| `finalizer_add` | Adding the controller finalizer to a `NodeReadinessRule` (`ensureFinalizer`). |
| `finalizer_remove` | Removing the controller finalizer during rule deletion (`reconcileDelete`). |
| `rule_status_node_write` | Updating a single node's evaluation in rule `status` (`processNodeAgainstAllRules`). |
| `rule_status_rule_sweep` | Updating rule status for all nodes or removing deleted nodes (`updateRuleStatus`, `cleanupDeletedNodes`). |

## Reporter Metrics

The `readiness-condition-reporter` serves its own Prometheus metrics on `/metrics`, on the address configured by `METRICS_BIND_ADDRESS`. See [Reporter Configuration](../reference/reporter-configuration.md) for deployment details.
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ module sigs.k8s.io/node-readiness-controller
go 1.26.0

require (
github.com/go-logr/logr v1.4.4
github.com/onsi/ginkgo/v2 v2.32.0
github.com/onsi/gomega v1.40.0
github.com/prometheus/client_golang v1.24.0
Expand All @@ -29,7 +30,6 @@ require (
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/fsnotify/fsnotify v1.10.1 // indirect
github.com/fxamacker/cbor/v2 v2.9.2 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
github.com/go-openapi/jsonpointer v1.0.0 // indirect
Expand Down
Loading
Loading