Skip to content

feat: label the CRD with its managing installation and guard duplicat… - #440

Open
yindia wants to merge 1 commit into
kubernetes-sigs:mainfrom
yindia:feat/438-managed-by-label
Open

feat: label the CRD with its managing installation and guard duplicat…#440
yindia wants to merge 1 commit into
kubernetes-sigs:mainfrom
yindia:feat/438-managed-by-label

Conversation

@yindia

@yindia yindia commented Aug 22, 2026

Copy link
Copy Markdown

Summary

Stamps the NodeReadinessRule CRD with a managed-by label naming the installation that owns it, and refuses a fresh helm install when the CRD already exists.

Design notes

Why the CRD carries the label. It's cluster scoped and singleton, so it's the one artifact every install flow contends over. It's set with a controller-gen marker, so make manifests regenerates it in config/crd/bases, and verify-chart-drift.sh diffs that against the chart's crds/ copy to catch the two drifting apart.

Why a template guard, not a pre-install hook. The Helm guard uses lookup + fail in a template — same outcome without a ServiceAccount, ClusterRole or image, and it aborts at render time so the guidance lands in the terminal rather than Job logs.

Why it's keyed on .Release.IsInstall plus CRD existence, not the label value. Both installs may be carrying the default value, and that's precisely the accident being guarded against, so a value comparison would pass straight through it. Upgrades skip the check naturally.

Two things worth a reviewer's attention

Label is CRD-only. Helm 3 forces managed-by: Helm on resources it owns, so stamping the Deployment/RBAC/Services doesn't stick. The CRD works because Helm neither templates nor tracks crds/.

Guard only covers helm install. kubectl apply, kustomize and the static-pod path have none. A controller startup check would close the gap, but it's ruled out by alternative (1) in the issue — left out and documented. Happy to reconsider.

Also included

Chart-rendered NodeReadinessRule CRs now get the standard chart labels, which they were missing. Unrelated tidy-up, happy to split it out.

Related Issue

Fixes #438

Type of Change

/kind feature

Testing

  • helm install into a clean cluster: CRD is created with app.kubernetes.io/managed-by=node-readiness-controller.
  • helm install with a second release name while the CRD exists: render aborts with the guidance message, nothing is applied.
  • helm upgrade on the existing release: check is skipped, upgrade succeeds.
  • helm template output diffed against dist/crds.yaml; verify-chart-drift.sh passes.

Checklist

  • make test passes
  • make lint passes

Does this PR introduce a user-facing change?

Yes

`helm install` now fails when a `NodeReadinessRule` CRD from another installation is already present in the cluster. The CRD carries a `managed-by` label identifying the owning installation.

@kubernetes-prow kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 22, 2026
@netlify

netlify Bot commented Aug 22, 2026

Copy link
Copy Markdown

Deploy Preview for node-readiness-controller canceled.

Name Link
🔨 Latest commit aad7ea1
🔍 Latest deploy log https://app.netlify.com/projects/node-readiness-controller/deploys/6a96c9db66a4aa0008e37332

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 22, 2026
@kubernetes-prow

Copy link
Copy Markdown

Welcome @yindia!

It looks like this is your first PR to kubernetes-sigs/node-readiness-controller 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/node-readiness-controller has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@kubernetes-prow kubernetes-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 22, 2026
@kubernetes-prow

Copy link
Copy Markdown

Hi @yindia. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 22, 2026
@yindia

yindia commented Aug 22, 2026

Copy link
Copy Markdown
Author

@ajaysundarkI'm not sure it's the best approach, it false-positives on reinstall since Helm keeps crds/ CRDs after uninstall. Open to a better ideas if anyone has one

@DsThakurRawat

Copy link
Copy Markdown
Contributor

On the uninstall/reinstall false positive:

The reason lookup on the CRD trips is that Helm 3 leaves crds/ in place on helm uninstall by design. If crd-ownership-check.yaml fails solely on CRD presence, any standard helm uninstall -> helm install cycle in the same cluster gets blocked.

Two ways to resolve this without dropping the guard:

  1. Guard on active workload presence rather than CRD presence: query for the Deployment in the target namespace (lookup "apps/v1" "Deployment" .Release.Namespace (include "node-readiness-controller.fullname" .)). Helm deletes the Deployment on uninstall, so a fresh install after an uninstall proceeds cleanly, while a true concurrent installation in that namespace is caught.
  2. If the goal is catching hosted provider installs where the Deployment lives in a hidden namespace, check the label value rather than existence: only abort if $existing.metadata.labels["app.kubernetes.io/managed-by"] exists AND is not equal to "Helm" / our default manager. That way, lingering unmanaged CRDs from a previous uninstall do not block a fresh install.

@yindia

yindia commented Aug 25, 2026

Copy link
Copy Markdown
Author

@DsThakurRawat Thanks for the detailed look. This finds the false positive well.

Option 2 does not work here. Both the provider install and my install carry the default managed-by value. A test for != "Helm" passes through the exact case that the guard must catch. This is the reason I keyed the check on CRD existence, not on the label value.

Option 1 fixes the uninstall and reinstall cycle. But it misses the main case. A provider can run the controller in a control-plane namespace that I cannot see. Then there is no Deployment in .Release.Namespace to find. That hidden install is the reason the guard exists.

I think one version gets both. Keep the live workload as the signal, as you suggested. But look it up across all namespaces, not only the release namespace:

lookup "apps/v1" "Deployment" "" ""

Then match on app.kubernetes.io/name=node-readiness-controller. A leftover CRD after an uninstall has no running Deployment. So a fresh install works. A controller in any namespace, including a hidden one, still trips the guard. The cost is one cluster-wide Deployment read at install time. The installing user normally has this access.

Does this match the reinstall case that you saw? If yes, I will push it as a follow-up commit.

cc: @ajaysundark

@DsThakurRawat

Copy link
Copy Markdown
Contributor

yes, that matches the case i saw, and i checked it on a live cluster rather than reasoning about it. kind v1.31.2, helm 3.16.3, your branch at 55e6a23.

the uninstall half holds up: helm uninstall leaves the CRD behind (it ships from crds/, which helm 3 never removes) and does delete the Deployment, so a live workload really is the signal that separates the two states. your cluster-wide version catches the hidden install too. i faked one by putting a Deployment labelled app.kubernetes.io/name=node-readiness-controller into a namespace the installer would never look in, and lookup "apps/v1" "Deployment" "" "" found it; after deleting that Deployment the same lookup found nothing. so the answer to your question is yes.

while setting that up i hit something bigger than the reinstall case though, and i think it blocks the PR as it stands. the guard fires on a first install into an empty cluster. helm applies crds/ before it renders templates, so by the time crd-ownership-check.yaml runs its lookup, the CRD that this same helm install created a moment earlier is already there.

from a verified clean state, twice:

$ kubectl get crd | grep -c readiness
0
$ helm list -A --short | wc -l
0
$ helm install nrc charts/node-readiness-controller -n nrc-system --create-namespace
Error: INSTALLATION FAILED: execution error at (node-readiness-controller/templates/crd-ownership-check.yaml:31:4):
node-readiness-controller is already installed in this cluster.
$ helm list -A --short | wc -l
0
$ kubectl get crd nodereadinessrules.readiness.node.x-k8s.io -o jsonpath='{.metadata.creationTimestamp}'
2026-08-25T21:06:59Z

that timestamp is from the install that then refused itself, and no release was recorded. --set crds.preInstallCheck=false installs fine, and i confirmed helm upgrade still works with the guard left on, so the .Release.IsInstall gate is doing its job.

two reasons this got past CI, and i think both are worth fixing next to it:

  • the new "Verify the CRD ownership pre-install check" step runs kubectl apply -f "$CHART/crds/" before it tries the install, so it only ever exercises the path where the CRD legitimately pre-exists. nothing covers the first install.
  • the Helm workflow on this head is sitting at action_required and has not actually run yet, so ct install never got the chance to fail either. i suspect it would.

if you do go with the cluster-wide Deployment lookup, two things bit me while writing a probe chart for it.

lookup with an empty name returns a list object, and that object is truthy even when nothing matched. on my cluster the returned dict had keys [apiVersion items kind metadata] with items empty, so a bare if $deployments is always true. you have to range over .items and count what actually matched.

the other one is that a Deployment carrying no labels at all is ordinary rather than an edge case. one of the two in a stock kind cluster is local-path-storage/local-path-provisioner and it has none, so index .metadata.labels "app.kubernetes.io/name" on it dies with index of untyped nil. i had to switch to get (.metadata.labels | default dict) "app.kubernetes.io/name" to get through a full pass.

could be wrong about the ordering if you get a different result, but two clean runs said the same thing. can send the probe chart over if it saves you the setup.

@yindia
yindia force-pushed the feat/438-managed-by-label branch from 55e6a23 to 8103423 Compare August 26, 2026 18:24
@yindia

yindia commented Aug 26, 2026

Copy link
Copy Markdown
Author

You're right on all counts, and thanks for testing it on a live cluster. I reproduced the same thing: a fresh helm install into an empty cluster refuses itself, because Helm applies crds/ before it renders templates, so the guard's lookup finds the CRD this install just created. That's a real blocker, not the reinstall edge case.

Fixed by keying the guard on the controller Deployment instead of the CRD. The Deployment is applied after rendering, so one found during the check belongs to a different, already-running install. Both lookup foot-guns you hit are handled: I range .items and count matches rather than testing the list for truthiness, and I default the label map before indexing (get (.metadata.labels | default dict) ...) so a label-less Deployment like local-path-provisioner doesn't panic.

CI was passing because the verify step ran kubectl apply -f crds/ first, so it only ever exercised the pre-existing path. I changed it to install into an empty cluster with the guard on first (the path that was broken), then assert the second install is refused, the escape hatch works, and upgrade works.

Verified on kind:

  • first install, empty cluster, guard on: succeeds
  • second install while one runs: refused, names the Deployment
  • crds.preInstallCheck=false: installs
  • helm upgrade with guard on: works

No need for the probe chart, but thanks for the offer. Pushed the fix.

@ajaysundark

Copy link
Copy Markdown
Contributor

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 30, 2026
@ajaysundark

Copy link
Copy Markdown
Contributor

/assign @ajaysundark

annotations:
controller-gen.kubebuilder.io/version: v0.19.0
labels:
app.kubernetes.io/managed-by: node-readiness-controller

@ajaysundark ajaysundark Aug 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIU, this is set as 'managed-by: kustomize', we can default to managed-by: helm' for charts. If there's way to expose this to be user-set value in helm, that'd be preferred.

I expect in managed-environments this would map to managed-by: <provider>, we should document as best-practice. so if there's conflict, it'd be properly bubble up as an error

@yindia
yindia force-pushed the feat/438-managed-by-label branch from 8103423 to da1e7ab Compare September 1, 2026 12:40
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: yindia
Once this PR has been reviewed and has the lgtm label, please ask for approval from ajaysundark. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@yindia
yindia force-pushed the feat/438-managed-by-label branch from da1e7ab to aad7ea1 Compare September 1, 2026 12:49
@yindia
yindia requested a review from ajaysundark September 1, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] add a managed-by label to CRD

3 participants