Premature Consolidatable condition triggers create-delete cycle on uninitialized nodes

[tgoodwin]

Background

I'm developing a tool that systematically explores controller reconciliation ordering, staleness, and fault injection (kamera).

Observed Behavior

I observe that when the disruption controller marks a node as Consolidatable before the node is fully initialized (pods scheduled, kubelet registered), a subsequent disruption evaluation treats the node as a valid consolidation candidate and enqueues deletion. The provisioner then creates a replacement, producing a create-delete cycle.

The consolidation.ShouldDisrupt() predicate checks ConditionTypeConsolidatable (line 122) but does not gate on Initialized(). An uninitialized node has no pods scheduled yet, so it appears "empty" and is marked Consolidatable — triggering deletion before the node has had a chance to receive workloads. This occurs whenever the disruption controller evaluates the node before initialization completes.

Expected Behavior

The disruption controller should not evaluate nodes for consolidation until they are fully initialized (all expected pods scheduled, NodeClaim initialized, Node registered).

Proposed Fix

The consolidation.ShouldDisrupt() predicate at consolidation.go:88-123 checks ConditionTypeConsolidatable (line 122) but does not check whether the node is initialized. In contrast, ValidateNodeDisruptable() at statenode.go:210 correctly gates on Initialized() (which checks for the NodeInitializedLabelKey label at statenode.go:343-350).

possible fix: add an Initialized() check at the top of consolidation.ShouldDisrupt():

func (c *Consolidation) ShouldDisrupt(_ context.Context, cn *Candidate) bool {
    if !cn.Initialized() {
        return false
    }
    // ... existing checks ...
}

This mirrors the pattern in ValidateNodeDisruptable() and ensures nodes are never considered for consolidation before they are fully initialized. Without this check, the node appears "empty" (no pods scheduled yet) and is immediately marked Consolidatable, triggering a deletion before the node has had a chance to receive workloads.

I'm happy to put up a PR for this if it would be helpful.

Versions

• Karpenter: v1.8.0 (sigs.k8s.io/karpenter, commit 8ae07cf8)
• Kubernetes: simulated via kamera (based on k8s.io/client-go v0.35.0 / Kubernetes 1.35)

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ryan-mist]

I observe that when the disruption controller marks a node as Consolidatable before the node is fully initialized (pods scheduled, kubelet registered)

Do you have the logs for when this occurs? We should only be setting Consolidatable to true after initialization -

karpenter/pkg/controllers/nodeclaim/disruption/consolidation.go

Line 73 in 07260ac

nodeClaim.StatusConditions().SetTrue(v1.ConditionTypeConsolidatable)

[tgoodwin]

Thanks for looking at this, @ryan-mist! I realized my original description was inaccurate. When I said "before the node is fully initialized," what I really meant was "before the node is initialized and the pod it was created for has been bound to it." You're right that the consolidation reconciler guards against setting Consolidatable before Initialized is true, and NewCandidate also checks ValidateNodeDisruptable() which gates on Initialized().

The issue occurs after initialization:

1. Provisioner creates a NodeClaim for a pending pod
2. NodeClaim completes its lifecycle (Initialized=True)
3. nodeclaim.disruption evaluates: Initialized is true, consolidateAfter: 0s has elapsed → sets Consolidatable=True
4. disruption evaluates the node. No pods are bound yet (the scheduler hasn't placed the pending pod), so the node appears empty → marks for deletion
5. Provisioner creates a replacement for the still-pending pod

The result is a race that produces create-delete-create churn (liveness issue).

The node passes both the consolidation reconciler's Initialized check and ValidateNodeDisruptable(). The gap is between "node is initialized" and "the pod that triggered provisioning has been scheduled to it." consolidateAfter: 0s makes this race trivial to reproduce, but the window exists for any value where the scheduler hasn't bound the pod before the timer expires.

My initial fix suggestion was incorrect. Looking at the code more, I noticed that ValidateNodeDisruptable already checks Nominated(), and the provisioner's scheduler calls NominateNodeForPod for existing nodes it routes pods to (scheduler.go:262). But new NodeClaims don't get nominated since they don't have a ProviderID at scheduling time. Nomination already operates at the StateNode level, so extending it to cover newly provisioned nodes (e.g., calling Nominate() when a new StateNode is first added to cluster state via UpdateNodeClaim) would cause ValidateNodeDisruptable to reject the new node as a disruption candidate, giving the scheduler time to bind the pending pod before consolidation can act.

[ryan-mist]

Even if consolidateAfter is 0 seconds, we have an additional validation period for any disruption commands

karpenter/pkg/controllers/disruption/validation.go

Line 133 in defdfae

func (e *EmptinessValidator) Validate(ctx context.Context, cmd Command, validationPeriod time.Duration) (Command, error) {

When we create a disruption command, we wait 15 seconds before revalidating it. This should give the scheduler plenty of time to bind the pod to the node, in which case we would invalidate the command.

[nathangeology]

It's also worth adding that it would be dangerous to wait for pod binding for too long because while Karpenter simulates placement, it doesn't control it and so we can't really be 100% sure what is going to show up on the node in the end.