Skip to content

site init: a node CIDR that matches no node is accepted silently #649

Description

Summary

kubectl unbounded site init accepts a --cluster-node-cidr (or --node-cidr)
that no node in the target cluster falls inside, and nothing downstream reports
it. The Site is created, the nodes are never claimed, and the only symptom is
things quietly not working.

This is not hypothetical: it happened on unbounded-stable, where
Site/cluster declares 10.200.0.0/16 while every node is in 10.224.0.0/12.
Those nodes carry no unbounded-cloud.io/* labels at all and belong to no Site.

Why nothing catches it

Validation is syntactic only. cmd/kubectl-unbounded/app/site_init.go:352-358
checks the value is non-empty and parses as an IPv4 CIDR. The flag is required
with no default and no discovery (:484, :497), so the value is whatever the
caller typed. The command already holds a client for the cluster it is
configuring, so it could list nodes and compare.

Node-to-Site assignment fails closed and says nothing.
internal/net/controller/site_controller.go:2534-2563 matches a node to a Site
by testing its InternalIP against every Site's spec.nodeCidrs, and on no match
returns ("", nil, "", false). The caller moves on. No error, no event, no
condition on the Site. An unclaimed node is indistinguishable from one that was
never meant to be claimed.

There is no provenance. managedFields records the field manager
(kubectl-unbounded), which is a self-declared string, not an identity. No
annotation records who applied the Site, when, or with what values. Working out
where a wrong CIDR came from currently requires API-server audit logs, which are
not enabled by default on AKS.

Suggested fixes

Roughly in order of value for effort:

  1. Warn, or fail, when no node matches. In site init, after resolving the
    CIDR, list nodes and report how many fall inside it. Zero is almost always a
    mistake. A warning is enough; the command may legitimately run before any node
    has joined.
  2. Surface it on the Site. A condition or event when a Site's nodeCidrs
    match no node in the cluster. This is the part that would have made the
    unbounded-stable case visible during the five days and one failed release it
    survived.
  3. Record provenance. Annotate the Site with the invoking user, timestamp and
    the CIDR values applied. That turns "who set this and why" into
    kubectl get site <name> -o yaml, with no audit infrastructure, and works on
    clusters that are not AKS.

Notes

Item 1 alone would have prevented this instance. Item 2 is what makes it
recoverable when a Site drifts for some other reason, such as a cluster being
rebuilt into a different VNet while the Site resource stays behind.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions