Skip to content

Status freeze persists on the scale and member-deletion early-returns (same class as #367) #371

Description

Follow-up to #367 / #369.

#369 cures the freeze on the converged path (a healthy current == desired cluster whose pods all go down): the promote-after-converged pass now threads its transient requeue through updateStatus instead of returning early, so ReadyMembers / Available / Degraded stop advertising a last-healthy snapshot.

The same freeze class remains on every scale-related early return, which still skip updateStatus:

  1. In-flight EtcdMember deletion — controllers/etcdcluster_controller.go deletion-wait loop returns RequeueAfter: 5s before updateStatus. With surviving peer CRs and etcd down, handleDeletion's removeMemberFromEtcd errors forever, reconciliationComplete is false (count mismatch), and deadlineExpired is false because ProgressDeadline was cleared at the prior convergence. ReadyMembers / Available=True/QuorumHealthy freeze exactly as in EtcdCluster status freezes at its last good value when etcd is unreachable #367. Repro: break-glass member deletion (or the memory-backed self-delete on pod loss), then all pods go down.

  2. Any scale path — the deletion-wait, hasPendingMember → scaleUp, !allMembersReady, scaleDown, and the inner 5s/10s returns in scaleUp / completePendingMember / scaleDown all skip updateStatus. Healthy 3/3, replicas: 5, member 4 added as a learner, then a node shutdown takes all pods down: status advertises healthy with 0 ready pods for the full ProgressDeadlineSeconds, after which handleDeadlineExceeded writes Available=False/DeadlineExceeded but leaves ReadyMembers, Degraded, and the PDB untouched.

Design question to settle first

Both are the "derive member-based status on paths that early-return" problem. The durable shape (raised as the first open question on #367) is to derive the member-based status (ReadyMembers, Available/Degraded, BrokenMembers, PDB) before the scale fan-out — or on every non-error return — and keep it separate from the reconcile-state conditions that other writers own (the cert-manager gate, surfaceDiscoveryError, handleDeadlineExceeded).

The subtlety that makes this more than a mechanical refactor: during progression the ready/desired ratio counts not-yet-joined members against quorum, so a naive always-write flips Degraded on healthy scale-ups (the #369 review's item 1). #369 handles that for the converged path by withholding the health rewrite while Progressing=True && !reconciliationComplete; a general "write health on every return" needs the quorum denominator to be the established voter count during progression, not the desired replica count. Worth agreeing on the shape before landing the broader change.

Deferred from #369 per reviewer suggestion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions