You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#369 cures the freeze on the converged path (a healthy current == desired cluster whose pods all go down): the promote-after-converged pass now threads its transient requeue through updateStatus instead of returning early, so ReadyMembers / Available / Degraded stop advertising a last-healthy snapshot.
The same freeze class remains on every scale-related early return, which still skip updateStatus:
In-flight EtcdMember deletion — controllers/etcdcluster_controller.go deletion-wait loop returns RequeueAfter: 5s before updateStatus. With surviving peer CRs and etcd down, handleDeletion's removeMemberFromEtcd errors forever, reconciliationComplete is false (count mismatch), and deadlineExpired is false because ProgressDeadline was cleared at the prior convergence. ReadyMembers / Available=True/QuorumHealthy freeze exactly as in EtcdCluster status freezes at its last good value when etcd is unreachable #367. Repro: break-glass member deletion (or the memory-backed self-delete on pod loss), then all pods go down.
Any scale path — the deletion-wait, hasPendingMember → scaleUp, !allMembersReady, scaleDown, and the inner 5s/10s returns in scaleUp / completePendingMember / scaleDown all skip updateStatus. Healthy 3/3, replicas: 5, member 4 added as a learner, then a node shutdown takes all pods down: status advertises healthy with 0 ready pods for the full ProgressDeadlineSeconds, after which handleDeadlineExceeded writes Available=False/DeadlineExceeded but leaves ReadyMembers, Degraded, and the PDB untouched.
Design question to settle first
Both are the "derive member-based status on paths that early-return" problem. The durable shape (raised as the first open question on #367) is to derive the member-based status (ReadyMembers, Available/Degraded, BrokenMembers, PDB) before the scale fan-out — or on every non-error return — and keep it separate from the reconcile-state conditions that other writers own (the cert-manager gate, surfaceDiscoveryError, handleDeadlineExceeded).
The subtlety that makes this more than a mechanical refactor: during progression the ready/desired ratio counts not-yet-joined members against quorum, so a naive always-write flips Degraded on healthy scale-ups (the #369 review's item 1). #369 handles that for the converged path by withholding the health rewrite while Progressing=True && !reconciliationComplete; a general "write health on every return" needs the quorum denominator to be the established voter count during progression, not the desired replica count. Worth agreeing on the shape before landing the broader change.
Follow-up to #367 / #369.
#369 cures the freeze on the converged path (a healthy
current == desiredcluster whose pods all go down): the promote-after-converged pass now threads its transient requeue throughupdateStatusinstead of returning early, soReadyMembers/Available/Degradedstop advertising a last-healthy snapshot.The same freeze class remains on every scale-related early return, which still skip
updateStatus:In-flight
EtcdMemberdeletion —controllers/etcdcluster_controller.godeletion-wait loop returnsRequeueAfter: 5sbeforeupdateStatus. With surviving peer CRs and etcd down,handleDeletion'sremoveMemberFromEtcderrors forever,reconciliationCompleteis false (count mismatch), anddeadlineExpiredis false becauseProgressDeadlinewas cleared at the prior convergence.ReadyMembers/Available=True/QuorumHealthyfreeze exactly as in EtcdCluster status freezes at its last good value when etcd is unreachable #367. Repro: break-glass member deletion (or the memory-backed self-delete on pod loss), then all pods go down.Any scale path — the deletion-wait,
hasPendingMember → scaleUp,!allMembersReady,scaleDown, and the inner5s/10sreturns inscaleUp/completePendingMember/scaleDownall skipupdateStatus. Healthy 3/3,replicas: 5, member 4 added as a learner, then a node shutdown takes all pods down: status advertises healthy with 0 ready pods for the fullProgressDeadlineSeconds, after whichhandleDeadlineExceededwritesAvailable=False/DeadlineExceededbut leavesReadyMembers,Degraded, and the PDB untouched.Design question to settle first
Both are the "derive member-based status on paths that early-return" problem. The durable shape (raised as the first open question on #367) is to derive the member-based status (
ReadyMembers,Available/Degraded,BrokenMembers, PDB) before the scale fan-out — or on every non-error return — and keep it separate from the reconcile-state conditions that other writers own (the cert-manager gate,surfaceDiscoveryError,handleDeadlineExceeded).The subtlety that makes this more than a mechanical refactor: during progression the ready/desired ratio counts not-yet-joined members against quorum, so a naive always-write flips
Degradedon healthy scale-ups (the #369 review's item 1). #369 handles that for the converged path by withholding the health rewrite whileProgressing=True && !reconciliationComplete; a general "write health on every return" needs the quorum denominator to be the established voter count during progression, not the desired replica count. Worth agreeing on the shape before landing the broader change.Deferred from #369 per reviewer suggestion.