Harden terminal-Pod replacement: cap the retry loop and backfill Status.PodUID

[lllamnyp]

Background

#370 gave the EtcdMember controller a new self-heal path: when a member's Pod reaches a terminal phase (Succeeded/Failed), the controller writes MemberReady=False, deletes the Pod, and lets the next reconcile recreate it (PVC-backed) or replace the member (memory-backed). That fixed #368 — a whole-cluster node reboot used to leave every etcd Pod terminal and the cluster down until the Pods were deleted by hand.

The path is correct as merged. Three follow-ups would harden it; none of them were worth blocking #370 on.

1. The replacement path has no cap and no backoff

The terminal-Pod branch has neither a counter (compare dataLossRestartThreshold = 5 on the crash-loop path) nor a rate limit. A Pod that lands in Failed deterministically — repeated admission failure (UnexpectedAdmissionError, OutOfcpu) or repeated eviction under node memory pressure — produces an unbounded create → Failed → delete → create loop at roughly the requeue cadence, one loop per affected member, minting a new Pod object each turn.

It never escalates to the existing etcdContainerStuck path either, because each replacement resets the container's RestartCount to zero.

This threatens neither quorum nor data: the PVC and the member ID are preserved, so etcd sees a restart rather than a membership change. The cost is churn against the apiserver plus a member that never comes up — still much better than the permanent wedge #370 fixed, which is why this is a follow-up and not a revert.

Worth considering: stamp the last replacement time in EtcdMember.Status and require a minimum interval between replacements, or count consecutive terminal replacements and fall through to the existing member-replace path once the count says the Pod will never start.

Note that the quorum gate should stay off this path. Replacement is non-destructive for PVC-backed members, and gating it on quorum would make the exact incident #368 describes — every member terminal at once — unrecoverable, since quorum is gone precisely when every member needs replacing.

2. Backfill Status.PodUID before deleting a terminal Pod

The memory-backed branch of the fix depends on Status.PodUID being non-empty: the Pod-loss gate (member.Spec.Storage.Medium == StorageMediumMemory && member.Status.PodUID != "") is what turns the deleted Pod into a member replacement rather than a fresh Pod on an empty tmpfs under the old member identity. The terminal-Pod branch deliberately avoids ensurePodAbsent for exactly this reason, since that helper clears the UID.

updateStatus records PodUID on every pass in which the Pod object exists, regardless of readiness, so for the UID to still be empty the Pod would have to reach a terminal phase within roughly one reconcile of being created — meaning it never became an etcd member, so there is no identity to return under and no data to lose. The window is real but benign; it degenerates into item 1 rather than into data loss.

Closing it is a one-liner, and the branch already performs a Status().Update, so it costs nothing: set member.Status.PodUID = string(pod.UID) when it is empty, inside that same update. Worth doing so the memory path stops depending on when status was last persisted.

3. Missing Reconcile-level negative test for a Running Pod

TestPodInTerminalPhase covers the phase predicate, and the rest of the suite passing unchanged is indirect evidence that a live Pod is untouched, but there is no reconcile test that asserts it directly. TestReconcile_MemoryMemberStablePodIsNotLost is the closest fit and uses an empty-string phase rather than an explicit PodRunning, so it does not exercise the new branch against a genuinely running Pod.

A reconcile pass over a Running Pod asserting the Pod object survives with its original UID would pin the guarantee explicitly.