Version-state-aware minReplicas for WRT autoscalers (floor while Current, zero once Drained)

[noamyehudai]

Problem

WorkerResourceTemplate renders an identical autoscaler for every version — the template cannot express anything conditioned on version state. With the KEDA ScaledObject flavor (v1.8.0, #351) this forces a lose-lose choice for minReplicaCount:

• minReplicaCount >= 1 — KEDA enforces the floor on every version, including Drained ones (KEDA applies minReplicaCount regardless of trigger activity). The controller force-zeroes drained versions at sunset ("Drained versions and inactive versions that are not the rollout target are always scaled to zero during sunset"), so keda-operator and the controller fight a ~30s scale-up/scale-down write war per drained version — observed live, same family as [Bug] HorizontalPodAutoscaler continuously rescales due to controller interference #350. Worse: getDeleteDeployments requires replicas == 0 before deleting, so a floored autoscaler blocks version Deployment deletion indefinitely.

• minReplicaCount: 0 — the war disappears, but the Current version scales to zero too. The scaler's metric is ApproximateBacklogCount only; a keeping-up worker reads backlog ≈ 0 at nearly every 30s poll, so one cooldownPeriod of samples that miss the bursts parks the Current version, and the next task pays a ~2min cold start (poll interval + pod boot). Measured on our production queues over 14 days: a bursty queue had 70+ windows of ≥60 min with zero visible backlog (parks ~5×/day at the 3600s cooldown we run); even a queue with ~1.5k tasks/hour parked several times a day purely via sampling-miss. Related failure mode: a version deployed while traffic is idle is held at 0 pollers and can fail promotion ([Bug] New version can get stuck INACTIVE and never promote when a scale-to-zero autoscaler holds its Deployment at 0 replicas #438).

The semantic actually needed is state-aware:

minReplicas: N while the version is Current (or Ramping); 0 once it is Draining/Drained.

Verified nothing supports this today: v1.8.0 RenderWorkerResourceTemplate passes only (wrt, deployment, buildID, temporalNamespace) — no version state — and the KEDA temporal scaler (pkg/scalers/temporal_scaler.go, v2.20.1 == main) contains zero references to version status/drainage.

The data already exists in the scaler's response

In deployment-version mode the KEDA scaler calls DescribeWorkerDeploymentVersion with ReportTaskQueueStats: true and reads only version_task_queues. The same response carries worker_deployment_version_info — including status (CURRENT / RAMPING / DRAINING / DRAINED), ramp and drainage info — which is currently discarded.

Options

A. KEDA scaler (smallest change, covers floor = 1): an opt-in trigger param (e.g. activateOnCurrentVersion: true) that reports the trigger active whenever the version status is CURRENT/RAMPING, regardless of backlog. With minReplicaCount: 0, activation + the HPA's implicit minReplicas: 1 keep the Current version warm; Draining/Drained versions activate only on real backlog and rest at 0, so the controller's sunset is unopposed. Uses data the scaler already fetches. (Happy to file this on kedacore/keda — flagging here first since this repo owns the versioning semantics; note KEDA declined an adjacent ask in kedacore/keda#7368.)

B. Controller (general, covers floor = N): per-version-state rendering/patching — e.g. a currentVersionMinReplicas field or per-state template overrides, where the controller sets minReplicaCount on the Current version's ScaledObject and removes it on the transition to Draining. This is essentially #104 ("Per-Version-Status min/max Replicas" — closed, but the knob never shipped) and fits the requirements list in #14 ("Scale independently per-version", "Scale to 0 after drain", "Preemptively scale up target version before cutting over"); the same mechanism applied to the rollout target version would also address #438.

C. Composite activity metrics (kedacore/keda#7459) or per-version metric-label templating (#355) mitigate the premature-park half but not the drained-floor half.

Workaround we run meanwhile

minReplicaCount: 0 + a very long cooldownPeriod (86400): the Current version effectively never parks (any active sample resets the clock), and Drained versions are unaffected because the controller zeroes them at scaledownDelay and KEDA never scales up without activation — cooldownPeriod gates only KEDA's own scale-down. Verified live: with scaledownDelay (15m) far below the cooldown, rollouts zero freshly-drained versions with unexpired cooldowns and no flap-back occurs. Cost: Draining versions holding open pinned workflows keep one warm pod up to 24h after their last straggler task.

Related: #14, #104, #210, #332, #350, #355, #438; kedacore/keda#7368, kedacore/keda#7459.

[noamyehudai]

Update — we've validated a better workaround than the long-cooldown TTL described above, worth recording for anyone hitting this:

minReplicaCount: 1 + a short deleteDelay, letting version deletion (not scale-to-zero) be the terminal state. The insight: drained versions don't need to rest at 0 — they need to be gone. Observed end-to-end on a production-shaped staging cluster (scaledownDelay 5m / deleteDelay 30m, watcher at 45s resolution):

• rollout: new versions got their pod immediately (min floor) → promotion never deadlocks (also mitigates [Bug] New version can get stuck INACTIVE and never promote when a scale-to-zero autoscaler holds its Deployment at 0 replicas #438)
• drained → held 1 warm pod for exactly scaledownDelay (a nice side effect: this retention window is warm, so rollback during it is instant and closed-workflow queries still work)
• then the anticipated controller-vs-KEDA tension on the zeroed Deployment turned out to be much gentler than modeled: KEDA's floor re-enforcement on an already-zeroed target runs on a lazy resync cadence, not the 30s trigger poll — we saw 2–3 raises in the whole 30-minute window, each reverted by the controller in ~200ms, zero errors in either operator
• at drained + scaledownDelay + deleteDelay the gate in getDeleteDeployments passed on the first reconcile (replicas sit at 0 essentially always during the window) and the Deployment + rendered ScaledObject were deleted on schedule to the second, ending the tension permanently — no orphaned ScaledObjects (getDeleteWorkerResources handled cleanup correctly)

So in practice the "indefinite write-war / deletion blocked by the floor" worst case doesn't materialize: the controller wins the race decisively and deletion lands on time. The residual costs are one warm pod per DRAINING version (arguably a feature — stragglers keep a poller) and a handful of flap events per retired version.

A first-class state-aware floor would still be cleaner (no flap events, no reliance on the controller winning a race that isn't contractual), but min-1 + short deleteDelay is a solid bridge.

[noamyehudai]

Correction to my previous comment — the "lazy resync cadence, 2–3 raises in 30 minutes" claim was wrong. It was sourced from kubectl get events, which aggregates identical messages and spam-filters after a ~25-event burst, so it undercounts high-frequency occurrences by ~25x.

Ground truth from the cluster audit log for the same 30-minute window (two drained versions): keda-operator issued 120 deployments/scale update raises (~one per 30s poll per version), the controller issued 122 re-zeroes, and each flap created, scheduled, and deleted a real Pod (120 pod creations, 120 scheduler bindings, ~240 pod deletions). So the tension does run at the full pollingInterval cadence, exactly as the code reads — each raise reverted in ~200ms, zero errors in either operator, and deletion still landed on schedule at drained + scaledownDelay + deleteDelay, ending it.

Practical amendment to the workaround: with minReplicaCount: 1, keep deleteDelay short — the flap window burns ~2 wasted pod lifecycle cycles per minute per retired version for its whole duration, and retention during it is illusory anyway (replicas are forced ~0 throughout, so closed-workflow queries can't be served; only rollback-by-re-promote survives). We run 30m.

This strengthens rather than weakens the case for a first-class state-aware floor: the bridge works, but it costs ~60 pod create/bind/delete cycles per retired version even at a 30m window.