[Bug] New version can get stuck INACTIVE and never promote when a scale-to-zero autoscaler holds its Deployment at 0 replicas

[noamyehudai]

Summary

With AllAtOnce rollout and a backlog-based scale-to-zero autoscaler on the versioned worker Deployments (e.g. KEDA's Temporal scaler keyed per workerDeploymentBuildId, minReplicaCount: 0), a newly rolled-out version can deadlock and never become Current: the autoscaler holds its Deployment at 0 replicas, so it has no pollers; the controller then refuses to promote it; the next deploy supersedes it; and it is left permanently as an INACTIVE / never-Current version. It never self-recovers.

Environment

• temporal-worker-controller v1.8.0 (Helm chart 0.27.0), controller-runtime v0.24.0
• Temporal Server v1.30.x, CLI v1.6.x
• Rollout strategy: AllAtOnce
• Per-version pod autoscaling via a KEDA ScaledObject (temporal scaler), minReplicaCount: 0, trigger keyed to the version's own workerDeploymentBuildId backlog

The deadlock

1. A new build lands → the controller creates the versioned Deployment.
2. The autoscaler scales that Deployment on the backlog for that build ID. A version that is not yet Current has no workflows pinned to its build ID → its build-id backlog is 0 → the autoscaler scales it to 0 replicas → 0 pollers.
3. The controller (AllAtOnce) attempts to set it Current and fails:

   unable to set current deployment version: proposed current version
   'default/example-worker:1.4.0-a1b2' is missing active task queues from the
   current version; these would become unversioned if it is set as the current version
   

   A version with 0 pollers polls no task queues, so it is "missing the active task queues from the current version." (This guard is correct — promoting a poller-less version would orphan those task queues as unversioned.)
4. Deadlock: no pollers → can't be promoted; not Current → no backlog → the autoscaler grants no pollers.
5. The next deploy retargets the controller to a newer build. The stranded version stays never-Current → INACTIVE forever — nothing ever routes to it, so it never gets pollers and never recovers.

Trigger condition

Deploying while the task queue is idle (no backlog to activate the per-build-id scaler). With steady traffic at deploy time — or a minReplicas ≥ 1 autoscaler — the new version comes up with pollers and promotes normally, so this does not reproduce. It's intermittent and correlates strongly with idle-time deploys (in one environment we see a long tail of consecutive never-Current INACTIVE versions from quiet periods, and none from busy periods).

Impact

• Deploys made during idle windows silently never go live — the new build cannot become Current, so the previous version keeps serving indefinitely.
• Stranded INACTIVE versions accumulate toward matching.maxVersionsInDeployment, adding to the version-record cleanup problem (cf. Controller never deletes Temporal-side Worker Deployment version records during normal rollout (only on CRD deletion) #377).

Reproduction

1. Configure a worker with AllAtOnce rollout and a per-version KEDA ScaledObject (temporal scaler, minReplicaCount: 0, trigger keyed to the version's workerDeploymentBuildId).
2. With no traffic on the task queue, roll out a new build.
3. Observe: the new version's Deployment is scaled to 0; the controller logs the "missing active task queues" error on the set-current attempt; the version never becomes Current.
4. Roll out another build → the previous one is left INACTIVE permanently.

Expected behavior

During a rollout the controller should ensure the target version has at least one active poller before/while calling SetWorkerDeploymentCurrentVersion — e.g. warm a poller for the target version (ensure ≥1 ready replica, or wait for readiness) prior to set-current, then hand scaling back to the autoscaler. That keeps scale-to-zero autoscaling compatible with rollout, instead of forcing users onto a permanent minReplicas ≥ 1 (which loses scale-to-zero and keeps drained versions from scaling down during sunset — see also #350).

Workarounds

• Use a minReplicas ≥ 1 autoscaler (e.g. a plain HPA). Removes the deadlock, but gives up scale-to-zero and holds non-current/drained versions at ≥1 replica until the sunset delete.
• Ensure task-queue traffic at deploy time.

Related

• Optimization: Requeue sooner when new deployment version has just started polling but isn’t “active” yet #30 — requeue sooner when a new version has just started polling but isn't "active" yet (adjacent readiness/timing concern).
• Revive retained zero-replica Deployments when a drained version becomes draining again #332 — revive retained zero-replica Deployments when a drained version becomes draining again (related zero-replica lifecycle).
• [Bug] HorizontalPodAutoscaler continuously rescales due to controller interference #350 — HPA continuously rescales due to controller interference (relevant to the minReplicas ≥ 1 workaround).
• [Feature Request] Support per-version metrics label templating to close loop on KEDA per-version scaling #355 / Horizontally autoscale worker replicas per-version #14 / Automatically scale up worker versions with reachability ClosedOnly on query #15 — per-version KEDA autoscaling (the context this arises in).
• Controller never deletes Temporal-side Worker Deployment version records during normal rollout (only on CRD deletion) #377 — version records accumulate during normal rollout (the cleanup problem stranded INACTIVE versions add to).