Skip to content

Surface Temporal-native poller health as Conditions + Events (extends #50) #447

Description

@wankhede04

Is your feature request related to a problem? Please describe.

Investigating #50 (surface "Unhealthy" version after rollout acceptance test / worker healthcheck fails), I found that today's per-version HealthySince field (api/v1alpha1/workerdeployment_types.go) is computed only from Kubernetes Deployment readiness — internal/controller/state_mapper.go calls k8s.IsDeploymentHealthy(deployment), which checks pod/replica status. It has no visibility into whether Temporal is actually receiving polls from those pods. A Deployment can look perfectly healthy to Kubernetes (all replicas Ready) while its workers are misconfigured, stuck, or unable to reach Temporal — and today the controller has no way to surface that.

Separately, the controller already fetches real poller data from Temporal: internal/temporal/worker_deployment.go's getPollers() calls client.DescribeTaskQueue(...) and gets back the actual poller list per task queue. Today this is used only to compute one internal boolean (AllTaskQueuesHaveUnversionedPoller, a versioning-migration heuristic) — the poller data itself is discarded, never surfaced to the user.

@carlydf's comment on #50 confirms this is in scope now (not gated on the post-GA rollout-acceptance-testing work): "We should surface worker healthcheck error and Gate workflow failure in the TemporalWorkerDeployment events for sure."

Describe the solution you'd like

Extend VersionInfo with a poller-health summary derived from the poller data already fetched in getPollers() (no new Temporal API calls needed for the base case).
Add a new condition type on WorkerDeploymentStatus (following the existing Reason* constant pattern already in workerdeployment_types.go), e.g. ConditionWorkersHealthy, with reasons:
ReasonPollersHealthy — all expected task queues have active pollers
ReasonNoActivePollers — a version has zero pollers on one or more task queues
ReasonPollerStatusUnknown — couldn't determine (e.g. transient API error) — should not regress to "unhealthy" on a fetch failure
Emit K8s Events on health-state transitions (not every reconcile, to avoid event spam) using the existing recorder pattern (r.Recorder.Eventf(workerDeploy, corev1.EventTypeWarning/Normal, reason, "%s", message), already used in worker_controller.go/execplan.go):
Warning when a version loses all pollers on a task queue
Normal when a version's pollers become healthy again
Gate workflow failure surfacing (explicitly called out by @carlydf above): the controller already tracks TestWorkflows []WorkflowExecution with status including Failed/Canceled/Terminated/TimedOut. Emit a Warning event referencing the workflow ID when a test/gate workflow transitions into one of these terminal-failure states.

This is purely additive — new condition type, new event emissions, one new struct field — with no change to rollout/reconciliation decision logic (internal/planner/planner.go). Health reporting, not health gating; the post-GA rollout-acceptance-testing work in #50 is a separate, later step that could consume this data but isn't required for it.

Describe alternatives you've considered

Waiting for the post-GA rollout-acceptance-testing feature in #50 to land first — but @carlydf's comment explicitly separates worker-healthcheck/Gate-failure surfacing from that post-GA work, so this can land independently now.
Relying only on the existing k8s-native HealthySince — insufficient, since it can't detect a worker that's running but not actually polling Temporal.

Additional context

Also sets up naturally for #61 (add URL field to version metadata) — once real poller/Gate-failure diagnostics exist, encoding workflow URLs to link directly to the failing execution becomes a much smaller follow-up. Not attempting #61's CRD/metadata change in this issue, since it's separately milestoned Post-GA.

Happy to contribute a PR for this if the team is open to the approach.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions