feat: add nodepool_provisioning_duration metric by Asrarfarooq · Pull Request #40 · nstogner/megamon

Asrarfarooq · 2026-05-04T18:23:24Z

This PR introduces a new metric, nodepool_provisioning_duration, to track and report the time taken for GKE NodePools to become ready after creation. This helps in monitoring provisioning latency, detecting scaling issues, and capturing provisioning failures.

Results

We verified the new metric on (megamon-test-cluster) by creating test TPU node pools and observing the output.

Raw Metrics Output (Ongoing State)

Curling the metrics endpoint (:8080/metrics) while the NodePool was still provisioning confirmed that the metric is emitted with the "provisioning" state:

# HELP megamon_alpha_nodepool_provisioning_duration_seconds Time spent provisioning.
# TYPE megamon_alpha_nodepool_provisioning_duration_seconds gauge
megamon_alpha_nodepool_provisioning_duration_seconds{nodepool_name="tpu-validation-pool-1763155065",otel_scope_name="megamon",otel_scope_version="",provisioning_state="provisioning",tpu_accelerator="tpu-v4-podslice",tpu_topology="2x2x1"} 319.972495761

Raw Metrics Output (Success State)

Once the NodePool entered the RUNNING status, the metric successfully transitioned to the "success" state with the final duration frozen:

# HELP megamon_alpha_nodepool_provisioning_duration_seconds Time spent provisioning.
# TYPE megamon_alpha_nodepool_provisioning_duration_seconds gauge
megamon_alpha_nodepool_provisioning_duration_seconds{nodepool_name="tpu-validation-pool-1763155065",otel_scope_name="megamon",otel_scope_version="",provisioning_state="success",tpu_accelerator="tpu-v4-podslice",tpu_topology="2x2x1"} 330.000998025

Controller Logs

The GCS-serialized event record confirmed the internal calculation of provisioningDuration (in nanoseconds) matching the external metrics:

{
  "type": "nodepools",
  "summary": {
    "downTimeProvisioned": 330000998025,
    "provisioningDuration": 330000998025,
    "provisioningState": "success"
  }
}

These results confirm that:

The metric is correctly registered and emitted.
It dynamically tracks provisioning states (provisioning, success, and failed).
It attaches the correct labels for topology and accelerator type.

Key Changes

internal/records/events.go: Added StateProvisioning, StateSuccess, and StateFailed constants to eliminate magic strings, and implemented the failed provisioning timeline calculation.
internal/records/report.go: Added the Failed flag to Upness to bridge data from the poller to events.
internal/aggregator/poller/node_poller.go: Mapped GKE "ERROR" and "RUNNING_WITH_ERROR" statuses to the internal Failed state.
internal/metrics/metrics.go: Registered and implemented the emission of the new nodepool_provisioning_duration metric.
internal/records/events_test.go: Added unit tests for ongoing and failed provisioning scenarios.
test/integration/cases_test.go: Added integration specs verifying both ongoing and failed provisioning scenarios, added DownTimeInitial assertions, and implemented label assertions for metrics.
docs/metrics.md: [NEW] Created metric documentation and examples.
docs/RELEASE_NOTES.md: Updated release notes for the upcoming v1.1.1 release.
go.mod: Resolved direct dependency alignment for sigs.k8s.io/lws.

Context

This branch has been rebased on top of the latest main (which includes the consolidated workload reconciler changes from PR #39) to ensure a clean and isolated diff containing only this feature. Conflict in events_test.go was resolved by combining assertions from both branches.

Updated internal/records/events_test.go to expect ProvisioningDuration: 1h and ProvisioningState: "success" for older test cases that calculate these values but previously defaulted to zero values in the test table.

echiugoog · 2026-06-17T15:20:07Z

Can you update the release notes and document the metric definition? perhaps start a docs/metrics.md and show what's expected for nodepools still being provisioned, failed and succeeded

echiugoog · 2026-06-17T15:22:23Z

Can you add a test case for the provisioning and failed case as well?

Added unit test cases for ongoing provisioning (StateProvisioning) and failed provisioning (StateFailed) inside internal/records/events_test.go.

echiugoog · 2026-06-17T15:23:07Z

 	})
+
+	// Added to test provisioning duration for resources that never become UP.
+	It("should correctly summarize a flow that never becomes UP", func() {


can you assert the expectation of downtime_initial as well?

Added the assertion for summary.DownTimeInitial to the integration test case.

echiugoog · 2026-06-17T15:23:18Z

Can we add in the failed provisioning case?

Added the new integration test case It("should correctly summarize a failed provisioning flow") to cover this.

echiugoog · 2026-06-17T16:13:34Z

 	if n == 1 {
 		summary.DownTime = now.Sub(r.UpEvents[0].Timestamp)
+		summary.ProvisioningDuration = summary.DownTime
+		summary.ProvisioningState = "provisioning"


lets use const for valid values

Defined StateProvisioning, StateSuccess, and StateFailed constants in internal/records/events.go and updated the code and tests to use them.

echiugoog · 2026-06-17T19:10:18Z

Let's also assert the metric is correctly emitted as well

Added the WithLabel helper to the test metric structure and implemented assertions to verify nodepool_provisioning_duration_seconds is correctly emitted with the right labels during integration runs.

echiugoog

Please also update RELEASE_NOTES.md

Asrarfarooq · 2026-06-26T18:22:43Z

Can you update the release notes and document the metric definition? perhaps start a docs/metrics.md and show what's expected for nodepools still being provisioned, failed and succeeded

Created docs/metrics.md containing the metric definitions, labels, and raw output examples for all three states (provisioning, success, and failed). I've also updated docs/RELEASE_NOTES.md.

Asrarfarooq · 2026-06-26T18:26:49Z

Please also update RELEASE_NOTES.md

Updated docs/RELEASE_NOTES.md with details for this feature under the version v1.1.1.

echiugoog · 2026-07-06T14:22:55Z

+
+## Nodepool Metrics
+
+### `megamon.nodepool.provisioning.duration`


maybe add a version introduced in

Added the metric introduction version (v1.1.1) to the documentation.

echiugoog · 2026-07-06T14:24:14Z

+## v1.1.1
+
+### New features and improvements
+* Add `nodepool.provisioning.duration` metric to track GKE NodePool provisioning latency with state tracking (`provisioning`, `success`, `failed`).


can you reference the docs/metrics.md file as well?

Added a reference link pointing to the docs/metrics.md file.

echiugoog · 2026-07-06T14:39:55Z

 				Attrs:        utils.ExtractNodePoolAttrs(np),
 				Status:       np.Status,
 				ExpectedDown: np.Status == "STOPPING" || np.Status == "DELETING",
+				Failed:       np.Status == "ERROR" || np.Status == "RUNNING_WITH_ERROR",


Let's introduce const for these strings

Defined constants for GKE status strings inside node_poller.go

echiugoog · 2026-07-06T14:41:12Z

 	UpEvents []UpEvent `json:"upEvents"`
 }

+const (


can we prefix this so we know it's for nodepool provisioning?

Prefix-renamed state constants

echiugoog · 2026-07-06T14:44:05Z

 				Attrs:        utils.ExtractNodePoolAttrs(np),
 				Status:       np.Status,
 				ExpectedDown: np.Status == "STOPPING" || np.Status == "DELETING",
+				Failed:       np.Status == "ERROR" || np.Status == "RUNNING_WITH_ERROR",


Let's update node_poller_test.go to cover the change as well

Expanded node_poller_test.go to cover these changes as well.

echiugoog · 2026-07-06T14:48:27Z

+
+	// ProvisioningDuration is the time spent provisioning.
+	ProvisioningDuration time.Duration `json:"provisioningDuration"`
+	// ProvisioningState is the state of provisioning (provisioning or success).


it's not just provisioning or success? perhaps omit this part of the comment for brevity

Omitted comment for brevity.

echiugoog · 2026-07-08T18:22:20Z

+
+## Nodepool Metrics
+
+### `megamon.nodepool.provisioning.duration`


echiugoog · 2026-07-08T18:26:33Z


 var log = logf.Log.WithName("upness")

+const (


can you use these in node_poller_test as well?

echiugoog · 2026-07-08T18:27:49Z

 	UpEvents []UpEvent `json:"upEvents"`
 }

+const (


Asrarfarooq · 2026-07-08T19:40:20Z

@nstogner could you please review this PR?

It adds the NodePool provisioning duration metric and refactors statuses and states into typed constants (per Edwin's feedback).

Asrarfarooq added 2 commits May 4, 2026 13:18

feat: add nodepool_provisioning_duration metric

0767d75

updated TestSummarize expectations for provisioning metric

bf9582a

Updated internal/records/events_test.go to expect ProvisioningDuration: 1h and ProvisioningState: "success" for older test cases that calculate these values but previously defaulted to zero values in the test table.

echiugoog suggested changes Jun 17, 2026

View reviewed changes

echiugoog suggested changes Jun 23, 2026

View reviewed changes

implement failed provisioning state and refactor constants

c1be901

Asrarfarooq requested a review from echiugoog June 26, 2026 18:35

echiugoog suggested changes Jul 6, 2026

View reviewed changes

replace strings with constants and rename provisioning state variables

be55466

Asrarfarooq requested a review from echiugoog July 7, 2026 17:10

echiugoog suggested changes Jul 8, 2026

View reviewed changes

use constants for node statuses

310b835


		## Nodepool Metrics

		### `megamon.nodepool.provisioning.duration`

Conversation

Asrarfarooq commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Results

Raw Metrics Output (Ongoing State)

Raw Metrics Output (Success State)

Controller Logs

Key Changes

Context

Uh oh!

echiugoog commented Jun 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Asrarfarooq Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

echiugoog left a comment

Choose a reason for hiding this comment

Uh oh!

Asrarfarooq commented Jun 26, 2026

Uh oh!

Asrarfarooq commented Jun 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Asrarfarooq commented Jul 8, 2026

Uh oh!

Reviewers

Asrarfarooq commented May 4, 2026 •

edited

Loading

Asrarfarooq Jun 23, 2026 •

edited

Loading