Skip to content

fix: handled MetricAgent plugin for multi matrix#4338

Open
Hardikl wants to merge 8 commits into
mainfrom
hl_sg
Open

fix: handled MetricAgent plugin for multi matrix#4338
Hardikl wants to merge 8 commits into
mainfrom
hl_sg

Conversation

@Hardikl

@Hardikl Hardikl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor
image

Added supporting test cases as well.

Copilot AI review requested due to automatic review settings July 2, 2026 10:21
@cla-bot cla-bot Bot added the cla-signed label Jul 2, 2026
@Hardikl

Hardikl commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Validated with below template changes,

name:   Prometheus
query:  prometheus
object: # leave blank to disable prefixing exported metrics with object name

schedule:
  - data: 3m

counters:
  - node_cpu_seconds_total                                                => storagegrid_node_cpu_seconds_total
  - storagegrid_content_buckets_and_containers
  - storagegrid_content_objects
  - storagegrid_ilm_awaiting_client_objects
  - storagegrid_ilm_awaiting_total_objects
  - storagegrid_ilm_objects_processed
  - storagegrid_ilm_scan_objects_per_second
  - storagegrid_metadata_queries_average_latency_milliseconds
  - storagegrid_network_received_bytes
  - storagegrid_network_transmitted_bytes
  - storagegrid_node_cpu_utilization_percentage
  - storagegrid_private_ilm_awaiting_delete_objects
  - storagegrid_private_load_balancer_storage_request_body_bytes_bucket
  - storagegrid_private_load_balancer_storage_request_count
  - storagegrid_private_load_balancer_storage_request_time
  - storagegrid_private_load_balancer_storage_rx_bytes
  - storagegrid_private_load_balancer_storage_tx_bytes
  - storagegrid_private_s3_total_requests
  - storagegrid_s3_data_transfers_bytes_ingested
  - storagegrid_s3_data_transfers_bytes_retrieved
  - storagegrid_s3_operations_failed
  - storagegrid_s3_operations_successful
  - storagegrid_s3_operations_unauthorized
  - storagegrid_storage_utilization_data_bytes
  - storagegrid_storage_utilization_metadata_allowed_bytes
  - storagegrid_storage_utilization_metadata_bytes
  - storagegrid_storage_utilization_total_space_bytes
  - storagegrid_storage_utilization_usable_space_bytes

plugins:
  - JoinRest
  - MetricAgent:
      compute_metric:
        - storagegrid_storage_utilization_used_percent PERCENT storagegrid_storage_utilization_data_bytes storagegrid_storage_utilization_total_space_bytes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the MetricAgent plugin so compute_metric rules can operate when the required operands are split across multiple matrices (instead of assuming a single object-keyed matrix), and adjusts/adds unit tests to cover the multi-matrix scenario.

Changes:

  • Changed MetricAgent.actions to accept the full dataMap (map[string]*matrix.Matrix) rather than a single *matrix.Matrix.
  • Updated computeMetrics to select operand matrices per rule/operand (enabling multi-matrix support).
  • Updated the single-matrix test to run through Run(), and added a new multi-matrix test case.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
cmd/poller/plugin/metricagent/parse_rules.go Updates action function signature to accept the full matrix map.
cmd/poller/plugin/metricagent/metric_agent.go Passes dataMap through actions and updates compute logic for multi-matrix operands.
cmd/poller/plugin/metricagent/metric_agent_test.go Renames/updates existing test to call Run() and adds a multi-matrix test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Copilot AI review requested due to automatic review settings July 2, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
metricVal = a.getMetric(m, r.metricNames[i])
if metricVal != nil {
v, _ = metricVal.GetValueFloat64(instance)
if m[i] != nil && m[i].GetInstance(iKey) != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SG metrics seems to have an index https://github.com/NetApp/harvest/blob/main/cmd/collectors/storagegrid/storagegrid.go#L219

Would getting by name work?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, validated locally and it's able to apply the given operator on the same instance of both metrics.

mat.NewInstance(metricName + "-" + strconv.Itoa(i)); Yes, it has index in instanceKey generation, but we use that instance key as-is for all m arrays.

@Hardikl

Hardikl commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Validated with this plugin changes,

plugins:
  - JoinRest
  - MetricAgent:
      compute_metric:
        - storagegrid_storage_utilization_used_percent1 PERCENT storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests
        - storagegrid_storage_utilization_used_percent2 PERCENT storagegrid_storage_utilization_data_bytes storagegrid_storage_utilization_total_space_bytes

Logs:

harvest % ./bin/poller -p sg-tudor --promPort 13004 -o Prometheus 
time=2026-07-09T14:47:32.646+05:30 level=INFO source=poller.go:244 msg=Init Poller=sg-tudor logLevel=INFO configPath=harvest.yml cwd=/Users/hardikl/Library/CloudStorage/OneDrive-NetAppInc/Documents/Harvest/harvest version="harvest version 26.07.09-1 (commit 3ab04ff6) (build date 2026-07-09T14:44:35+0530) darwin/amd64" fips=false options="&{Poller:sg-tudor Daemon:false Debug:false PromPort:13004 Config:harvest.yml HomePath: LogPath:/var/log/harvest/ LogFormat:plain LogLevel:2 LogToFile:false Version:26.07.09 Hostname:hardikl-mac-1 Collectors:[] Objects:[Prometheus] Profiling:0 Asup:false IsTest:false ConfPath:conf ConfPaths:[conf]}"
time=2026-07-09T14:47:32.647+05:30 level=INFO source=poller.go:283 msg="started in foreground" Poller=sg-tudor pid=23165
time=2026-07-09T14:47:35.049+05:30 level=INFO source=poller.go:1944 msg="Cluster info" Poller=sg-tudor remote="{Name:SG5712FP Model: UUID:4cca45b6-1e5a-4c15-b3a6-e29598de9d44 Version:12.0.0 Release: Serial: IsSanOptimized:false IsDisaggregated:false ZAPIsExist:false ZAPIsChecked:false HasREST:false HasRESTPerf:false IsClustered:false}"
time=2026-07-09T14:47:37.812+05:30 level=INFO source=helpers.go:113 msg="best-fit template" Poller=sg-tudor collector=StorageGrid:Prometheus path=conf/storagegrid/11.6.0/storagegrid_metrics.yaml v=12.0.0 jitter=none
time=2026-07-09T14:47:40.244+05:30 level=INFO source=poller.go:420 msg="Autosupport scheduled" Poller=sg-tudor asupSchedule=24h
time=2026-07-09T14:47:40.244+05:30 level=INFO source=poller.go:430 msg="poller start-up complete" Poller=sg-tudor
time=2026-07-09T14:47:40.246+05:30 level=INFO source=httpd.go:51 msg="server listen" Poller=sg-tudor exporter=prometheus1 url=http://:13004/metrics
time=2026-07-09T14:47:40.281+05:30 level=INFO source=poller.go:661 msg="updated status" Poller=sg-tudor collectors.up=1 collectors.total=1 exporters.up=1 exporters.total=1
time=2026-07-09T14:47:40.294+05:30 level=INFO source=poller.go:1757 msg=Metadata Poller=sg-tudor remote.name=SG5712FP remote.version=12.0.0 version="harvest version 26.07.09-1 (commit 3ab04ff6) (build date 2026-07-09T14:44:35+0530) darwin/amd64" mem.liveHeapMB=1 mem.heapMB=1 mem.heapGoalMB=4 mem.rssMB=19 mem.maxRssMB=19 uptimeSeconds=7
time=2026-07-09T14:47:42.323+05:30 level=WARN source=storagegrid.go:169 msg="no instances on storagegrid" Poller=sg-tudor collector=StorageGrid:Prometheus metric=storagegrid_ilm_awaiting_client_objects
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.619+05:30 level=WARN source=metric_agent.go:136 msg="computeMetrics: skipping compute metric calculation as instance labels are not matching for given metrics" Poller=sg-tudor plugin=StorageGrid:MetricAgent object=Prometheus metric=storagegrid_storage_utilization_used_percent1 "given metrics"="[storagegrid_storage_utilization_data_bytes storagegrid_private_s3_total_requests]"
time=2026-07-09T14:47:50.628+05:30 level=INFO source=collector.go:629 msg=Collected Poller=sg-tudor collector=StorageGrid:Prometheus apiMs=8954 bytesRx=300814 calcMs=0 exportMs=8 instances=1069 instancesExported=1069 metrics=1069 metricsExported=1072 numCalls=33 parseMs=0 pluginInstances=0 pluginMs=1418 pollMs=10373 renderedBytes=270358 zBegin=1783588660247
image

Copilot AI review requested due to automatic review settings July 9, 2026 09:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Comment thread cmd/poller/plugin/metricagent/metric_agent.go Outdated
Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Copilot AI review requested due to automatic review settings July 9, 2026 10:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:155

  • If an operand metric isn't found, this appends err (typically nil from the preceding Atoi failure) and breaks out without setting skipMetric, so the derived metric can be written with a partial/incorrect result and the missing-metric details are lost. Append a concrete error and mark skipMetric before breaking.
					} else {
						metricNotFound = append(metricNotFound, err)
						break
					}

Copilot AI review requested due to automatic review settings July 9, 2026 10:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comment thread cmd/poller/plugin/metricagent/metric_agent.go
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
Comment thread cmd/poller/plugin/metricagent/metric_agent_test.go
Copilot AI review requested due to automatic review settings July 9, 2026 13:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cmd/poller/plugin/metricagent/metric_agent.go:155

  • When an operand metric is missing, this appends err to metricNotFound, but err here is the strconv.Atoi parse error for the metric name (or nil) rather than a "missing metric" error. Record a concrete error tied to the missing metric so the warning at the end is actionable.
					} else {
						metricNotFound = append(metricNotFound, err)
						break
					}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MetricAgent crashes with nil pointer dereference on StorageGRID Prometheus collector

3 participants