Adopt an existing cluster monitoring stack instead of installing into `ztest-obs`

[nachog00]

Summary

When targeting a cluster that already runs a monitoring stack (e.g. kube-prometheus-stack / Prometheus Operator), ztest has no way to use it. It either installs its own Prometheus + Pyroscope + Grafana into ztest-obs, or — with --no-observability — installs nothing and connects to nothing. There is no "adopt the existing stack" path, even though the discovery machinery for it already exists in the tree.

This makes ztest awkward to run on a shared/long-lived infra cluster: you get a second, redundant Prometheus/Grafana in ztest-obs, and the cluster's existing Grafana can't see ztest runs without extra out-of-band scrape config.

Current behavior (verified against main)

• ObservabilityProvider::probe only recognizes ztest's own named Deployments (ztest-prometheus/ztest-pyroscope/ztest-grafana) in ztest-obs; any other stack is Absent → it installs its own.

• Every reader hardwires the ztest-obs services by constant — e.g. src/metrics/query.rs:357,497,521,534 do Api::namespaced(client, naming::OBS_NAMESPACE).get(naming::PROMETHEUS_SERVICE) and reach it via port-forward. Same for the Pyroscope reader behind ztest sync perf. The names are compile-time constants (src/naming.rs:179-183); there is no URL/endpoint override.

• The --no-observability help text (cli/src/cluster/setup.rs:38-42) says:

  For a cluster whose operator owns these — configure the stack's endpoints on the cluster profile instead, and ztest uses those.

  This wiring does not exist. The Profile struct (src/cluster_config.rs:82-110) has fields only for context, kubeconfig, push, pull, storage_driver, runtime, class — no prometheus/pyroscope/grafana endpoint. A grep for PROMETHEUS_URL / PROMETHEUS_ENDPOINT / any external-obs override finds nothing. So --no-observability today means "install nothing and connect to nothing": ztest's components go unscraped, and ztest sync status / ztest sync perf lose their data source.

Net: metrics-server is already reuse-if-serving (MetricsApiProvider::probe, src/resource/impls/metrics_api.rs:327), and storage/CSI/snapshots are strictly bring-your-own (detected, never installed). The observability TSDB stack is the one component with no bring-your-own path.

Why it matters

Running ztest against an existing infra cluster (in our case k3s + kube-prometheus-stack + TopoLVM + metrics-server) works cleanly for storage and metrics-server, but observability forces a choice between (a) a redundant second stack in ztest-obs, or (b) no scraping / no sync perf at all. Neither lets the cluster's existing Grafana surface ztest runs.

Worth noting the component pods are already portable: ztest labels every scrape target ztest.io/component-name and names the port metrics, and zaino/zebra emit standard zaino_* / zebra* series. So a foreign Prometheus can scrape them today with a PodMonitor replicating ztest's keep-rules (SCRAPE_CONFIG in src/resource/impls/observability.rs). The missing half is on ztest's own readers, which can't be pointed anywhere but ztest-obs.

Proposal

Make the observability stack adoptable, in two parts:

1. Install side — adopt, don't duplicate. Give ObservabilityProvider::probe (and the MetricsApi sibling) an adopt-by-label path like the one that already exists for the read-only cluster check: src/capability.rs:273 probe_metrics already discovers a foreign prometheus/pyroscope/grafana by app.kubernetes.io/name (src/capability.rs:19,271-279) precisely so "an operator's own stack" is recognized. Reuse that discovery in the installer so a detected stack is treated as Ready and nothing is installed into ztest-obs.

2. Reader side — configurable endpoints. Add optional prometheus / pyroscope (and optionally grafana) endpoint fields to Profile, and have prometheus_backend / pyroscope_backend resolve those (or the adopted Service) instead of the hardcoded OBS_NAMESPACE + *_SERVICE constants. This is what the --no-observability help text already promises; the change makes the docs true.

Either part alone is useful; together they let ztest cluster setup no-op the TSDB on a cluster that already has one, and let sync status / sync perf read from it.

Non-goals / notes

• Not asking ztest to manage the foreign stack (no dashboards pushed, no retention config) — just discover it, skip installing, and read from it.
• RunIdentity and Buildkit stay ztest-owned; they're ztest-specific and don't conflict with shared infra.
• Scraping of ztest's component pods by a foreign Prometheus is already achievable operator-side (PodMonitor on ztest.io/component-name + port metrics); this request is specifically about ztest's own install/read coupling to ztest-obs.

Environment

Surfaced while evaluating ztest against an existing k3s GitOps cluster (kube-prometheus-stack via Prometheus Operator, TopoLVM CSI + VolumeSnapshots, metrics-server). Everything except observability slots onto the existing infra without changes.