monitor-exporter: native Prometheus /metrics for SMC/power (Swift, reuses MonitorSources)

[evandhoffman]

Goal

Add a monitor-exporter executable that serves the SMC and power sensors — and
the rest of what MonitorSources already reads — as Prometheus metrics, so an
M5 Max gets historic retention, Grafana and alerting.

This is the native-Swift answer to
evanwtf/local-llm#361, which
debated two ways to get these sensors into Prometheus: wrap macmon (Python
textfile job or Go), or reimplement Apple's IOReport/IOKit in Go. Both exist
only to obtain sensor readers. This repo already has them —
MonitorSources reads CPU, memory, disk, network, GPU and the SMC through
native mach/IOKit/sysctl, and the readers are tested and shipping. So the
cheapest durable exporter reuses that code here, in the language the repo is
already written in. No macmon dependency, no CGo against undocumented Apple
APIs, and no Go added to a Python-preferred tree.

Why a fourth binary, not a flag on monitord

monitord writes rotating CSV on a fixed clock; the exporter answers a scrape
at scrape time. Different clock, different consumer, different failure mode.
Keeping them separate holds the same boundary the package already enforces
between monitor (no disk) and monitord (the disk logger): one binary, one
job. The exporter links MonitorSources and MonitorCore, never MonitorLog
or MonitorStore.

Approach — own port (recommended)

Serve GET /metrics over HTTP and let Prometheus scrape it directly, the way
node_exporter itself works. The daemon reads the sensors at scrape time, so
the scrape interval is the sampling cadence — no timer to configure.

Why this over the textfile-collector relay:

• A dead exporter fails the scrape, so Prometheus surfaces it as up == 0.
  The textfile path serves the last temperature forever and looks healthy until
  a separate mtime-staleness alert fires.
• No launchd timer, no atomic temp-file + os.replace dance, no per-file
  staleness alert to maintain. #361's whole correctness section (skip-absent-
  keys, one # HELP/# TYPE per metric, atomic write) exists only because of
  the textfile format. Own-port deletes that class of bug.
• One long-running process instead of a spawn per interval.

Textfile fallback (documented, not built first)

node_exporter is not on the Mac yet but is coming. Its textfile collector is the
one reason to keep the relay in mind: if inbound scrape of a new port is ever
awkward, monitor-exporter --textfile <dir> could write an atomic .prom into
node_exporter's directory instead of listening. Same rendering code, different
sink. Build the listener first; add the textfile sink only if reachability
forces it.

Metrics

The real monitord CSV columns are the source of truth for the metric set (see
the sample-data comment on this issue), not the guesses in #361.

Scope: the macOS gap by default, --all to mirror the rest. node_exporter
is coming to this Mac and its darwin collectors already read CPU, memory, disk
and network — the first ~28 CSV columns. What node_exporter does not read on
macOS is the SMC (temperature, fans, power) and the GPU. So the default export
is that gap; a running node_exporter covers the rest, and no series is scraped
twice under two names. --all also exposes CPU/memory/disk/network for a host
with no node_exporter — deferred, and its disk/network series must be
cumulative counters (_total), not monitord's pre-differenced rates, so
rate() works.

Default (gap) set — all gauges, all instantaneous:

monitord column	Prometheus metric
sensor.temperature.{cpu,gpu,storage,battery,enclosure} °C	macos_smc_temperature_celsius{sensor="…"}
sensor.fan.{1,2}.speed rpm	`macos_smc_fan_rpm{fan="1
sensor.power.{input,soc} W	`macos_smc_power_watts{rail="input
gpu.utilization fraction	macos_gpu_utilization_ratio (0–1)
gpu.vram.used B	macos_gpu_vram_used_bytes

Plus monitor_exporter_build_info{commit,version} 1 from the existing
StampCommit stamp, so a scrape says which build answered.

Two rules the CSV forces:

• One canonical unit. The CSV carries every temperature in both °C and °F
  because a flat file must stand alone; the exporter emits _celsius only and
  Grafana converts. Exporting °F too is a redundant series.
• No hostname label. The scrape target's instance already identifies
  the Mac; a hostname label duplicates it and fights federation.

Dependencies

None new. The repo keeps its single Apple dependency
(swift-argument-parser), and this ships against MonitorCore,
MonitorSources and Network.framework only.

• Exposition format: rendered in-house in a MonitorPrometheus target
  (pure MonitorCore). The default set is entirely simple gauges, and the
  gauge text format is small and frozen — # HELP / # TYPE … gauge / a value
  line — so a golden-file test pins it exactly. A metric library
  (swift-server/swift-prometheus) was considered and set aside: its API
  shifts across versions and it pulls swift-nio transitively, which is a lot of
  surface for a handful of gauges. Reach for it only if --all later adds
  counters and histograms. (The monitord: --help, --version, and unknown flags all start the daemon instead of printing usage #48 lesson is about parsing input, where
  ambiguity bites; this is rendering fixed output, which a golden test nails.)
• HTTP: Network.framework NWListener, no dependency. One read-only
  GET /metrics route, so a thin Apple-native listener is enough. Hummingbird
  is the fallback only if hand-rolled HTTP/1.1 response framing proves fiddly.

Correctness and testing

• The MonitorPrometheus renderer is pure and lives on MonitorCore only: a
  Sample plus its MetricDescriptor in, an exposition line out. Golden-file
  test for the format, and a mapping test that each sensor MetricID produces
  the right metric name, labels and unit — both with fake readings, no hardware.
• The HTTP handler is tested directly: GET /metrics returns 200 with the
  Prometheus content type and the rendered body; any other path returns 404.
• A failed reading is a gap, never a zero — the same rule the app follows.
  Skip the sample and let the scrape omit that series, so an idle machine and a
  broken sensor do not read alike.
• CLI parsing (--bind-port, --bind-address, later --textfile) via
  swift-argument-parser with a CommandLineTests case, matching how
  monitord/monitorctl are covered — monitord: --help, --version, and unknown flags all start the daemon instead of printing usage #48's front-door bug was invisible to
  every other suite. Default port 9650: verified unallocated in the
  Prometheus default-port-allocations registry and clear of the dense exporter
  band at 9100–9130 (node=9100, blackbox=9110). It is a local default, not a
  public registration — the registry asks projects not to squat FREE slots — so
  it stays overridable. Default bind address is loopback; --bind-address 0.0.0.0 opens it to the LAN for a remote Prometheus.
• A --help smoke run in CI, like the other binaries.

Deployment

• Ships in the release zip as a bare binary, next to monitord.
• Runs as a launchd LaunchAgent.
• Prometheus adds a scrape target for the Mac's port; the scrape interval is the
  sample rate. node_exporter arriving on the Mac is independent — the exporter
  does not need it in own-port mode.

Constraint

Do not run the exporter during a live thermal benchmark — it adds a process and
load to the exact thing under measurement (carried over from #361). It touches
no git state, so it will not void a run through harness_dirty.

Resolved

1. Scope: the macOS gap (SMC + GPU) by default; --all mirrors CPU/memory/
   disk/network (deferred; disk/network as _total counters).
2. Namespace: macos_smc_* for the sensors, macos_gpu_* for the GPU, per
   the table above. No node_* reuse — node_exporter owns those.
3. Dependency + HTTP: no new dependency — exposition rendered in-house in a
   MonitorPrometheus target, NWListener for the single route.
4. Port + bind: --bind-port default 9650 (verified free in the Prometheus
   default-port-allocations registry), --bind-address default loopback,
   0.0.0.0 for a remote Prometheus. Auth is out of scope for a read-only
   endpoint on a trusted network.

Open questions

1. Target and library names: monitor-exporter binary, and does the mapping
   earn its own MonitorPrometheus library target or fold into an existing one?
2. Whether --all ships in the first cut or waits until a host without
   node_exporter actually needs it (leaning: wait — YAGNI).

--opus

[evandhoffman]

Sample values already collected by monitord

~/Library/Logs/monitor ❯ head sensors.macbook-pro_local.2026_09_13_08_20_43.csv
hostname,time_iso8601,time_epoch_ms,cpu.total (fraction),cpu.user (fraction),cpu.system (fraction),cpu.perflevel.0 (fraction),cpu.perflevel.1 (fraction),memory.used (B),memory.app (B),memory.wired (B),memory.compressed (B),memory.cached (B),memory.free (B),memory.swap.used (B),memory.pagein.rate (op/s),memory.pageout.rate (op/s),disk.bytes.read (B/s),disk.bytes.written (B/s),disk.ops.read (op/s),disk.ops.write (op/s),disk.latency.read (s),disk.latency.write (s),net.bits.in (bit/s),net.bits.out (bit/s),net.packets.in (op/s),net.packets.out (op/s),gpu.utilization (fraction),gpu.vram.used (B),sensor.temperature.cpu (°C),sensor.temperature.cpu (°F),sensor.temperature.gpu (°C),sensor.temperature.gpu (°F),sensor.temperature.storage (°C),sensor.temperature.storage (°F),sensor.temperature.battery (°C),sensor.temperature.battery (°F),sensor.temperature.enclosure (°C),sensor.temperature.enclosure (°F),sensor.power.input (W),sensor.power.soc (W),sensor.fan.1.speed (rpm),sensor.fan.2.speed (rpm)
macbook-pro.local,2026-09-13T12:20:43Z,1789302043431,,,,,,102253346816,13017874432,86832807936,2402664448,31729745920,2107555840,170655744,,,,,,,,,,,,,0.99,82528321536,72.33,162.19,76.78,170.21,29.54,85.16,30.30,86.54,30.31,86.56,115.31,86.69,3187,3434
macbook-pro.local,2026-09-13T12:20:44Z,1789302044457,0.10,0.05,0.05,0.20,0.06,98926133248,13282803712,83240665088,2402664448,32191741952,4970872832,170655744,0.00,0.00,23950.02,8833566.18,5.85,698.74,0.00,0.00,7983.34,0.00,25.34,2.92,0.99,78557085696,72.45,162.42,79.56,175.21,29.34,84.81,30.30,86.54,30.31,86.56,118.11,93.62,3185,3426
macbook-pro.local,2026-09-13T12:20:45Z,1789302045530,0.09,0.04,0.05,0.19,0.04,101408702464,13246824448,85759213568,2402664448,32091226112,2592964608,170655744,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,15270.69,7635.34,18.64,1.86,0.99,81375559680,73.55,164.38,82.95,181.32,29.53,85.15,30.30,86.54,30.31,86.56,118.94,102.14,3178,3435
macbook-pro.local,2026-09-13T12:20:46Z,1789302046572,0.09,0.04,0.04,0.18,0.04,101544689664,13246660608,85895364608,2402664448,32057917440,2490744832,170655744,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,23597.48,7865.83,22.08,4.80,0.99,81511579648,74.09,165.37,84.53,184.16,29.49,85.08,30.30,86.54,30.31,86.56,125.07,110.93,3181,3431
macbook-pro.local,2026-09-13T12:20:47Z,1789302047653,0.11,0.06,0.05,0.22,0.06,102114869248,12973473792,86738763776,2402631680,31805489152,2175713280,170655744,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,378778.81,219691.71,80.45,54.56,0.99,82510233600,75.34,167.62,84.16,183.48,29.41,84.95,30.30,86.54,30.31,86.56,119.82,100.76,3177,3427
macbook-pro.local,2026-09-13T12:20:48Z,1789302048712,0.10,0.05,0.05,0.20,0.05,101814484992,13254836224,86157017088,2402631680,32192086016,2087092224,170655744,0.00,0.00,11600.46,181740.60,2.83,1.89,0.00,0.00,116004.64,0.00,49.09,4.72,0.99,81776148480,75.80,168.43,84.69,184.44,29.57,85.22,30.30,86.54,30.31,86.56,119.56,101.82,3183,3424
macbook-pro.local,2026-09-13T12:20:49Z,1789302049768,0.10,0.05,0.05,0.20,0.05,101343363072,13253296128,85687435264,2402631680,32134725632,2622816256,170655744,63.46,0.00,1039722.91,0.00,30.31,0.00,0.00,0.00,23277.38,15518.25,26.52,5.68,0.99,81306828800,75.95,168.72,84.52,184.13,29.49,85.09,30.40,86.72,30.31,86.56,118.03,94.90,3170,3421
macbook-pro.local,2026-09-13T12:20:50Z,1789302050840,0.10,0.05,0.05,0.21,0.05,103851720704,13434044416,88015044608,2402631680,31883362304,358858752,170655744,26.11,0.93,0.00,19099.83,0.00,1.87,0.00,0.00,68759.39,45839.59,56.89,16.79,0.99,83632537600,77.44,171.39,86.31,187.36,29.41,84.95,30.40,86.72,30.31,86.56,116.91,95.67,3168,3421
macbook-pro.local,2026-09-13T12:20:51Z,1789302051866,0.10,0.05,0.05,0.19,0.05,105404776448,13537132544,89433899008,2433744896,30597136384,69697536,170655744,7.80,54.58,183648.83,886305.23,8.77,10.72,0.00,0.00,7984.73,0.00,25.34,7.80,0.99,85050212352,77.95,172.32,87.73,189.92,29.49,85.08,30.40,86.72,30.31,86.56,127.38,112.01,3171,3417

[evandhoffman]

Two test machines, by luck of hardware:

• Dev machine (this build): a fanless MacBook Air, thin/absent GPU. Exercises the absent-sensor path — MonitorSources returns nothing for the fans and (likely) the GPU, so the exporter must omit macos_smc_fan_rpm and macos_gpu_* entirely, never emit a 0. Mirrors the app's "no Fans card on a fanless Mac" rule: an idle sensor and a missing sensor must not read alike.
• M5 Max: exercises the full set — the fan/GPU/power values in the sample CSV above.

So both branches of the gap-not-zero rule get covered on real hardware, not just a fake reading.

[evandhoffman]

Verified live on the dev machine (fanless MacBook Air, M-series), release build from #59:

• GET /metrics -> 200, GET /nope -> 404.
• macos_smc_fan_rpm and sensor="ambient" correctly absent (this machine has neither) rather than emitted as zeros — the gap-not-zero rule, on real hardware.
• build_info carries the clean commit (no -dirty), so the served build matches the pushed one.

$ curl -s localhost:9650/metrics
# HELP macos_gpu_utilization_ratio GPU utilization, 0 to 1.
# TYPE macos_gpu_utilization_ratio gauge
macos_gpu_utilization_ratio 0.02
# HELP macos_gpu_vram_used_bytes GPU video memory in use, in bytes.
# TYPE macos_gpu_vram_used_bytes gauge
macos_gpu_vram_used_bytes 116850688
# HELP macos_smc_power_watts SMC power rail draw, in watts.
# TYPE macos_smc_power_watts gauge
macos_smc_power_watts{rail="input"} 4.84136438369751
macos_smc_power_watts{rail="soc"} 3.9266679286956787
# HELP macos_smc_temperature_celsius SMC temperature sensor reading, in degrees Celsius.
# TYPE macos_smc_temperature_celsius gauge
macos_smc_temperature_celsius{sensor="battery"} 25.79998779296875
macos_smc_temperature_celsius{sensor="cpu"} 50.609375
macos_smc_temperature_celsius{sensor="enclosure"} 25.25
macos_smc_temperature_celsius{sensor="gpu"} 42.015625
macos_smc_temperature_celsius{sensor="storage"} 28.5118408203125
# HELP monitor_exporter_build_info Build information; the value is always 1.
# TYPE monitor_exporter_build_info gauge
monitor_exporter_build_info{commit="v1.6.2-7-gb5c672c5",version="1.6.2"} 1