Add node metrics receiver for host-level system metrics

[neeme-praks-sympower]

Adds an opt-in node_metrics receiver that periodically scrapes host system metrics from Linux /proc and /sys and emits them as OTLP metrics, using Prometheus node_exporter metric naming.

This is the "Node metrics scraper" item from #299. It is gated behind the node_metrics_receiver cargo feature (off by default), so default builds are unaffected. The feature pulls in no new dependencies.

Usage

cargo build --features node_metrics_receiver
rotel start --receivers node_metrics --exporter otlp ...

Collectors

19 collectors, 18 enabled by default, each individually toggleable:

• CPU, load average, memory, network, filesystem, uname, time
• kernel stat counters, processes, disk I/O, vmstat
• network statistics (/proc/net/netstat + /proc/net/snmp), socket statistics (sockstat + sockstat6)
• file descriptor stats, CPU frequency, thermal zones and cooling devices
• NVMe device info, hwmon sensors
• textfile collector for custom Prometheus-format .prom files (opt-in, needs a directory)

Full option table and the emitted metric list are in README.md.

Attribution

Metric names, help strings, the default filesystem-type and mount-point exclusion lists, the virtual-device and partition heuristics, the hwmon chip-naming approach (from its collector/hwmon.go) and the textfile collector's semantics are derived from Prometheus node_exporter (Apache-2.0, copyright The Prometheus Authors). There is a notice in collector/mod.rs enumerating this per submodule, plus a pointer in each derived file so relocating one cannot lose it. Both projects are Apache-2.0, so this is licence-compatible — flagging it explicitly rather than leaving you to discover it. Say the word if you would rather have a top-level NOTICE file.

Design notes for review

• Direct procfs/sysfs reads rather than sysinfo (as originally floated in Interest in Rotel for embedded IoT + planned contributions #299): zero added dependencies, exact node_exporter parity, and coverage of Linux-only sources (hwmon, thermal_zone, nvme, sockstat) that sysinfo does not expose.
• Blocking work and shutdown. The scrape runs on spawn_blocking and is raced against the cancellation token, as is the pipeline send. Since tokio cannot abort a blocking task, a timed-out scrape is retained and no new scrape starts until it finishes — otherwise a hung statfs would leak one blocking thread per interval into the pool shared with the exporters. Note the honest limitation: this bounds the scrape loop, not process exit — a wedged blocking task can still delay runtime shutdown, since #[tokio::main] waits for in-flight blocking tasks.
• Filesystem metrics in a container need the host root reachable under --node-metrics-receiver-rootfs-path, not just a bind-mounted /proc. The mount table is read from /proc/1/mounts rather than /proc/self/mounts (the latter is always the reading process's own mount namespace, whatever procfs it comes from), and statfs is called on the rootfs-composed path while the label keeps the host's own path. Without this the receiver would report a container's own bind mounts labelled as though they were the host's filesystems — plausible, well-labelled and wrong. The README recipe was corrected accordingly.
• Textfile memory is bounded in three ways: the 10 MiB per-file limit is enforced during the read rather than from the file's reported size, at most 100k samples are taken per scrape, and a metric's HELP text is attached once rather than cloned onto every sample. A 10 MiB file of one-line samples would otherwise expand to well over 200 MB, which matters on a 256 MB gateway.
• Implausible boot times are rejected. The kernel formats btime from a signed value as %llu, so a machine whose clock is near the epoch can print a wrapped value; accepting it published a boot time ~580 billion years out and latched it into every counter's start time.
• hwmon series identity follows node_exporter: sensor is the raw sysfs name (temp1), with the human-readable text published separately as node_hwmon_sensor_label and joinable on chip+sensor. Using the label text as the identity would collide when two sensors on one chip share the same text, and would move series whenever a driver gained a label file.
• statfs64 on Linux, since plain statfs is the 32-bit interface on glibc and returns EOVERFLOW for a filesystem whose counts do not fit — which would drop large filesystems entirely on the 32-bit targets this receiver supports.
• Network filesystems are excluded by default (NFS, SMB/CIFS, Ceph, GlusterFS, 9p, AFS, and remote FUSE subtypes such as sshfs/s3fs), because statfs on an unreachable server blocks uninterruptibly. Local FUSE filesystems still report. The list is not currently overridable — happy to add a regex flag like node_exporter's if you would prefer that.
• OTLP data model. Metrics are grouped by name and type, and duplicate attribute sets within a metric are dropped, so no metric carries conflicting data points. A name reported as both gauge and counter (reachable via textfiles) is warned about rather than silently resolved. Counters use the system boot time as start_time_unix_nano (see the boot-time note below).
• Conventions. Flags use the --<name>-receiver-* prefix, durations parse with humantime (60s, 2m), paths and defaults live with the receiver config, and internal telemetry reuses rotel_receiver_accepted_metric_points / rotel_receiver_refused_metric_points with the same descriptions as the OTLP receivers (the descriptions are part of instrument identity, so a mismatch would create duplicate streams). Two counters are new: rotel_receiver_scrape_failures and rotel_receiver_empty_scrapes.
• Platform. The crate compiles everywhere; off Linux the receiver starts, warns, and only uname, time and textfile return anything. It is not cfg-gated to Linux the way kmsg is, since nothing here fails to build elsewhere — say the word if you would rather it were gated.
• 32-bit targets: AtomicU64 goes through portable-atomic on targets without native 64-bit atomics, matching what file/persistence/json_file.rs already does — needed because the ARMv5 job below builds this feature.
• Boot time is re-read every scrape rather than latched: the kernel's btime moves when the wall clock is stepped, which matters on devices with no RTC that boot near the epoch and jump when NTP syncs. The start time stamped on cumulative counters stays latched at the first known value, so a clock step is not reported downstream as a counter reset.
• Two latent CLI defects elsewhere, left alone deliberately: --kafka-receiver-check-crcs and --file-receiver-include-file-name are bool args with default_value = "true", which clap derives as SetTrue, so they cannot be disabled from the CLI either. That is the same bug this receiver had; fixing it changes user-visible behaviour in unrelated receivers, so it belongs in its own PR. Happy to open one.

Tests

126 unit tests across the receiver, plus a real-kernel integration suite. Every collector is exercised through the injectable procfs_path / sysfs_path seams against temp-directory fixtures, asserting values, units, metric types and label sets — so they run identically on Linux and macOS rather than skipping. No unit test reads the host's /proc or /sys.

tests/node_metrics_integration_tests.rs is gated exactly like the kmsg suite (NODE_METRICS_INTEGRATION_TESTS=true + target_os = "linux" + the feature). It asserts what fixtures cannot: that live /proc and /sys formats still parse, that no value is NaN or infinite (and that byte/second/filesystem metrics are non-negative — temperatures legitimately are not), and that a real host's mount table and hwmon set do not produce two data points sharing an attribute set. It skips rather than fails on hosts with no thermal zones, no NVMe, an overlay-only root, or a clock that has not yet been set by NTP.

Review history

This has been through six rounds of review. Things worth knowing that came out of them: the collector was split into collector/{mod,procfs,sysfs,textfile,util}.rs (the largest, procfs.rs, is ~1.4k lines of code plus its tests); the exclusion lists were reworked so that local FUSE mounts and a volume mounted exactly at /var/lib/docker are reported (only sub-paths are excluded, as node_exporter does); duplicate label sets are compared as sets rather than ordered lists; and each fix is covered by a test that was verified to fail when the fix is reverted.

Two things are deliberately left unpinned, since testing them would cost more infrastructure than the bugs justify: the internal-telemetry counters are only observable through the global OTel meter, so the tests assert the corresponding log lines instead; and reaching the panicking-scrape branch would need an injectable scrape seam, so only the equivalent reaping path is covered.

CI changes included

• The test job builds with --features node_metrics_receiver. Without it none of this code or its tests are compiled on a PR — which is how this branch silently broke when opentelemetry-proto's KeyValue gained a key_strindex field.
• The 32-bit ARM job builds the feature too, since libc's statfs field widths differ there and that is a target this receiver is specifically aimed at.

kmsg_receiver still has the same CI gap; happy to extend the same treatment in a separate PR. If you would rather see per-feature coverage as its own job than as flags on that line, say so and I will restructure it.

Happy to split

This is large for one commit. collector/ is already divided into mod/procfs/sysfs/textfile/util, so it splits cleanly into: (1) plumbing + config + convert + the procfs collectors, (2) the sysfs/hardware collectors, (3) the textfile collector and its exposition parser. Say the word and I will restack it that way.

[mheffner]

Thanks for the contribution! Do you have any remaining work on this or is it ready? I'm sharing my own agent's feedback below if it's helpful, but don't feel it's a requirement.

I do wonder if "node metrics" could be confusing with node.js, instead of "host metrics" or "system metrics". However, I guess "host metrics" could be confused with the hostmetricsreceiver in OTel, especially since this is different. I don't have a strong opinion on the name and node metrics would be fine, curious what you think.

---

1. The textfile per-scrape sample cap is bypassable (verified)

The PR description says "at most 100k samples are taken per scrape." That bound doesn't hold. In collector/textfile.rs, node_textfile_mtime_seconds and node_textfile_scrape_error are pushed per file (lines 92 and 232) before the cap check at line 235, and the break at line 251 only fires once parsed samples exceed MAX_TEXTFILE_SAMPLES. Files that yield zero samples never advance total_samples, so the loop never terminates early.

I confirmed this with a probe test: 5,000 .prom files containing only a comment produced 10,000 metrics. There is also no cap on the file count, and each file is read_capped up to 10 MiB, so scrape time is unbounded in the number of files too — on a shared blocking thread, with no way to abort it.

The textfile directory is often the least-trusted input the collector touches. Suggested fix: count every metric pushed for a file against the cap (not just parsed samples), and add a file-count limit alongside the size limit.

2. A wedged scrape can prevent process exit

You disclose this, but I want to underline how it presents operationally. src/bin/rotel/main.rs:160 uses #[tokio::main], so the runtime is dropped at the end of main, and that drop blocks on spawn_blocking tasks that have already started. A statfs on a dying local disk or a hung hwmon I2C read means SIGTERM does not terminate rotel — Kubernetes waits out terminationGracePeriodSeconds and SIGKILLs it, systemd does the same. The scrape-loop bound is well built, but it only protects the loop.

Two options: build the runtime explicitly with Builder::new_multi_thread() and call runtime.shutdown_timeout(...) (a change outside this receiver, reasonable as a follow-up), or at minimum say this in the README beside the filesystem-collector caveats. Right now nothing user-facing says the receiver can delay shutdown.

3. statfs failure warns on every scrape

procfs.rs:601 logs at warn! per failing mount per scrape, and the comment argues for it. But this module elsewhere reasons the opposite way and is right to — textfile.rs:107 and convert.rs:66 both explicitly choose debug because "this repeats every scrape for as long as the file is oversized." A stale NFS handle or a permission-denied mount is exactly as persistent, and at the 1s minimum interval that's one warning per second forever. I'd match the textfile pattern: debug! plus a durable per-mount error metric, or warn once per distinct mount point.

---

Medium

4. refused/accepted counters over-count on the abandon path. receiver.rs:400 records the batch as refused, but the code comment at 395 correctly notes flume's SendFut "may still be delivered" when dropped. So an abandoned batch can be counted refused and exported. Same in the shutdown-timeout branch at 362. Either use try_send semantics so the outcome is definite, or say so in the counter description.

5. Nothing warns when --procfs-path is set without --rootfs-path. This is the "plausible, well-labelled and wrong" case your description says you fixed — but only via the rootfs flow. --node-metrics-receiver-procfs-path=/host/proc alone reads the host mount table while statfs-ing container-namespace paths, labelled with host paths. README:1410 warns; the code doesn't. A startup warn! when Filesystem is enabled and procfs_path != DEFAULT_PROCFS_PATH and rootfs_path == "/" would close it, and build_config is already the place that rejects the sibling misconfiguration.

6. hwmon chip identity can collide silently. sysfs.rs:315 derives chip from the backing device's basename alone, while the fallback deliberately appends hwmonN for uniqueness. Two hwmon instances registered under one device (multi-page PMBus parts, some drivers) yield identical chip+sensor attribute sets, and convert.rs drops the duplicate with only a debug!. I checked this host — 12 hwmon entries, all distinct device basenames — so this is uncommon, not impossible. Always appending the index, or detecting duplicates within a scrape, is cheap insurance for a silent data-loss path.

7. NaN/±Inf from textfiles reach OTLP. parse_prometheus_line accepts them (the comment at line 381 notes it deliberately). Your integration test asserts host metrics are finite, but textfile samples bypass that entirely, and some backends reject a whole payload containing a non-finite double. Worth dropping non-finite textfile samples, or at least documenting it.

Low

8. is_partition over-matches. The letter run after sd/vd/xvd/hd/dasd is unbounded, so sdcard0, hdmi0, sdio1, vdisk1 all classify as partitions (verified). node_exporter bounds those prefixes to a single letter. No real kernel block-device name hits this today — /proc/diskstats only ever carries kernel names — so it's latent, but the multi-letter case you're supporting (sdaa1) only needs two.

9. Allocation per data point. name, unit and description on CollectedMetric are String but are static literals at nearly every call site — roughly 5k needless allocations per scrape at the ~1,300 points I measured. &'static str or Cow<'static, str> for those three fields removes almost all of it. Negligible at 60s; less so at the 1s minimum the CLI permits.

10. No per-collector duration/success metrics. node_exporter has node_scrape_collector_duration_seconds and _success. Given that one wedged collector stalls the entire scrape and the receiver-level rotel_receiver_scrape_failures can't say which, this is the diagnostic an operator actually needs during the incident your timeout logic is designed for.

11. The wrapped-btime case never reaches its own warning. collector/mod.rs:146 parses as i64, so 18446744073709551606 — the exact value the comment describes — fails to parse and returns None before the plausibility check at 151. The rejection is correct and tested, but silent. Parsing as u64/u128 and then range-checking would make the warn! fire for the case it was written for.