Skip to content

inventory-agent: deliver and schedule it on the node #635

Description

inventory-agent is built and released but nothing installs it, configures it, or
runs it. It is the only inventory component with no deployment path at all, which
means the aggregator, viewer and inspector have nothing to aggregate.

Why it is a host process, not a workload

designs/inventory.md says "The agent runs as root on each node", and lists
deployment manifests for the aggregator, viewer and inspector only. The code
agrees, and rules out a DaemonSet as currently written:

  • One-shot. Execute() (internal/inventory/agent/collect.go:104-136) collects,
    writes SQLite, publishes over gRPC and returns. No loop, no interval flag. A
    DaemonSet pod can only use restartPolicy: Always, so a cleanly-exiting binary
    would be restarted forever.
  • Root required. collect.go:105-107 hard-fails when os.Geteuid() != 0.
  • Deep host access. /dev/ipmi0, the 0400 product_serial and product_uuid
    under /sys/class/dmi/id, /proc/cpuinfo, /proc/meminfo.
  • --db ./inventory.db is relative to the working directory; the design calls
    it a node cache.

#633 accordingly ships it as a GoReleaser binary and does not build an image for
it.

Work

1. systemd service plus timer. The one-shot, root-only shape fits a timer
exactly. Precedent for unit assets the node agent installs:
pkg/agent/phases/rootfs/assets/localdns.service, config-regeneration.service,
pkg/agent/phases/nodestart/assets/kubelet.service.

2. Binary delivery. Either through the node agent's goalstate machinery
(pkg/agent/goalstates/agentupgrade.go shows the blue-green pattern for the
agent's own binary) or baked into images/host-ubuntu2404. Whichever is chosen has
to handle upgrade alongside the rest of the node's software.

3. Collector tool packages via pkg/agent/phases/host/apt.go. Each backs a
specific collector, and a missing binary means that collector silently reports
nothing:

Package Collector
dmidecode cpu.go, memory.go - DMI processor and DIMM tables
pciutils gpu.go - lspci enumeration
ethtool network.go - NIC driver and firmware
kmod disk.go, gpu.go - modinfo driver versions
udev disk.go - udevadm disk properties
lldpd lldp.go - lldpctl switch neighbours

The NVIDIA tools (nvidia-smi, nvidia-imex-ctl) come from the host driver and
are version-locked to it; they cannot be installed here.

4. The collector endpoint. --collector defaults to
inventory-collector:50051 (cmd/inventory/inventory-agent/main.go:28), a cluster
Service DNS name that cannot resolve from the host. It needs a NodePort,
LoadBalancer, hostNetwork Service or an explicitly configured address, and that
address has to reach each node as config. This couples to #636, where the Service
becomes inventory-collector-<site>.

5. --db path. ./inventory.db should become something durable such as
/var/lib/unbounded/inventory.db rather than depending on the unit's working
directory.

6. Cadence and failure handling. At provisioning only, or periodically, and
with what jitter. What happens when the aggregator is unreachable: the local SQLite
cache already exists, so the question is whether the agent retries, or the next
timer firing simply re-publishes.

Open question: how to prove it works

This is where a true end-to-end test becomes possible, and it is also where it is
hardest. The collectors read DMI, IPMI, PCI, LLDP and NVLink, which return little
or nothing on a cloud VM, so a meaningful test needs real hardware - pointing at a
bare-metal site such as boulderlab, which is also the site that goes unreachable
and would need the tolerance treatment the DaemonSet deploy gate already has.

Cheaper options worth considering first: assert the unit is installed and enabled
after provisioning; assert one timer firing produces a non-empty local SQLite
database; assert a record for the node's serial appears in PostgreSQL within one
interval.

Acceptance criteria

  • After a node is provisioned, inventory-agent is installed, its unit is enabled,
    and it has run at least once.
  • A record identified by the node's DMI serial reaches the aggregator.
  • The agent survives an unreachable aggregator without wedging the node or filling
    the disk.
  • The collector address is configuration, not a hardcoded cluster Service name.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions