Store the LibreNMS RRD files through RRDCached, and bring the role up to the LFOps standards - #341
Merged
Merged
Conversation
The selinux role was handed `selinux__fcontexcts__dependent_var`, a name it never reads. The role has no argument_specs to reject an unknown parameter, so the run continued without a word and /opt/librenms kept the default /opt label, leaving the RRD tree unlabelled for anything that is not the owning user.
…M starts
php-fpm resolves the user of every pool while starting up and aborts the whole
daemon when one of them is unknown ("cannot get uid for user", exit code 78).
The php role deploys the LibreNMS pool, which runs as `librenms`, and starts
php-fpm before the librenms role creates that user, so a fresh installation
never got past it.
The group and user tasks move into their own file, imported by the role where
they sat before and by the playbook's pre_tasks. Tagged `librenms` and `php`
there, so a run limited to either tag still creates the user.
Verified on Rocky 8: a pool with an unknown user fails the unit with
status=78/CONFIG, not just that pool.
The checkout was cloned as root and the tree handed to `librenms` afterwards. Every later run then hit git's safe.directory protection, "detected dubious ownership in repository at '/opt/librenms'", so the role could never update an installation it had created. LibreNMS updates itself as its own user, and the role now clones the same way, into a directory created for that user beforehand. Three tasks reported a change on every run and made the molecule idempotence step impossible: - `chown -R` and `chmod -R` had no changed_when. The chown now derives it from `--changes` output, as CONTRIBUTING prescribes. - `chmod -R g=rwx` on rrd, logs, bootstrap/cache and storage can never settle: the application creates files below them with umask 022 as it runs, 294 of them on a host a few minutes after installation. The group bits are dropped instead of chased. PHP-FPM, cron and the scheduler all run as `librenms`, which owns those trees, so nothing needs them. - `composer install` now keys its changed_when on composer's own "Nothing to install, update or remove". `restorecon` loses its --force. A file the application creates gets the right type but the SELinux user `unconfined_u`, and -F rewrites that back to `system_u` on every run. Without it there is nothing to relabel, and the type, which is what the targeted policy enforces, is unaffected. Verified on Rocky 8, 9 and 10: converge, verify and idempotence pass.
The module was compiled and installed by the role itself, with checkmodule and semodule_package driven from a template in templates/tmp/, and only when `semodule --list-modules` did not know the name yet. An edit to the source therefore never reached a host that already had an older version. The selinux role gained a `content_te` subkey for exactly this case, so the module moves into librenms__selinux__modules__dependent_var next to the role's booleans and file contexts, and the playbook hands it over. That role compares the compiled package against the installed one, so a changed source is deployed and an unchanged one reports nothing.
LibreNMS writes one RRD update per metric per poll. RRDCached collects them in memory and writes them out every 30 minutes, which upstream puts at 30% to 40% less disk I/O on the poller. The earlier attempt sat commented out in the role with "currently librenms cannot find the rrdcached socket". The reason is that it listened on /tmp/rrdcached.sock, and both httpd.service and php-fpm.service ship PrivateTmp=true, so the socket the daemon creates is not the one the web interface looks for. It now listens on /run/rrdcached.sock. Details worth knowing: - rrdcached.service and rrdcached.socket both come with the rrdtool package on RHEL 8, 9 and 10, contrary to what the LibreNMS docs say. The service is configured with a drop-in rather than by templating the unit, and the socket unit is disabled: it binds /tmp/rrdcached.sock, and socket activation would make rrdcached ignore the listen address of the drop-in. - The rrdcached_librenms policy module goes through the selinux role. Without it rrdcached_t may not touch the RRD tree (httpd_sys_rw_content_t) or create its socket in /run (var_run_t), and httpd_t may not connect to it. - $config['rrdtool_version'] is set from the installed rrdtool. LibreNMS assumes 1.4 otherwise and then creates and tunes the files itself instead of letting rrdcached do it. - The journal goes to /var/tmp, where the base policy lets rrdcached_t write (type_transition to rrdcached_tmp_t), which is also what upstream's unit does. Verified on Rocky 8, 9 and 10 under enforcing SELinux: rrdcached runs in rrdcached_t, the RRD files it writes carry httpd_sys_rw_content_t, PHP-FPM reaches the socket from httpd_t, and ausearch reports no denials.
z00-linuxfabrik.conf is unambiguous while the role and the unit are the same software, as in mariadb_server and redis. A role configuring a unit that belongs to something else has no such claim on the name: roles/librenms writes into rrdcached.service.d, and a role managing RRDCached in its own right would overwrite it with its own z00-linuxfabrik.conf.
Asserts what only the running system can confirm: rrdcached is up and confined to rrdcached_t, its socket sits outside /tmp and belongs to the librenms group, data written through the daemon lands in the RRD tree with the httpd_sys_rw_content_t label, PHP-FPM answers from httpd_t through the vHost, and ausearch reports no denials. A plain "the service is running" check would pass while every write is denied, which is how the earlier attempt at this failed. The PHP probe served through the LibreNMS vHost is the only way to exercise the httpd_t side: the domain follows from who executes the code, not from the user it runs as, and runcon cannot get there because httpd_t may not execute the interpreter. COMPATIBILITY.md moves librenms on RHEL 9 and 10 from (x) to x, which this scenario is what proves.
poller-wrapper.py and discovery-wrapper.py import the LibreNMS package, whose queuemanager imports pymysql at module level, so a host without PyMySQL loses every poll cycle before it starts. dotenv reads /opt/librenms/.env, psutil reaps the worker processes. Until now PyMySQL only reached the host through mariadb_server__python__modules__dependent_var, which the playbook feeds into the python role. A host running with librenms__skip_mariadb_server, an external database, therefore never got it. redis is deliberately left out: LibreNMS imports it for distributed polling only, which this role does not set up.
LibreNMS/Util/Number.php computes counter deltas with GMP, falls back to bcmath, and then to float. We installed neither, so every rate derived from a 64-bit interface counter went through a 53-bit mantissa. The precision warning in that fallback only fires above PHP_FLOAT_MAX, which a uint64 never reaches, so the loss was silent. The upstream package list carries php-gmp on every RHEL version.
Our copy was the version the LibreNMS install docs carried until PR #20308 (2026-08-15) replaced it. The current one additionally allows httpd_t to create an ICMP socket, to bind a raw socket, and node_bind on node_t, which fping needs depending on how it opens its socket. Taken verbatim from the docs again, and compiled with checkmodule plus semodule_package on Rocky 8, 9 and 10. The molecule scenario pings 127.0.0.1 through a probe served by the LibreNMS vHost, so the rules are exercised in the httpd_t domain they are written for.
librenms__rrdcached_enabled: false only rewrote config.php. The daemon kept running with the role's drop-in and with up to 30 minutes of unflushed updates, while LibreNMS started writing the same files directly. Stopping it shuts it down cleanly, which flushes what it holds. The rrdcached_librenms policy module now stays installed instead of following the toggle: the selinux role runs before this one, so removing it in the same run would deny exactly that shutdown flush. The drop-in and the RRD files are left alone, so the switch can be reversed with another run of the role.
* acl: composer's postInstall runs setfacl when no .env exists yet. It is not part of a minimal RHEL 10 installation, so that call failed there while it succeeded on 8 and 9. The comment about the group bits described something the role does not do and was corrected. * ImageMagick: not referenced anywhere in the LibreNMS tree and in none of the upstream package lists. * httpd_cache_t for /opt/librenms/cache, which upstream labels and we did not. * httpd_can_sendmail, and httpd_can_network_connect_db as soon as the database is on another host. httpd_execmem, which upstream also sets, stays off: it is only needed with OPcache JIT, which the php role does not enable. * README: the minimum PHP version LibreNMS supports is 8.4, not 7.3.
The role had no argument spec, so a missing librenms__database_login only surfaced when a task dereferenced it. Declares the variables documented in the README, with librenms__database_login and librenms__fqdn required. librenms__rrdcached_service_state carries no default: defaults/main.yml derives it from librenms__rrdcached_service_enabled with a Jinja2 expression, which argument_specs cannot evaluate.
APP_TRUSTED_PROXIES was only written when the inventory listed a proxy, so a host with an empty list kept LibreNMS's own default, which trusts 127.0.0.1 since upstream 26.5.0. A host that trusts a proxy accepts its X-Forwarded-* headers, so anything able to reach LibreNMS from that address decides what LibreNMS records as the client and as the protocol. The setting is now written on every run, including with an empty value. LibreNMS treats the empty string as a deliberate "no proxies": EnvHelper::parseArray keeps "" out of the comma split, and the upstream security docs use exactly that value. Writing it unconditionally also means the role converges the setting instead of leaving a stale line behind when a proxy is dropped from the inventory. APP_URL moves into its own task, because it has no meaningful empty value: writing it empty sets Laravel's app.url to the empty string rather than falling back to anything, and LibreNMS then reports it as misconfigured.
LibreNMS is a Laravel application: the session cookie is issued by the
framework's session middleware, which reads config('session.secure'). LibreNMS
config/session.php defines only driver, lifetime and same_site, so that key
falls through to Laravel's base config, env('SESSION_SECURE_COOKIE') from
/opt/librenms/.env. Without it, validate.php fails with "Secure session
cookies are not enabled" on every HTTPS install.
php__ini_session_cookie_secure of the php role cannot cover this. It renders
session.cookie_secure into php.ini, which governs the native session handler
that Laravel never uses.
The default follows librenms__config_app_url rather than being on outright:
the role ships a port 80 vHost, and the flag on a plain-HTTP host means the
browser stops sending the cookie back and the login fails with 419 Page
Expired. An https base URL is the host's own statement that it has TLS, and
the setting is written on every run, so it follows that URL in both
directions.
…ocabulary
librenms__scheduler_service_enabled and librenms__scheduler_service_state make
the LibreNMS scheduler timer behave like every other service in the collection,
with enabled and state as separate tasks instead of one "systemctl enable --now".
The blocks are split along the controlled tag vocabulary of CONTRIBUTING.md, so
librenms:cron, librenms:logrotate and librenms:state each manage their own area.
The scheduler units come out of the git checkout and an update of LibreNMS can
change them, but nothing told systemd to read them again: it kept running the
version it had loaded before. Both units now go through one loop whose result
gates a daemon-reload.
git is installed by the role itself. It arrived through the general-purpose apps
role via librenms__apps__apps__dependent_var, which CONTRIBUTING.md rules out as
a dependency, and setup_librenms.yml therefore no longer runs apps. Its skip
variables are named after the playbook now (setup_librenms__skip_*), the form
the README documents; two of them already were.
Further alignment, none of it changing what a run does:
* restorecon is guarded by the SELinux status check
* the role-internal registers carry the __ prefix
* the .env key regexps are anchored on the "=", with a note on why the file is
edited with lineinfile rather than rendered from a template
* config.php.j2 loses three d() fallbacks whose variables all have a default,
and whose update_channel value contradicted it ('master' vs 'release')
The README gains the Requirements and Post-Installation Steps sections, the
git-checkout and self-update behaviour, the subkeys of librenms__database_login
and the corrected trigger of the librenms tag. The Molecule scenario asserts the
scheduler timer is active and enabled.
The clone runs as librenms, whose home is /opt/librenms, and Ansible creates the become user's remote_tmp before it runs the module. git clone therefore found ~/.ansible in the directory it was about to fill and aborted with "already exists and is not an empty directory" on every fresh host, which the Molecule scenario reproduced on Rocky 8, 9 and 10. Pointing remote_tmp outside the home fixes it. The composer task and the two verify tasks that become librenms get the same treatment, which also keeps ~/.ansible out of the git working tree that the chown and the restorecon walk. No CHANGELOG entry: cloning as librenms arrived earlier in this same branch, so no release ever carried the broken fresh install.
The fuller wording replaced this line in "chore(roles/librenms): align packages and SELinux settings with upstream". Rebasing the branch onto main resurrected the older one next to it.
The setting is only observable in what LibreNMS records as the client address, and the Auth History page shows exactly that value: authlog.address is Request::ip(), which is what the TrustProxies middleware decides. Also notes the negative test from an unlisted host, and that an unset APP_TRUSTED_PROXIES falls back to trusting 127.0.0.1 while an empty one trusts nothing, which the deployed .env line alone does not tell apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviews
roles/librenmsandplaybooks/setup_librenmsagainstCONTRIBUTING.mdandroles/example, adds RRDCached in front of the RRD files, and fixes what the review and the new Molecule scenario turned up. 18 commits, each reviewable on its own.Breaking change
LibreNMS trusts no reverse proxy any more, where it previously trusted one running on the LibreNMS host itself. It therefore ignores the
X-Forwarded-*headers of such a proxy, and logs, access control and the detected protocol use the proxy's address instead of the client's. Hosts with a proxy in front of LibreNMS list it inlibrenms__config_app_trusted_proxies;['127.0.0.1']restores the previous behaviour.RRDCached
LibreNMS wrote every RRD file on every poll cycle. The role now puts RRDCached in front of them, collecting a cycle in memory and writing it out every 30 minutes, which takes roughly a third of the poller's disk I/O off the host. The RRD files keep their format, so switching it on or off needs no migration.
The socket lives in
/run, not/tmp, becausehttpdandphp-fpmrun withPrivateTmp=trueand would not see it there; therrdcached.socketunit of therrdtoolpackage is disabled, since socket activation makes the daemon ignore the configured path. On RHEL therrdcached_librenmspolicy module (taken verbatim from upstreamdist/rrdcached/) is deployed through theselinuxrole.librenms__rrdcached_enabled: falsestops the daemon, which flushes on the way out, and points LibreNMS back at the files.Correctness
/opt/librenmswere never applied.http_fpingcarries the rules the upstream documentation lists, so pinging from the web interface also works where fping uses an ICMP socket or binds a source address.librenms__config_app_urlishttps://, which LibreNMS' own validation flags when missing. Laravel reads this from.env, sophp__ini_session_cookie_securenever reached it.Added
meta/argument_specs.ymldeclaring every user-facing variable.librenms__scheduler_service_enabled/librenms__scheduler_service_state, and thelibrenms:cron,librenms:logrotate,librenms:rrdcachedandlibrenms:statetags.setup_librenmsMolecule scenario.selinux: a policy module can be defined inline viacontent_te, so a role can inject one without a directory on the controller.Consistency
The role installs
gititself rather than injecting it into the general-purposeappsrole, whichCONTRIBUTING.mdforbids as a dependency, sosetup_librenmsno longer runsapps. Its skip variables are allsetup_librenms__skip_<role>now.restoreconis guarded by the SELinux status check, role-internal registers carry the__prefix, and the README gained the Requirements and Post-Installation Steps sections plus the git-checkout and nightly self-update behaviour.Verification
The Molecule scenario passes the full sequence on rocky8, rocky9 and rocky10: converge
changed=35, idempotencechanged=0, verify green in both passes.verify.ymlasserts the running system rather than the role's mechanics:rrdcachedruns in therrdcached_tdomain (not theunconfined_service_ta missing transition would leave), its socket belongs to thelibrenmsgroup, anrrdtool create/update/flushcachedround-trip as thelibrenmsuser actually lands a file in the RRD tree with the right label, PHP-FPM reaches the socket through a probe served by the LibreNMS vHost (the only way to exercise thehttpd_tpath),fpingruns from that same domain,ausearchreports no denials, the poller can import its Python modules, the PHP CLI loadsgmp, the two.envsettings are in place, and the scheduler timer is active and enabled.The scenario earned its keep immediately: on genuinely fresh VMs the clone aborted with
destination path '/opt/librenms' already exists and is not an empty directory. Ansible creates the become user'sremote_tmpunder that user's home, which forlibrenmsis/opt/librenms, sogit clonenever saw an empty directory. Fixed by pointingremote_tmpelsewhere for the tasks that becomelibrenms.Note for the reviewer
COMPATIBILITY.mdis unchanged. Debian and Ubuntu were not attempted.ansible_remote_tmppoints at a fixed path in/tmp, which Ansible creates0700owned bylibrenms. Say so if you want it somewhere only root and the service user can reach.