diff --git a/CHANGELOG.md b/CHANGELOG.md index 5bd71f09f..18ea8d5b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -**Highlights:** On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag. +**Highlights:** On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag. PHP-FPM finally reports what it is doing: pools log their application errors on Debian instead of dropping them, a hung worker is reclaimed after 65 minutes, and repeated worker crashes trigger a reload. ### Breaking Changes +* **role:php**: PHP-FPM kills a request after 65 minutes instead of never. A worker blocked in a system call, on a database socket that never answers for example, held its slot forever and the pool bled capacity until it was full. The limit sits five minutes above the 3600 seconds Nextcloud raises `max_execution_time` to for large uploads, so PHP's own limit still fires first there. Raise `php__fpm_pool_conf_request_terminate_timeout__group_var` (or the `__host_var`) for workloads with legitimately longer requests, or set it to `0` to restore the previous behaviour. * **role:mariadb_server**: The InnoDB buffer pool grows from 128 MiB to 512 MiB, so a database with more than a trivial amount of data is served from memory instead of from disk. Every host running this role therefore uses roughly 384 MiB more RAM after the next restart of the service. * **role:mariadb_server**: The InnoDB redo log grows from 32 MiB to the 96 MiB MariaDB itself ships, so a write-heavy server no longer stalls waiting for a checkpoint on a redo log sized for much smaller workloads. InnoDB resizes the log itself when the service next restarts, also after an unclean shutdown, but the data directory needs 64 MiB more free space for it; check that on hosts that are tight before deploying. Set `mariadb_server__cnf_innodb_log_file_size__group_var: '32M'` (or the `__host_var`) to keep the previous size. * **role:mariadb_server**: `innodb_snapshot_isolation` now defaults to `OFF`. Turning it on requires support from the application: a transaction in `REPEATABLE READ` that modifies a row another transaction changed after its snapshot was taken is aborted with `ER_CHECKREAD`. The application has to catch that error and retry the transaction, otherwise the write fails under concurrent load. To restore the previous behaviour on hosts whose application is known to handle it, set `mariadb_server__cnf_innodb_snapshot_isolation__group_var: 'ON'` (or the `__host_var`). @@ -23,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +* **role:php**: The `php:logrotate` tag rotates the per-pool PHP-FPM logs on Debian, which the packaged configuration does not cover. +* **role:php**: The `[global]` section of the PHP-FPM configuration is managed from the inventory, so the log level and the emergency reload after repeated worker crashes can be set; the reload is now on by default after ten crashes within a minute. * **role:files**: A file can opt out of the backup copy that is written before it is overwritten, via the `backup` subkey of `files__files__*_var`. * **role:collabora**: The `collabora:configure` tag deploys `coolwsd.xml` and the logrotate configuration without touching the packages. * **role:docker**: The address pools docker assigns container network subnets from (`default-address-pools`) can be configured. @@ -34,6 +37,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +* **role:apache_httpd**: OCSP stapling is off by default, so a certificate whose issuer runs no OCSP responder (every Let's Encrypt certificate today, and therefore everything the `acme_sh` role obtains) no longer floods the error log with `AH02218` and `AH02604` on every start and reload while stapling nothing; set `apache_httpd__mod_ssl_ssl_use_stapling: 'on'` for a CA that still answers. +* **role:php**: A PHP-FPM pool starts with 10 workers and keeps 10 idle instead of 5 each, so an ordinary traffic spike no longer makes PHP-FPM fork in bursts and warn about it every second, at the cost of five more resident workers per pool. * A service that depends on a kernel setting deployed by the `kernel_settings` role now starts after TuneD, so the setting is in place before the service reads it. Until now such a service could come up while TuneD was still applying the profile and then run with the old value until its next restart, while `sysctl` and `tuned-adm verify` already reported the new one (roles `graylog_datanode`, `graylog_server`, `mariadb_server`, `mongodb`, `redis`). * A repository file that carries mirror credentials is deployed with mode `0600` instead of `0644`, so an unprivileged `dnf` or `zypper` no longer lists those repositories (all `repo_*` roles). * **role:collabora**: A host running a Collabora version the role has no configuration template for aborts with that version and the list of supported ones, instead of failing on a missing file. @@ -42,6 +47,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +* **role:php**: On Debian a pool's `slowlog` points at a directory that exists, so a backtrace is actually written once `php__fpm_pool_conf_request_slowlog_timeout__*_var` is set. +* **role:php**: On Debian the applications' PHP errors reach `/var/log/php-fpm/-error.log` instead of being discarded without a trace, because the path used before was not writable by the pool user. * **plugin:bitwarden_item, module:bitwarden_item**: A vault that is not unlocked is reported with the `bw serve` endpoint it was read from and the status it actually has, plus the hint that `bw serve` keeps the session it was started with. The previous message pointed at `bw login` and `bw unlock`, which do not reach a running `bw serve`. * **plugin:bitwarden_item**: Error messages no longer carry a doubled period in the middle. * **role:rocketchat**: The environment file no longer sets `MONGO_OPLOG_URL`, which Rocket.Chat has ignored since 5.0.1. A host without a MongoDB replica set also gets a usable environment file again, instead of one whose MongoDB URL and port ended up on the same line. diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 7db4f2a19..6f2dbb5d9 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -113,7 +113,7 @@ Which Ansible role is proven to run on which OS? | open_vm_tools | | | x | x | (x) | | | | | | opensearch | | | x | x | (x) | | | | | | openvpn_server | | | x | x | x | | | | | -| php | x | x | x | x | (x) | (x) | (x) | (x) | | +| php | x | x | x | x | x | x | x | - | | | podman_containers | | | (x) | x | (x) | | | | | | policycoreutils | | | x | x | x | | | | Fedora 35 | | postfix | x | x | x | x | x | (x) | (x) | (x) | Fedora 35 | diff --git a/roles/apache_httpd/README.md b/roles/apache_httpd/README.md index ddd95c74e..676908ace 100644 --- a/roles/apache_httpd/README.md +++ b/roles/apache_httpd/README.md @@ -794,14 +794,15 @@ apache_httpd__skip_mod_security_coreruleset: true `apache_httpd__mod_ssl_ssl_use_stapling` -* See [SSLUseStapling](https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslusestapling). +* See [SSLUseStapling](https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslusestapling). Stapling only does something for a certificate whose issuer runs an OCSP responder. Let's Encrypt, which the `acme_sh` role obtains certificates from, no longer publishes one, so its certificates carry no OCSP URI at all. Switch it on for a CA that still answers, or point `SSLStaplingForceURL` at a responder via a vhost of your own. * Type: String. -* Default: `'on'` +* Default: `'off'` +* Matches the upstream default. It used to be `'on'`, which on a certificate without an OCSP URI makes mod_ssl log `AH02218` and `AH02604` at error level, once per certificate and vhost, on every start and every reload, without stapling anything. Example: ```yaml # optional - mod_ssl -apache_httpd__mod_ssl_ssl_use_stapling: 'on' +apache_httpd__mod_ssl_ssl_use_stapling: 'off' ``` diff --git a/roles/apache_httpd/defaults/main.yml b/roles/apache_httpd/defaults/main.yml index 2258c8f14..83e2bc7f0 100644 --- a/roles/apache_httpd/defaults/main.yml +++ b/roles/apache_httpd/defaults/main.yml @@ -205,7 +205,7 @@ apache_httpd__mod_security_coreruleset_url: 'https://github.com/coreruleset/core apache_httpd__mod_security_coreruleset_version: '4.27.0' # mod_ssl -apache_httpd__mod_ssl_ssl_use_stapling: 'on' +apache_httpd__mod_ssl_ssl_use_stapling: 'off' # upstream default: off # Apache Virtual Host Configuration diff --git a/roles/apache_httpd/meta/argument_specs.yml b/roles/apache_httpd/meta/argument_specs.yml index 38c50e404..81db51752 100644 --- a/roles/apache_httpd/meta/argument_specs.yml +++ b/roles/apache_httpd/meta/argument_specs.yml @@ -222,7 +222,7 @@ argument_specs: apache_httpd__mod_ssl_ssl_use_stapling: type: 'str' required: false - default: 'on' + default: 'off' description: >- Whether OCSP stapling is used (`SSLUseStapling`). diff --git a/roles/apache_httpd/templates/etc/httpd/conf-available/ssl.conf.j2 b/roles/apache_httpd/templates/etc/httpd/conf-available/ssl.conf.j2 index d3eaa7a34..88661abbe 100644 --- a/roles/apache_httpd/templates/etc/httpd/conf-available/ssl.conf.j2 +++ b/roles/apache_httpd/templates/etc/httpd/conf-available/ssl.conf.j2 @@ -29,7 +29,13 @@ SSLSessionCacheTimeout 1800 # ensures Perfect Forward Secrecy is not compromised if the server is not restarted regularly: SSLSessionTickets off -# OCSP Stapling, only in httpd 2.3.3 and later +# OCSP Stapling, only in httpd 2.3.3 and later. +# Off by default: stapling needs an OCSP responder in the certificate's AIA extension, and +# Let's Encrypt, where the acme_sh role gets its certificates, publishes none any more. With +# stapling on, such a certificate makes mod_ssl log AH02218 and AH02604 at error level per +# certificate and vhost on every start and reload, and staple nothing. +# The cache and timeout directives below are inert while stapling is off; they are kept so that +# switching the variable on is enough for a CA that does run a responder. SSLUseStapling {{ apache_httpd__mod_ssl_ssl_use_stapling | d('off') }} SSLStaplingCache shmcb:/var/run/ocsp(128000) SSLStaplingResponderTimeout 3 diff --git a/roles/php/README.md b/roles/php/README.md index 01c7cb2c8..b734d4e07 100644 --- a/roles/php/README.md +++ b/roles/php/README.md @@ -33,6 +33,17 @@ This role never exposes to the world that PHP is installed on the server, no mat *Available since LFOps `2.0.0`.* +## How the Role Behaves + +**Where PHP-FPM logs, and what a monitoring check sees there.** PHP-FPM keeps two kinds of log apart, and only one of them is the process manager's own. The master writes its error log to the path the distribution's `php-fpm.conf` names (`/var/log/php-fpm/error.log` on RedHat, `/var/log/phpX.Y-fpm.log` on Debian); that file holds pool saturation, worker crashes, request timeouts and the start / reload / shutdown markers, and it is what the [php-fpm-logfile](https://linuxfabrik.github.io/monitoring-plugins/check-plugins/php-fpm-logfile.html) check reads. The applications' own errors go to `/var/log/php-fpm/-error.log`, in PHP's format, and need a separate check. Nothing of either reaches the journal: the unit's journal entries are systemd's own start and stop lines. + +**Pool logs need a directory the workers may write to.** `/var/log/php-fpm` is created owned by the web server user, mode `0770`, mirroring what the RedHat package ships. A pool that runs as a different user than `php__fpm_pools__*_var` defaults to cannot create its log there; give it its own directory and point `raw` at it. + +**The `[global]` section is deployed as a drop-in.** `php-fpm.conf` belongs to the package, so the role writes `z00-linuxfabrik-global.conf` into the pool directory instead. That covers `log_level` and the two `emergency_restart_*` directives. It deliberately does not set `error_log`: RedHat reads the pool directory *before* its own `[global]` and would override the value again, while Debian and Fedora read it after and would not, so the same drop-in would move the log on some hosts and not on others. + +**Spawn pressure is loud by design.** Whenever `pm.min_spare_servers` cannot be met, PHP-FPM doubles its spawn rate every second and logs a `seems busy` warning from rate 8 on, once per second, unthrottled. A single traffic spike therefore writes a block of warnings even when the pool never fills up. `php__fpm_pool_conf_pm_start_servers__*_var` and `php__fpm_pool_conf_pm_min_spare_servers__*_var` keep a warm reserve that pushes the point where this starts; on a host where spikes are normal, pass `--ignore='seems busy'` to the monitoring check so that only the actual pool saturation alerts. + + ## Dependent Roles Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/README.md) that installs this role runs these for you. Optional ones can be disabled via the playbook's skip variables. @@ -48,7 +59,9 @@ Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/RE * Installs php, php-fpm and composer. * Installs and removes the configured PHP modules. * Deploys the `z00-linuxfabrik.ini` for every SAPI. -* Deploys and removes the PHP-FPM pools. +* Deploys and removes the PHP-FPM pools, and the `[global]` drop-in next to them. +* Creates the PHP-FPM log directory. +* Deploys the logrotate configuration for the per-pool logs (Debian only). * Manages the state of the php-fpm service. * Pins the `php`, `phar` and `phar.phar` alternatives (Debian with `php__version` set only). * Triggers: php-fpm.service restart. @@ -60,7 +73,7 @@ Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/RE `php:fpm` -* Deploys and removes the PHP-FPM pools. On Debian these live under the declared version's tree, on RedHat under `/etc/php-fpm.d`. +* Deploys and removes the PHP-FPM pools, the `[global]` drop-in and the log directory. On Debian the configuration lives under the declared version's tree, on RedHat under `/etc/php-fpm.d`. * Triggers: php-fpm.service restart. `php:ini` @@ -68,6 +81,11 @@ Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/RE * Deploys the `z00-linuxfabrik.ini`. RedHat has a single `/etc/php.d`, Debian one conf.d per SAPI (apache2, cli and fpm) below the declared version's tree. * Triggers: php-fpm.service restart. +`php:logrotate` + +* Debian only. Deploys `/etc/logrotate.d/php-fpm-pools` for the per-pool logs. On RedHat the packaged logrotate configuration already covers them. +* Triggers: none. + `php:modules` * Installs and removes the PHP modules from `php__modules__combined_var`. @@ -386,6 +404,40 @@ php__ini_upload_max_filesize__host_var: '10000M' ``` +## Optional Role Variables - PHP-FPM Global Config Directives + +Variables for the `[global]` section of the PHP-FPM configuration, deployed as `z00-linuxfabrik-global.conf` next to the pools. + +`php__fpm_conf_emergency_restart_interval__group_var` / `php__fpm_conf_emergency_restart_interval__host_var` + +* The window `php__fpm_conf_emergency_restart_threshold__*_var` counts within. Available units: s(econds), m(inutes), h(ours), or d(ays). +* Type: String. +* Default: `'1m'` +* Deviates from the upstream default `0`: PHP-FPM needs a non-zero threshold and a non-zero interval before it reloads itself after repeated worker crashes, so leaving either at zero turns the safety net off. + +`php__fpm_conf_emergency_restart_threshold__group_var` / `php__fpm_conf_emergency_restart_threshold__host_var` + +* Reload PHP-FPM once this many workers died on `SIGSEGV` or `SIGBUS` within `php__fpm_conf_emergency_restart_interval__*_var`. A value of `0` means off. +* Type: Number. +* Default: `10` +* Deviates from the upstream default `0`: an extension or opcode cache that corrupts its workers otherwise keeps crashing them until someone notices, while a reload of the master usually restores service. PHP-FPM writes a WARNING when it triggers, so the underlying crash still surfaces in monitoring rather than being papered over. + +`php__fpm_conf_log_level__group_var` / `php__fpm_conf_log_level__host_var` + +* The log level of PHP-FPM's own error log. Possible values: `alert`, `error`, `warning`, `notice`, `debug`. +* Type: String. +* Default: `'notice'` +* Matches the upstream default, but is pinned rather than left unset: PHP-FPM keeps the unset value at zero internally, so `php-fpm -tt` dumps `log_level = unknown value` and an administrator cannot read the level that is actually in effect. Raising it to `warning` drops the start, reload and shutdown markers that make a pool restarting in a loop visible. + +Example: +```yaml +# optional +php__fpm_conf_emergency_restart_interval__host_var: '1m' +php__fpm_conf_emergency_restart_threshold__host_var: 10 +php__fpm_conf_log_level__host_var: 'notice' +``` + + ## Optional Role Variables - PHP-FPM Pool Config Directives Variables for PHP-FPM pool directives and their default values, defined and supported by this role. @@ -412,25 +464,29 @@ Variables for PHP-FPM pool directives and their default values, defined and supp * The desired minimum number of idle server processes. * Type: Number. -* Default: `5` +* Default: `10` +* Deviates from the upstream default (`5` on RedHat, `1` on Debian): against `php__fpm_pool_conf_pm_max_children__*_var` of 50 the packaged value keeps so small a warm reserve that an ordinary traffic spike exhausts it in seconds, and PHP-FPM then forks in doubling bursts and logs a `seems busy` warning every second. Each idle worker costs its own memory, so lower it again on hosts that are tight. `php__fpm_pool_conf_pm_start_servers__group_var` / `php__fpm_pool_conf_pm_start_servers__host_var` * The number of child processes created on startup. Must be greater than `php__fpm_pool_conf_pm_min_spare_servers__*_var` but less than `php__fpm_pool_conf_pm_max_spare_servers__*_var`. * Type: Number. -* Default: `5` +* Default: `10` +* Deviates from the upstream default (`5` on RedHat, `2` on Debian): the first requests after a restart otherwise arrive while the pool is still forking. `php__fpm_pool_conf_request_slowlog_timeout__group_var` / `php__fpm_pool_conf_request_slowlog_timeout__host_var` * The timeout for serving a single request after which a PHP backtrace will be dumped to the slowlog file. A value of `0` means off. Available units: s(econds, default), m(inutes), h(ours), or d(ays). * Type: Number. * Default: `0` +* Off by default on purpose. PHP-FPM collects the backtrace with `ptrace`, which SELinux denies to the `httpd_t` domain the master and its workers both run in. On an enforcing RedHat host every slow request therefore produces `ERROR: failed to ptrace(ATTACH) child N: Operation not permitted` in the error log while the slowlog stays empty, which turns a monitoring check reading that log critical without a finding. Set it on Debian, or on RedHat only together with an SELinux policy module that grants `httpd_t` the `sys_ptrace` capability. `php__fpm_pool_conf_request_terminate_timeout__group_var` / `php__fpm_pool_conf_request_terminate_timeout__host_var` * The timeout for serving a single request after which the worker process will be killed. This option should be used when the `max_execution_time` ini option does not stop script execution for some reason. A value of `0` means off. Available units: s(econds, default), m(inutes), h(ours), or d(ays). * Type: Number. -* Default: `0` +* Default: `3900` +* Deviates from the upstream default `0`: a worker blocked in a system call, on a database socket that never answers for example, holds its slot forever and the pool bleeds capacity until it is full. The value sits five minutes above the 3600 seconds Nextcloud raises `max_execution_time` to for large uploads, so PHP's own limit always fires first and this one only catches what PHP cannot stop itself. Raise it for workloads with legitimately longer requests, or set `0` to restore the previous behaviour. `php__fpm_pools__group_var` / `php__fpm_pools__host_var` @@ -465,10 +521,10 @@ Example: php__fpm_pool_conf_pm__host_var: 'dynamic' php__fpm_pool_conf_pm_max_children__host_var: 50 php__fpm_pool_conf_pm_max_spare_servers__host_var: 35 -php__fpm_pool_conf_pm_min_spare_servers__host_var: 5 -php__fpm_pool_conf_pm_start_servers__host_var: 5 -php__fpm_pool_conf_request_slowlog_timeout__host_var: '10s' -php__fpm_pool_conf_request_terminate_timeout__host_var: '60s' +php__fpm_pool_conf_pm_min_spare_servers__host_var: 10 +php__fpm_pool_conf_pm_start_servers__host_var: 10 +php__fpm_pool_conf_request_slowlog_timeout__host_var: 0 +php__fpm_pool_conf_request_terminate_timeout__host_var: '3900s' php__fpm_pools__host_var: - name: 'librenms' user: 'librenms' diff --git a/roles/php/defaults/main.yml b/roles/php/defaults/main.yml index f46d121b2..c3ae1774f 100644 --- a/roles/php/defaults/main.yml +++ b/roles/php/defaults/main.yml @@ -1,3 +1,36 @@ +php__fpm_conf_emergency_restart_interval__combined_var: '{{ + php__fpm_conf_emergency_restart_interval__host_var if (php__fpm_conf_emergency_restart_interval__host_var | string | length) else + php__fpm_conf_emergency_restart_interval__group_var if (php__fpm_conf_emergency_restart_interval__group_var | string | length) else + php__fpm_conf_emergency_restart_interval__dependent_var if (php__fpm_conf_emergency_restart_interval__dependent_var | string | length) else + php__fpm_conf_emergency_restart_interval__role_var + }}' + +php__fpm_conf_emergency_restart_interval__dependent_var: '' +php__fpm_conf_emergency_restart_interval__group_var: '' +php__fpm_conf_emergency_restart_interval__host_var: '' +php__fpm_conf_emergency_restart_interval__role_var: '1m' # upstream default: 0 +php__fpm_conf_emergency_restart_threshold__combined_var: '{{ + php__fpm_conf_emergency_restart_threshold__host_var if (php__fpm_conf_emergency_restart_threshold__host_var | string | length) else + php__fpm_conf_emergency_restart_threshold__group_var if (php__fpm_conf_emergency_restart_threshold__group_var | string | length) else + php__fpm_conf_emergency_restart_threshold__dependent_var if (php__fpm_conf_emergency_restart_threshold__dependent_var | string | length) else + php__fpm_conf_emergency_restart_threshold__role_var + }}' + +php__fpm_conf_emergency_restart_threshold__dependent_var: '' +php__fpm_conf_emergency_restart_threshold__group_var: '' +php__fpm_conf_emergency_restart_threshold__host_var: '' +php__fpm_conf_emergency_restart_threshold__role_var: 10 # upstream default: 0 +php__fpm_conf_log_level__combined_var: '{{ + php__fpm_conf_log_level__host_var if (php__fpm_conf_log_level__host_var | string | length) else + php__fpm_conf_log_level__group_var if (php__fpm_conf_log_level__group_var | string | length) else + php__fpm_conf_log_level__dependent_var if (php__fpm_conf_log_level__dependent_var | string | length) else + php__fpm_conf_log_level__role_var + }}' + +php__fpm_conf_log_level__dependent_var: '' +php__fpm_conf_log_level__group_var: '' +php__fpm_conf_log_level__host_var: '' +php__fpm_conf_log_level__role_var: 'notice' # upstream default: notice php__fpm_pool_conf_pm__combined_var: '{{ php__fpm_pool_conf_pm__host_var if (php__fpm_pool_conf_pm__host_var | string | length) else php__fpm_pool_conf_pm__group_var if (php__fpm_pool_conf_pm__group_var | string | length) else @@ -19,7 +52,7 @@ php__fpm_pool_conf_pm_max_children__combined_var: '{{ php__fpm_pool_conf_pm_max_children__dependent_var: '' php__fpm_pool_conf_pm_max_children__group_var: '' php__fpm_pool_conf_pm_max_children__host_var: '' -php__fpm_pool_conf_pm_max_children__role_var: 50 +php__fpm_pool_conf_pm_max_children__role_var: 50 # upstream default: 50 (RedHat), 5 (Debian) php__fpm_pool_conf_pm_max_spare_servers__combined_var: '{{ php__fpm_pool_conf_pm_max_spare_servers__host_var if (php__fpm_pool_conf_pm_max_spare_servers__host_var | string | length) else php__fpm_pool_conf_pm_max_spare_servers__group_var if (php__fpm_pool_conf_pm_max_spare_servers__group_var | string | length) else @@ -30,7 +63,7 @@ php__fpm_pool_conf_pm_max_spare_servers__combined_var: '{{ php__fpm_pool_conf_pm_max_spare_servers__dependent_var: '' php__fpm_pool_conf_pm_max_spare_servers__group_var: '' php__fpm_pool_conf_pm_max_spare_servers__host_var: '' -php__fpm_pool_conf_pm_max_spare_servers__role_var: 35 +php__fpm_pool_conf_pm_max_spare_servers__role_var: 35 # upstream default: 35 (RedHat), 3 (Debian) php__fpm_pool_conf_pm_min_spare_servers__combined_var: '{{ php__fpm_pool_conf_pm_min_spare_servers__host_var if (php__fpm_pool_conf_pm_min_spare_servers__host_var | string | length) else php__fpm_pool_conf_pm_min_spare_servers__group_var if (php__fpm_pool_conf_pm_min_spare_servers__group_var | string | length) else @@ -41,7 +74,7 @@ php__fpm_pool_conf_pm_min_spare_servers__combined_var: '{{ php__fpm_pool_conf_pm_min_spare_servers__dependent_var: '' php__fpm_pool_conf_pm_min_spare_servers__group_var: '' php__fpm_pool_conf_pm_min_spare_servers__host_var: '' -php__fpm_pool_conf_pm_min_spare_servers__role_var: 5 +php__fpm_pool_conf_pm_min_spare_servers__role_var: 10 # upstream default: 5 (RedHat), 1 (Debian) php__fpm_pool_conf_pm_start_servers__combined_var: '{{ php__fpm_pool_conf_pm_start_servers__host_var if (php__fpm_pool_conf_pm_start_servers__host_var | string | length) else php__fpm_pool_conf_pm_start_servers__group_var if (php__fpm_pool_conf_pm_start_servers__group_var | string | length) else @@ -52,7 +85,7 @@ php__fpm_pool_conf_pm_start_servers__combined_var: '{{ php__fpm_pool_conf_pm_start_servers__dependent_var: '' php__fpm_pool_conf_pm_start_servers__group_var: '' php__fpm_pool_conf_pm_start_servers__host_var: '' -php__fpm_pool_conf_pm_start_servers__role_var: 5 +php__fpm_pool_conf_pm_start_servers__role_var: 10 # upstream default: 5 (RedHat), 2 (Debian) php__fpm_pool_conf_request_slowlog_timeout__combined_var: '{{ php__fpm_pool_conf_request_slowlog_timeout__host_var if (php__fpm_pool_conf_request_slowlog_timeout__host_var | string | length) else php__fpm_pool_conf_request_slowlog_timeout__group_var if (php__fpm_pool_conf_request_slowlog_timeout__group_var | string | length) else @@ -63,7 +96,7 @@ php__fpm_pool_conf_request_slowlog_timeout__combined_var: '{{ php__fpm_pool_conf_request_slowlog_timeout__dependent_var: '' php__fpm_pool_conf_request_slowlog_timeout__group_var: '' php__fpm_pool_conf_request_slowlog_timeout__host_var: '' -php__fpm_pool_conf_request_slowlog_timeout__role_var: 0 +php__fpm_pool_conf_request_slowlog_timeout__role_var: 0 # upstream default: 0 php__fpm_pool_conf_request_terminate_timeout__combined_var: '{{ php__fpm_pool_conf_request_terminate_timeout__host_var if (php__fpm_pool_conf_request_terminate_timeout__host_var | string | length) else php__fpm_pool_conf_request_terminate_timeout__group_var if (php__fpm_pool_conf_request_terminate_timeout__group_var | string | length) else @@ -74,7 +107,7 @@ php__fpm_pool_conf_request_terminate_timeout__combined_var: '{{ php__fpm_pool_conf_request_terminate_timeout__dependent_var: '' php__fpm_pool_conf_request_terminate_timeout__group_var: '' php__fpm_pool_conf_request_terminate_timeout__host_var: '' -php__fpm_pool_conf_request_terminate_timeout__role_var: 0 +php__fpm_pool_conf_request_terminate_timeout__role_var: 3900 # upstream default: 0 php__fpm_pools__combined_var: '{{ ( php__fpm_pools__role_var + php__fpm_pools__dependent_var + diff --git a/roles/php/tasks/main.yml b/roles/php/tasks/main.yml index c994491fd..f5d2b6854 100644 --- a/roles/php/tasks/main.yml +++ b/roles/php/tasks/main.yml @@ -170,6 +170,31 @@ - block: + # Ownership and mode mirror what the RedHat php-fpm package ships, so this task is a + # no-op there and creates the equivalent directory on Debian, which has none. The pool + # logs are written by the workers, so the directory has to be writable by the pool + # user; directly under /var/log (root:root 0755) they are not, and PHP reports that + # failure to a stderr PHP-FPM discards. + - name: 'mkdir -p {{ __php__fpm_log_path }}' + ansible.builtin.file: + path: '{{ __php__fpm_log_path }}' + state: 'directory' + owner: '{{ __shared__apache_httpd_user }}' + group: 'root' + mode: 0o770 + + # The `[global]` section lives next to the pools instead of in php-fpm.conf, which + # belongs to the package. See the template for what that costs and why it works. + - name: 'Deploy the global config to {{ __php__fpm_pools_path }}/z00-linuxfabrik-global.conf' + ansible.builtin.template: + backup: true + src: 'etc/php-fpm.d/global.conf.j2' + dest: '{{ __php__fpm_pools_path }}/z00-linuxfabrik-global.conf' + owner: 'root' + group: 'root' + mode: 0o644 + notify: 'php: php-fpm --test; restart php-fpm' + - name: 'Remove absent pools from {{ __php__fpm_pools_path }}' ansible.builtin.file: path: '{{ __php__fpm_pools_path }}/{{ item["name"] }}.conf' @@ -200,6 +225,30 @@ - 'php:update' +- block: + + # RedHat needs nothing here: its packaged /etc/logrotate.d/php-fpm already globs + # __php__fpm_log_path. Debian's covers PHP-FPM's own error log only, so the per-pool + # logs would grow without bound. + # 'php:update' is included because the reopen helper carries the PHP version in its name. + - name: 'Deploy /etc/logrotate.d/php-fpm-pools' + ansible.builtin.template: + backup: true + src: 'etc/logrotate.d/php-fpm-pools.j2' + dest: '/etc/logrotate.d/php-fpm-pools' + owner: 'root' + group: 'root' + mode: 0o644 + + when: + - 'ansible_facts["os_family"] == "Debian"' + + tags: + - 'php' + - 'php:logrotate' + - 'php:update' + + - block: - name: 'systemctl {{ php__fpm_service_enabled | bool | ternary("enable", "disable") }} {{ __php__fpm_service_name }}' diff --git a/roles/php/templates/etc/logrotate.d/php-fpm-pools.j2 b/roles/php/templates/etc/logrotate.d/php-fpm-pools.j2 new file mode 100644 index 000000000..461bcb91f --- /dev/null +++ b/roles/php/templates/etc/logrotate.d/php-fpm-pools.j2 @@ -0,0 +1,27 @@ +# {{ ansible_managed }} +# 2026082801 + +# The per-pool logs the role points `slowlog` and `php_admin_value[error_log]` at. +# On RedHat the packaged /etc/logrotate.d/php-fpm already globs this directory, so this +# file is deployed on Debian only, where the packaged configuration covers nothing but +# PHP-FPM's own error log. +# +# No `create` directive: logrotate reuses the mode and ownership of the rotated file, +# which is what the pool user needs to keep appending. The application error log is +# opened with O_CREAT|O_APPEND per message and needs no signal, the slowlog is held open +# by the master and is reopened by the helper below. + +{{ __php__fpm_log_path }}/*.log { + compress + delaycompress + missingok + notifempty + rotate 12 + sharedscripts + weekly + postrotate + if [ -x /usr/lib/php/php{{ __php__installed_version }}-fpm-reopenlogs ]; then + /usr/lib/php/php{{ __php__installed_version }}-fpm-reopenlogs + fi + endscript +} diff --git a/roles/php/templates/etc/php-fpm.d/Debian-pool.conf.j2 b/roles/php/templates/etc/php-fpm.d/Debian-pool.conf.j2 index 3aef6674e..8f034073f 100644 --- a/roles/php/templates/etc/php-fpm.d/Debian-pool.conf.j2 +++ b/roles/php/templates/etc/php-fpm.d/Debian-pool.conf.j2 @@ -379,7 +379,11 @@ ping.response = pong ; The log file for slow requests ; Default Value: not set ; Note: slowlog is mandatory if request_slowlog_timeout is set -slowlog = log/{{ item["name"] }}-slow.log +; The relative path the Debian package suggests here expands against the global +; prefix and resolves to /usr/log/$pool-slow.log, a directory that does not exist, +; so PHP-FPM has nowhere to write the backtrace. Use the same absolute directory as +; on RedHat instead. Verified against php-fpm 8.4 on Debian 13. +slowlog = {{ __php__fpm_log_path }}/{{ item["name"] }}-slow.log ; The timeout for serving a single request after which a PHP backtrace will be ; dumped to the 'slowlog' file. A value of '0s' means 'off'. @@ -495,7 +499,11 @@ env[PATH] = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ; specified at startup with the -d argument ;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com ;php_flag[display_errors] = off -php_admin_value[error_log] = /var/log/php-fpm-{{ item["name"] }}-error.log +; Written by the workers, not by the master, so it has to live in a directory the +; pool user may create files in. Directly under /var/log (root:root 0755) PHP cannot, +; and it reports that failure to a stderr PHP-FPM discards unless +; catch_workers_output is on, so the application errors were lost without a trace. +php_admin_value[error_log] = {{ __php__fpm_log_path }}/{{ item["name"] }}-error.log php_admin_flag[log_errors] = on ;php_admin_value[memory_limit] = 32M diff --git a/roles/php/templates/etc/php-fpm.d/RedHat-pool.conf.j2 b/roles/php/templates/etc/php-fpm.d/RedHat-pool.conf.j2 index 6bb27fddd..e1e71eb09 100644 --- a/roles/php/templates/etc/php-fpm.d/RedHat-pool.conf.j2 +++ b/roles/php/templates/etc/php-fpm.d/RedHat-pool.conf.j2 @@ -329,7 +329,7 @@ ping.response = pong ; The log file for slow requests ; Default Value: not set ; Note: slowlog is mandatory if request_slowlog_timeout is set -slowlog = /var/log/php-fpm/{{ item["name"] }}-slow.log +slowlog = {{ __php__fpm_log_path }}/{{ item["name"] }}-slow.log ; The timeout for serving a single request after which a PHP backtrace will be ; dumped to the 'slowlog' file. A value of '0s' means 'off'. @@ -430,7 +430,7 @@ env[PATH] = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ; specified at startup with the -d argument ;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f www@my.domain.com ;php_flag[display_errors] = off -php_admin_value[error_log] = /var/log/php-fpm/{{ item["name"] }}-error.log +php_admin_value[error_log] = {{ __php__fpm_log_path }}/{{ item["name"] }}-error.log php_admin_flag[log_errors] = on ;php_admin_value[memory_limit] = 128M diff --git a/roles/php/templates/etc/php-fpm.d/global.conf.j2 b/roles/php/templates/etc/php-fpm.d/global.conf.j2 new file mode 100644 index 000000000..7de9d451d --- /dev/null +++ b/roles/php/templates/etc/php-fpm.d/global.conf.j2 @@ -0,0 +1,45 @@ +; {{ ansible_managed }} +; 2026082801 + +; The `[global]` section of the PHP-FPM configuration. It is deployed next to the pools +; rather than into `php-fpm.conf`, because that file belongs to the package and the +; distributions disagree on its contents. +; +; The distributions also disagree on where they place their `include=` line: RedHat reads +; the pool directory before its own `[global]`, Debian and Fedora after it. A directive +; the packaged `php-fpm.conf` sets itself (`error_log`, `pid`, `daemonize`) is therefore +; overridden again on RedHat but not on Debian, which is why none of them are set here. +; The directives below are set by no packaged `php-fpm.conf`, so they take effect +; everywhere. Verified against php-fpm 7.2 on Rocky 8, 8.4 on Debian 13 and 8.5 on +; Fedora 44. + +[global] + +; Log level. PHP-FPM reports the events an administrator has to act on (a pool that ran +; out of workers, a worker that died on a signal, a request killed by +; `request_terminate_timeout`) at WARNING, and its own start, reload and shutdown at +; NOTICE. `notice` therefore keeps all of them; anything above drops the lifecycle +; markers that make a pool restarting in a loop visible. +; +; The value is pinned even though it matches the compiled-in default, because PHP-FPM +; leaves the unset value at 0 internally and `php-fpm -tt` then dumps +; `log_level = unknown value` instead of the level that is actually in effect. +; +; Possible Values: alert, error, warning, notice, debug +; Default Value: notice +log_level = {{ php__fpm_conf_log_level__combined_var }} + +; Reload PHP-FPM once this many workers died on SIGSEGV or SIGBUS within +; `emergency_restart_interval`. Both directives have to be set, a zero in either one +; turns the mechanism off. This is a safety net against an extension or an opcode cache +; that leaves the workers in a state only a fresh master recovers from; it is not a +; substitute for fixing the crash, and PHP-FPM writes a WARNING when it triggers. +; +; Default Value: 0 +emergency_restart_threshold = {{ php__fpm_conf_emergency_restart_threshold__combined_var }} + +; The window `emergency_restart_threshold` counts within. +; Available units: s(econds), m(inutes), h(ours), or d(ays) +; +; Default Value: 0 +emergency_restart_interval = {{ php__fpm_conf_emergency_restart_interval__combined_var }} diff --git a/roles/php/vars/Debian.yml b/roles/php/vars/Debian.yml index 9921686e9..f2a4273d3 100644 --- a/roles/php/vars/Debian.yml +++ b/roles/php/vars/Debian.yml @@ -32,5 +32,6 @@ __php__conf_dest: # Debian ships no unversioned php-fpm binary and registers none with alternatives, so the config # test has to address the installed version explicitly. __php__config_test_command: 'php-fpm{{ __php__installed_version }} --test' +__php__fpm_log_path: '/var/log/php-fpm' __php__fpm_pools_path: '/etc/php/{{ __php__installed_version }}/fpm/pool.d' __php__fpm_service_name: 'php{{ __php__installed_version }}-fpm' diff --git a/roles/php/vars/RedHat.yml b/roles/php/vars/RedHat.yml index 82f0562f7..796df5283 100644 --- a/roles/php/vars/RedHat.yml +++ b/roles/php/vars/RedHat.yml @@ -8,5 +8,6 @@ __php__base_packages: __php__conf_dest: - '/etc/php.d/z00-linuxfabrik.ini' __php__config_test_command: 'php-fpm --test' +__php__fpm_log_path: '/var/log/php-fpm' __php__fpm_pools_path: '/etc/php-fpm.d' __php__fpm_service_name: 'php-fpm' diff --git a/roles/php/vars/Ubuntu.yml b/roles/php/vars/Ubuntu.yml new file mode 100644 index 000000000..6ee3a3bc1 --- /dev/null +++ b/roles/php/vars/Ubuntu.yml @@ -0,0 +1,41 @@ +# Ubuntu is loaded on top of the Debian os_family, so this is a full copy of vars/Debian.yml +# today. It exists to keep Ubuntu visible at a glance and to give later Ubuntu-specific +# drift a dedicated home instead of silently inheriting the Debian values. + +# With php__version set, install the versioned packages so the host stays on that version. The +# unversioned metapackages come from the php-defaults source and follow whatever the configured +# repo declares as its default. Within a Debian release that never moves, but with the sury repo +# enabled it moves whenever sury promotes a new PHP version, turning a plain `apt upgrade` into an +# unplanned major migration. +# composer can stay the Debian package because every phpX.Y- package provides the unversioned +# php- name. apt therefore satisfies composer's dependency chain from the versioned packages +# and installs no floating metapackage at all. +# curl, intl, mbstring and zip are in this list purely to act as those providers, not because PHP +# needs them: without one of them apt falls back to the php- metapackage, which resolves to +# whatever version sury currently defaults to and drags that version's stack onto the host. The set +# is a superset across releases (Debian 12 composer needs intl and mbstring, Debian 13 composer +# needs curl and zip). The molecule scenario asserts that no unversioned metapackage survives, so a +# future composer release that adds another one is caught by the test rather than in production. +__php__base_packages: '{{ + [ + "php" ~ php__version ~ "-cli", + "php" ~ php__version ~ "-fpm", + "php" ~ php__version ~ "-curl", + "php" ~ php__version ~ "-intl", + "php" ~ php__version ~ "-mbstring", + "php" ~ php__version ~ "-zip", + "composer", + ] if (php__version | string | length) + else ["php", "php-fpm", "composer"] + }}' + +__php__conf_dest: + - '/etc/php/{{ __php__installed_version }}/apache2/conf.d/z00-linuxfabrik.ini' + - '/etc/php/{{ __php__installed_version }}/cli/conf.d/z00-linuxfabrik.ini' + - '/etc/php/{{ __php__installed_version }}/fpm/conf.d/z00-linuxfabrik.ini' +# Debian ships no unversioned php-fpm binary and registers none with alternatives, so the config +# test has to address the installed version explicitly. +__php__config_test_command: 'php-fpm{{ __php__installed_version }} --test' +__php__fpm_log_path: '/var/log/php-fpm' +__php__fpm_pools_path: '/etc/php/{{ __php__installed_version }}/fpm/pool.d' +__php__fpm_service_name: 'php{{ __php__installed_version }}-fpm'