lucie: nightly build starved the host on I/O for 4.5h and required a hardware reset

[gotha]

nix-nightly-build.service on lucie ran away on the night of 2026-08-28 and starved the whole host of I/O for 4.5 hours, taking down Nextcloud, journald and eventually the entire box. It required a hardware reset.

What happened

Aug 28 03:02:34  Starting Nightly NixOS devbox configuration build...
Aug 28 07:30:35  nix-nightly-build.service: start operation timed out. Terminating.
Aug 28 07:31:01  Main process exited, code=killed, status=15/TERM
Aug 28 07:32:31  State 'final-sigterm' timed out. Killing.
Aug 28 07:32:32  Failed with result 'timeout'.
Aug 28 07:32:32  Consumed 4h 2min 35.881s CPU time over 4h 29min 57.792s wall clock time,
                 8.1G memory peak, 748M memory swap peak,
                 55.4G read from disk, 47.7G written to disk,
                 10.7G incoming IP traffic, 40.8M outgoing IP traffic

For comparison, every previous run in the journal finished in 1-5 minutes (worst case ~40 min on Aug 24 and Aug 26).

Cause

nix flake update picked up a nixpkgs bump that invalidated the CUDA 12.9 closure. The build log shows it downloading and building the CUDA redistributables:

Aug 28 05:05:12  building '/nix/store/dvxz1p7pnpqm3ihvs47vxb9wp1abl0fz-cuda12.9-cuda_nvdisasm-12.9.88.drv'...
Aug 28 05:05:12  building '/nix/store/j0nmkrar6mghrlylrkgbp9rzp37xqbml-cuda12.9-cuda_nvml_dev-12.9.79.drv'...
Aug 28 05:05:13  building '/nix/store/524vzfhim1nqzajpq7awa0bl597m6c3y-cuda12.9-cuda_nvprune-12.9.82.drv'...
Aug 28 05:05:30  building '/nix/store/j2c2336p9k3iy7qddhcr5pydpp7gglhb-cuda12.9-cuda_nvcc-12.9.86.drv'...
Aug 28 05:06:32  building '/nix/store/xnv50bgq97ahnzw8j2capsx7d6m9yfg7-cuda12.9-cuda_nvrtc-12.9.86.drv'...
Aug 28 05:07:04  building '/nix/store/44wysgg5ir8pkgjl6c37ag27589vyhfm-cuda12.9-cuda_sanitizer_api-12.9.79.drv'...

After 05:07:04 the service logged nothing for 2h23m — that is when the host was thrashing.

47.7 GB of writes onto a root filesystem that is at 96% (/dev/nvme0n1p1 3.6T 3.3T 150G 96% /) was enough to stall the ext4 journal outright.

Why the existing guards did not help

hosts/lucie/nightly-build.nix sets Nice=19 and IOSchedulingClass=idle, but neither limits the thing that actually broke:

1. IOSchedulingClass = "idle" is a no-op on this host. /sys/block/nvme0n1/queue/scheduler reads [none] mq-deadline kyber. ionice scheduling classes are only honoured by BFQ (and the old CFQ). Under none the build got zero I/O throttling.
2. Nice = 19 only affects CPU. It does nothing about dirty page pressure. With vm.dirty_ratio = 20 on 62 GB of RAM the build could pin ~12 GB of dirty pages, repeatedly.
3. No memory limit. 8.1 GB peak + page cache pressure triggered the global OOM killer.
4. TimeoutStartSec = "4h" is too generous. It did fire, but only at 07:30 — the host had already been unusable since ~03:20. A build that normally takes 5 minutes does not need a 4 hour budget.

Collateral damage

Time	Event
03:02:34	nightly build starts
03:18:30	nextcloud-cron.service: condition operation timed out. Terminating.
03:22:23	phpfpm-nextcloud.service: Watchdog timeout (limit 15s)! — all 20 workers SIGABRT'd. Nextcloud down from here on
03:22:49	coredump storm: systemd-coredump.socket: Too many incoming connections (9), over core_pipe_limit, skipping core dump
05:04:44	global OOM killer fires, kills chromium
07:31:09	INFO: task jbd2/nvme0n1p1-:302 blocked for more than 245 seconds in jbd2_journal_wait_updates — ext4 journal stopped committing. journald, nix, transmission, Plex, chromium all pile up in ext4_dirty_inode
07:31:09	systemd-journald SIGABRT'd by its own watchdog; journal at 3.9G, max 4G, 46.6M free, replayed a 4 hour backlog on restart
07:46:15	hardware reset

No hardware fault involved: no NVMe errors, no controller resets, no EXT4-fs error in the kernel log. This was pure write-pressure starvation.

Recurrence

The build was killed at its timeout, so the CUDA closure was never completed or committed. The timer fires again tonight at 03:00 and will redo the same work.

Proposed fix

In hosts/lucie/nightly-build.nix:

serviceConfig = {
  # ...
  MemoryHigh = "4G";        # throttle rather than OOM the host
  MemoryMax = "8G";
  TimeoutStartSec = "1h";   # 4h let it wedge the box overnight
};

IOSchedulingClass = "idle" should either be dropped or paired with switching nvme0n1 to BFQ, otherwise it is misleading dead config.

The knob that actually bites under the none scheduler is capping dirty pages globally, e.g. vm.dirty_bytes = 2147483648 (2 GB) in place of the 12 GB that vm.dirty_ratio = 20 currently permits. That alone would have kept jbd2 committing while the build ran.

Worth considering separately: root is at 96% and /mnt/storage at 99%, which makes ext4 allocation under heavy write markedly worse.

[gotha]

Opened #15 with a fix.

Your diagnosis held up on all four points; I checked each before changing anything. Two notes where I went a bit further than the proposal:

IOSchedulingClass. Dropped rather than paired with BFQ. Since the caps that actually work are available without switching the elevator, IOWriteBandwidthMax = / 150M and IOReadBandwidthMax = / 300M get the containment without paying BFQ's IOPS cost on an NVMe that also serves Nextcloud and Plex. blk-throttle runs above the scheduler, so these hold under none.

Memory limits. Went to MemoryHigh = 8G / MemoryMax = 12G rather than 4G/8G. The run peaked at 8.1 G, so an 8 G hard cap would kill a build that was otherwise legitimate, and killing it nightly reproduces the recurrence problem in a quieter form. The important part is not the specific number but that the kill happens in the cgroup instead of the global OOM killer picking a victim. MemoryHigh also does more than throttling: capping the cgroup's memory forces writeback of its own dirty page cache, so it works on the journal stall from the unit side too, alongside vm.dirty_bytes.

vm.dirty_bytes = 2G and vm.dirty_background_bytes = 512M are in as you described.

On the 96% root, which turned out to have three distinct contributors:

• nixos-rebuild build leaves a result symlink, and nix registers that as an indirect GC root — so the 7-day retention was pinning seven complete system closures, CUDA and ollama included. Cut to 3 days.
• Added min-free/max-free so the daemon collects garbage mid-build instead of running the filesystem to zero between weekly GC runs.
• Added a 150 G preflight check that fails the unit. A speculative prebuild is never worth the last of the root filesystem, and failing loudly makes systemctl --failed the disk-space alarm — there isn't one right now, since the observability stack has no Prometheus exporters to hang an alert on.

Two things I left alone. /mnt/storage at 99% is untouched. More significantly, none of this shrinks the closure — cudaPackages.cudatoolkit in systemPackages and whisper-cpp.override { cudaSupport = true; } are what make a CUDA bump this expensive, and the override in particular guarantees a cache miss on a package that drags in the whole toolchain. Worth its own issue if nvcc isn't needed directly on this host.

One thing worth flagging about the recurrence: with the caps in place I expect the build to now complete rather than be killed, since 47.7 GB at the 150 MB/s floor is roughly 5 minutes of write and the 10.7 GB download becomes the bottleneck. That's the good outcome, but it does mean the first run after deploy will still be a long one — just a contained one.

Not deployed yet. The cgroup and sysctl behaviour is only really confirmed by watching a run, so I'd keep this issue open until one completes cleanly.