Skip to content

fix(daemon): bound the log, quiet the engine, and self-heal a wedged Meilisearch - #8

Merged
qdequele merged 1 commit into
mainfrom
qdequele/epic-moore-1258d0
Jul 16, 2026
Merged

fix(daemon): bound the log, quiet the engine, and self-heal a wedged Meilisearch#8
qdequele merged 1 commit into
mainfrom
qdequele/epic-moore-1258d0

Conversation

@qdequele

Copy link
Copy Markdown
Owner

What

Fixes the failure mode where an unhealthy Meilisearch turned memd.log into a disk-filling firehose (9 GB, then 42 GB in 50 minutes after a restart), which in turn kept the engine wedged.

  • New src/logging.rsSizeRotatingWriter: the daemon log is capped at 50 MB with one rotated backup (memd.log.1), so worst-case disk use is ~100 MB instead of unbounded. Oversized logs left by previous runs are rotated at startup. A LogLimiter caps repetitive warnings.
  • Meilisearch output is contained — the child now runs with --log-level WARN (no more per-request INFO logging) and its stdout/stderr are piped through the daemon's capped writer (ANSI codes stripped) instead of being inherited into launchd's unrotated StandardOutPath redirect.
  • Watchdog — a live child process is not proof of health: after 4 consecutive failed health checks (30 s apart) the daemon declares the engine wedged and restarts itself (launchd relaunches it; non-launchd re-execs). Verified end-to-end by SIGSTOPping the child: detection + recovery in ~2 minutes.
  • Crawler noise cap — per-file preparation warnings are limited to 10 per scan with a roll-up count (a wedged engine used to produce one warning per crawled file, 10k+ per cycle).
  • memd up detects the wedge — an answering MCP endpoint with an unhealthy engine is restarted instead of reported as "already running", and readiness now waits for both the MCP endpoint and the engine.

Why

Post-mortem of the 2026-07-16 incident: a disk-full event pushed Meilisearch's index scheduler into a persistent Invalid argument (os error 22) busy-loop (one ERROR line every ~3 µs). Every failing request was logged, the log filled the disk, and the full disk kept the engine failing — a self-sustaining loop that broke get_memory for every agent session. Each of the four changes above removes one link of that chain.

Notes for review

  • build_args was extracted from spawn_with_import so the engine CLI args are unit-testable.
  • Both spawn sites (daemon::serve and the engine-migration import) drain the now-piped child output; an undrained pipe would block the child once the buffer fills.
  • Live-tested on the incident machine: rotation held the log at 50 MB + tail under a real EINVAL flood, the watchdog restarted the daemon at exactly the 2-minute mark, and a 15-minute soak after recovery showed no wedge and a 23 MB log.

🤖 Generated with Claude Code

…al a wedged engine

A disk-full incident left Meilisearch in a persistent EINVAL busy-loop; with
no log rotation and the engine logging every request through launchd's
unbounded redirect, memd.log grew to 42 GB and re-filled the disk, feeding
the loop.

- new src/logging.rs: size-capped rotating writer (50 MB active + one .1
  backup, ~100 MB worst case), startup rotation for oversized logs, and a
  LogLimiter for repetitive warnings
- Meilisearch now runs with --log-level WARN and its stdout/stderr are piped
  through the daemon's capped writer (ANSI-stripped) instead of inherited
  into launchd's unrotated redirect
- daemon watchdog: 4 consecutive failed health checks (30s apart) on a live
  child means the engine is wedged; restart the daemon to clear it
- crawler: per-file preparation warnings capped at 10 per scan with a
  roll-up line for the rest
- memd up: an answering MCP endpoint with an unhealthy engine is now
  restarted instead of reported as 'already running', and readiness waits
  for both signals
@qdequele
qdequele merged commit aefe290 into main Jul 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant