fix(daemon): bound the log, quiet the engine, and self-heal a wedged Meilisearch - #8
Merged
Merged
Conversation
…al a wedged engine A disk-full incident left Meilisearch in a persistent EINVAL busy-loop; with no log rotation and the engine logging every request through launchd's unbounded redirect, memd.log grew to 42 GB and re-filled the disk, feeding the loop. - new src/logging.rs: size-capped rotating writer (50 MB active + one .1 backup, ~100 MB worst case), startup rotation for oversized logs, and a LogLimiter for repetitive warnings - Meilisearch now runs with --log-level WARN and its stdout/stderr are piped through the daemon's capped writer (ANSI-stripped) instead of inherited into launchd's unrotated redirect - daemon watchdog: 4 consecutive failed health checks (30s apart) on a live child means the engine is wedged; restart the daemon to clear it - crawler: per-file preparation warnings capped at 10 per scan with a roll-up line for the rest - memd up: an answering MCP endpoint with an unhealthy engine is now restarted instead of reported as 'already running', and readiness waits for both signals
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the failure mode where an unhealthy Meilisearch turned
memd.loginto a disk-filling firehose (9 GB, then 42 GB in 50 minutes after a restart), which in turn kept the engine wedged.src/logging.rs—SizeRotatingWriter: the daemon log is capped at 50 MB with one rotated backup (memd.log.1), so worst-case disk use is ~100 MB instead of unbounded. Oversized logs left by previous runs are rotated at startup. ALogLimitercaps repetitive warnings.--log-level WARN(no more per-request INFO logging) and its stdout/stderr are piped through the daemon's capped writer (ANSI codes stripped) instead of being inherited into launchd's unrotatedStandardOutPathredirect.memd updetects the wedge — an answering MCP endpoint with an unhealthy engine is restarted instead of reported as "already running", and readiness now waits for both the MCP endpoint and the engine.Why
Post-mortem of the 2026-07-16 incident: a disk-full event pushed Meilisearch's index scheduler into a persistent
Invalid argument (os error 22)busy-loop (one ERROR line every ~3 µs). Every failing request was logged, the log filled the disk, and the full disk kept the engine failing — a self-sustaining loop that brokeget_memoryfor every agent session. Each of the four changes above removes one link of that chain.Notes for review
build_argswas extracted fromspawn_with_importso the engine CLI args are unit-testable.daemon::serveand the engine-migration import) drain the now-piped child output; an undrained pipe would block the child once the buffer fills.🤖 Generated with Claude Code