Parallel cache restore with a hybrid start mode, and a fast /status refresh - #9
Merged
Merged
Conversation
… mode A 754k-entry / 19 GB archive took 17 min to list and would take ~2.3 h to copy at the ~90 files/s a single rsync achieves over NFS, which is per-file latency rather than bandwidth. The restore now walks the sixteen top-level directories concurrently and copies with CACHE_RESTORE_JOBS (default 8) rsync workers that take batches round-robin, so every worker starts near the top of the newest-first list. CACHE_RESTORE_MODE=hybrid starts nginx once the newest CACHE_RESTORE_BLOCKING_MAX_BYTES (default 2g) have landed and lets the remainder continue in the background, bounding start time regardless of archive size. The restore signals this through a ready file the entrypoint waits on; blocking mode is the same mechanism with the bound set to everything. Size values accept k/m/g/t suffixes.
/status only reflected cache-restore.sh/cache-backup.sh's progress on health-monitor.sh's own poll cycle (STATUS_POLL_INTERVAL, default 5s), so a restore/backup that finished between polls could sit "stale" for up to that long. Harmless for a real multi-hour restore, but a small or empty archive can complete inside one poll gap -- which is what made the CI startup test race against it (still "running"/"0 files" at the point it checked, even though the container had already moved on to starting nginx and scheduling backups). cache-restore.sh and cache-backup.sh now ask health-monitor.sh to re-embed the state files right away (SIGUSR1) at every state transition, instead of waiting for the next tick. Guard the signal itself against `set -e`: a `&&`-chained `kill` on a dead/stale pid exits non-zero, which would otherwise abort the calling script instead of no-op'ing -- caught by the new unit test for this.
health-monitor.sh always runs as root, but cache-restore.sh and cache-backup.sh drop to the nginx user before doing anything (su-exec, needed for the NFS archive). A non-root process cannot signal a root-owned one, so every kill -USR1 from inside those scripts was failing with EPERM and being swallowed by its own error guard -- the previous commit's fix never actually fired, which is why CI was still red with the exact same "0/1 files" snapshot. Move the signal to where it can actually succeed: docker-entrypoint.sh calls it (it already sources cache-lib.sh, as root) once cache-restore.sh returns in blocking and hybrid mode, and the crontab line start_backup_scheduler installs now sources cache-lib.sh itself and calls it after cache-backup.sh returns, in the same root shell crond invoked.
The previous commit's fix didn't fix anything: kill -USR1 from a root context did reach health-monitor.sh, but its trap never actually ran before the CI check looked -- confirmed by reproducing the whole restore + refresh sequence outside Docker. In ash/dash a pending trap is not serviced until the shell's current foreground command returns on its own, and health-monitor.sh's trap fires inside a `sleep $STATUS_POLL_INTERVAL` -- so the signal just sat queued for however much of the 5s interval was left, i.e. no improvement at all, which is exactly the identical "0/1 files" snapshot CI kept reporting. cache_signal_status_refresh now drops a flag file instead of sending a signal. health-monitor.sh's main loop sleeps in 1s ticks and checks for it between ticks, so a state change lands in /status within about a second regardless of shell trap/signal semantics. This also removes the root-vs-nginx privilege problem the signal had: writing a file in CACHE_STATE_DIR needs no special privilege (docker-entrypoint.sh already chowns it to nginx), so the calls move back into cache-restore.sh/cache-backup.sh themselves, and docker-entrypoint.sh reverts to its pre-signal form. Verified end to end outside Docker: health-monitor.sh running its real loop, cache-restore.sh run to completion against a fake archive, /status showing "done" 1.5s later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CACHE_RESTORE_JOBS, default 8) instead of onefindand one rsync process serially -- matters once the archive is large enough that listing alone takes minutes.CACHE_RESTORE_MODE=hybrid: nginx starts once the newestCACHE_RESTORE_BLOCKING_MAX_BYTES(default 2g) have landed, then the rest continues in the background -- bounds startup time independent of archive size, alongside the existingblocking(default) andbackgroundmodes./statusnow reflects a restore/backup finishing within about a second instead of up to 5s.Testing:
test/cache-sync-test.sh-- 56 checks, all passing. CI green.