fix(syncer): stop O(n²) CI-expiry query from pinning vacuum xmin#186
Merged
Conversation
The superseded-row passes of syncer.expire_stale_ci_for_repo used
exclude(id__in=<grouped Max(id) subquery>), which renders as
NOT (id IN (...)). Once the subquery result outgrew work_mem,
Postgres re-executed it per outer row — O(n²). In production these
queries ran for days (one new orphan per daily beat run surviving
worker restarts), pinning the vacuum xmin horizon database-wide:
494k unremovable dead tuples bloated django_celery_results_taskresult
to 586 MB and drove the admin "Task results" changelist into Heroku
H12 timeouts.
Fixes, in layers:
- Rewrite passes 3/4 as correlated Exists("newer sibling") anti-joins,
which Postgres plans as an index-probe anti-join.
- Run all four passes' deletes in id-cursor batches of 5000 (queue
window invalidation before each batch per design doc 040 I2), keeping
transactions short and memory bounded.
- Guard the task with a per-statement timeout, new env-backed setting
SYNCER_CI_EXPIRY_STATEMENT_TIMEOUT_SECONDS (default 300s), so any
recurrence fails loudly instead of orphaning.
- Admin defense-in-depth: TaskResult changelist no longer issues
SELECT DISTINCT scans per page load (task_name filter now reads the
Celery registry; periodic_task_name/worker AllValues filters and the
duplicate task_name filter removed; show_full_result_count off).
Also adds the previously-missing SYNCER_CI_EXPIRY_* vars to
.env.example and a multi-batch convergence test (batch size 1).
Tested: core + syncer suites green against dockerized Postgres
(3 pre-existing GH-token-dependent errors excepted, pass with .env);
generated SQL verified as correlated EXISTS; ruff check/format clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The admin "Task results" page had been getting slower for weeks and finally started timing out (Heroku H12). The root cause turned out to be in syncer.expire_stale_ci_for_repo: its superseded-row passes used exclude(id__in=<grouped Max(id) subquery>), which renders as NOT (id IN (...)). Once the subquery result outgrew work_mem, Postgres re-executed it per outer row — O(n²) — and the query effectively never finished. Each daily beat run wedged on it, the daily dyno restart killed the worker but left the query running orphaned server-side, and five of these (oldest 7 days) accumulated. They pinned the vacuum xmin horizon database-wide, leaving 494k unremovable dead tuples that bloated django_celery_results_taskresult to 586 MB; the admin changelist then did several full scans over that per page load.
The fix has three layers. The superseded passes are rewritten as correlated Exists("newer sibling") anti-joins, which Postgres plans as an index-probe anti-join (generated SQL verified). All four passes now delete in id-cursor batches of 5000, invalidating attributed queue windows before each batch per design doc 040 invariant I2, so transactions stay short and memory bounded even on a large first run. And the task runs under a per-statement Postgres timeout via the new env-backed setting SYNCER_CI_EXPIRY_STATEMENT_TIMEOUT_SECONDS (default 300s, added to base.py and .env.example, along with the previously-missing SYNCER_CI_EXPIRY_PERIOD_SECONDS / SYNCER_CI_STALE_PENDING_DAYS entries), so any recurrence fails loudly instead of orphaning.
As defense-in-depth for the original symptom, the TaskResult admin changelist no longer issues per-page-load SELECT DISTINCT scans: the task_name filter now sources its choices from the Celery registry (parameter_name kept as task_name so existing links work), the periodic_task_name/worker AllValues filters and a duplicate task_name filter are dropped, and show_full_result_count is off.
🤖 Generated with Claude Code