migration-runner: shared advisory-lock guard for package migrations - #480
Conversation
|
Code-review pass on the shared advisory-lock guard. Linear checkboxes (CL-7211) verified against the code — all three genuinely met: the advisory lock closes the concurrent-replica race (proven by the existing "two replicas booting concurrently" test), the pattern covers all six affected packages (bench, insights, preferences, inference-catalog, evals, access-policy — confirmed each Hash-collision verdict: verified empirically against a live Postgres, not assumed. Ran the actual six ledger table names through All six distinct — no collision among today's actual lock keys. But the README's original wording ("distinct packages never contend on the same lock") read as a guarantee, when the real property is narrower: Lock-release verdict: guaranteed on every exit path, including the one that had no test. All six call sites delegate to the single shared Deadlock verdict: none identified. Each Migration-path safety: confirmed non-destructive. The shared runner's ledger schema ( Fixes pushed (two commits, both docs/tests only — no production code changed, since the runner itself was already correct):
Verified locally, all foreground:
Not merging — leaving for CI and final review. |
|
Observation from PR #504 (CL-7241), which stacks on this branch to convert three more packages. That PR added a per-package concurrent-boot test to each of its three packages, each verified to fail against the pre-conversion racy code and pass after. None of this PR's six packages have an equivalent — they rely solely on That is defensible: the primitive is the thing with the interesting concurrency behaviour, and testing it once is not obviously wrong. But it does mean a package that is mis-wired to the primitive — wrong ledger table name, wrong schema, a call that silently no-ops — would not be caught by anything here. The shared test proves the lock works; it does not prove each caller uses it correctly. Not asking for a change to this PR. Recording it so the asymmetry between the six and the three is a deliberate choice rather than an accident, and so nobody later concludes from this PR that per-package tests are unnecessary. Also disclosed on #504: converting |
Covers the shared runner that will replace six identical migration runners (bench, insights, preferences, inference-catalog, evals, access-policy): idempotent apply, two callers racing the same migration set, and recovery after a lock holder's connection dies without releasing it.
A session-level pg_advisory_lock held across the check-and-insert closes the race where two hub replicas booting together both see a migration as unapplied and one crashes on the ledger's primary-key violation. Six package migration runners delegate to this next.
bench, insights, preferences, inference-catalog, evals, and access-policy each carried their own copy of the migration-apply loop without a lock, so two hub replicas booting together could both see a migration as unapplied and race the ledger insert. Point all six at @corbits/migration-runner's applyPackageMigrations instead of their own copy; schema and ledger table names are unchanged so existing ledgers keep working.
Document the advisory lock @corbits/migration-runner adds around the bootstrap-and-apply run, and point the self-contained-shape reference implementation at access-policy now that all six packages route through applyPackageMigrations instead of a hand-rolled loop.
Existing coverage proved the lock releases on success, under concurrent replicas, and after the holding connection dies, but not on the path this PR's own finally-block guards: a migration whose SQL fails. Proves a second run reaches the same lock immediately instead of hanging behind the first run's connection.
…unner README
The prior wording read as a guarantee ("distinct packages never
contend") when what's actually true is narrower: the lock key is a
32-bit hash of the ledger table name, not the name itself, so two
distinct names could in principle collide — a liveness cost (one
package's boot waits on another's), never a correctness one. Verified
against a live hashtext() that today's six ledger table names don't
collide, and says so, with the reminder to reverify for a seventh.
fc42d66 to
a7bc4a0
Compare
…ploy-source migrations Two hub replicas calling these packages' migration runners at once each see no ledger row and race the check-then-insert loop, crashing the loser on a duplicate-key violation (CL-7241). These tests reproduce that race directly against the current implementation and currently fail, matching CL-7211/PR #480's own before/after pattern for the six packages already converted to the shared advisory-lock guard.
Summary
@corbits/migration-runner, a sharedapplyPackageMigrationsrunner that bootstraps a package's schema/ledger, applies each migration transactionally, and holds a session-levelpg_advisory_lockaround the whole run.applyPackageMigrationsinstead of their own copy of the loop; schema and ledger table names are unchanged, so this is safe for databases that already have these ledgers populated (no drops, no truncates).packages/access-policy/src/migrations.ts.Test plan
bun run typecheck(scoped to the diff, plus targeted package runs) — cleanbun run lint— cleanbun run check:structural— clean, includingcheck:licenses(new package has LGPL-2.1-or-later LICENSE)bun testin migration-runner, bench, insights, preferences, inference-catalog, evals, access-policy — all pass (DB-gated cases skip withoutDATABASE_URL, as expected)gh pr checks) — pending, will report statusCL-7211