Skip to content

migration-runner: shared advisory-lock guard for package migrations - #480

Merged
TheGreatAxios merged 6 commits into
mainfrom
cl-7211-migration-advisory-lock
Aug 30, 2026
Merged

TheGreatAxios merged 6 commits into
mainfrom
cl-7211-migration-advisory-lock

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

Summary

  • Adds @corbits/migration-runner, a shared applyPackageMigrations runner that bootstraps a package's schema/ledger, applies each migration transactionally, and holds a session-level pg_advisory_lock around the whole run.
  • Fixes CL-7211: six package migration runners (bench, insights, preferences, inference-catalog, evals, access-policy) checked their ledger outside the insert transaction with no lock, so two hub replicas booting together could both see a migration as unapplied and crash the loser on a duplicate-key ledger insert.
  • Converts all six packages to call applyPackageMigrations instead of their own copy of the loop; schema and ledger table names are unchanged, so this is safe for databases that already have these ledgers populated (no drops, no truncates).
  • Updates docs/package-migrations.md to describe the advisory lock and point the reference implementation at packages/access-policy/src/migrations.ts.

Test plan

  • bun run typecheck (scoped to the diff, plus targeted package runs) — clean
  • bun run lint — clean
  • bun run check:structural — clean, including check:licenses (new package has LGPL-2.1-or-later LICENSE)
  • bun test in migration-runner, bench, insights, preferences, inference-catalog, evals, access-policy — all pass (DB-gated cases skip without DATABASE_URL, as expected)
  • CI (gh pr checks) — pending, will report status

CL-7211

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Code-review pass on the shared advisory-lock guard. Linear checkboxes (CL-7211) verified against the code — all three genuinely met: the advisory lock closes the concurrent-replica race (proven by the existing "two replicas booting concurrently" test), the pattern covers all six affected packages (bench, insights, preferences, inference-catalog, evals, access-policy — confirmed each migrations.ts now delegates to applyPackageMigrations with no leftover local copy of the loop), and a test boots two concurrent runs against the same database asserting both complete without throwing (already present in packages/migration-runner/test/migration-runner.test.ts).

Hash-collision verdict: verified empirically against a live Postgres, not assumed. Ran the actual six ledger table names through hashtext() on a local instance:

access_policy_migrations     -> -1263132300
inference_catalog_migrations -> -723349846
evals_migrations             -> -636957134
preferences_migrations       -> 657047452
insights_migrations          -> 1498834098
bench_migrations             -> 1753525330

All six distinct — no collision among today's actual lock keys. But the README's original wording ("distinct packages never contend on the same lock") read as a guarantee, when the real property is narrower: hashtext() is a 32-bit hash of the table name, not the name itself, so two distinct names could in principle land on the same key. That's a liveness cost only (one package's boot blocks behind another's, never a correctness issue — each still writes its own schema and ledger), but it should be stated plainly rather than implied away. Pushed a doc fix: collision-freedom is verified for today's six names and must be re-verified (recompute hashtext() for the new name against the existing five) when a seventh package is added to this pattern.

Lock-release verdict: guaranteed on every exit path, including the one that had no test. All six call sites delegate to the single shared applyPackageMigrations, which wraps the whole run in try { ... } finally { try { if (holdsLock) unlock } finally { sql.end() } } — release is attempted whenever the lock was actually acquired, and the connection always closes even if the explicit unlock itself fails, at which point Postgres's own session-death auto-release covers it. Existing tests proved success, concurrent-replica, and crash-recovery paths; there was no test proving release on a migration that throws. Added one (releases the lock when a migration fails, so the next run isn't blocked behind it) — a broken migration fails, then a second run against the same lock completes immediately rather than hanging behind the first run's still-open connection.

Deadlock verdict: none identified. Each applyPackageMigrations call opens its own dedicated max: 1 connection and acquires exactly one advisory lock for the duration of that one call — no nested second lock acquisition, and different packages use distinct keys (confirmed above), so there's no cross-package contention, let alone a cyclic wait. Two replicas racing the same package's lock is a straightforward mutex wait, not a deadlock (only one shared resource, no reverse dependency).

Migration-path safety: confirmed non-destructive. The shared runner's ledger schema (name text PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT now()) is byte-identical to what each of the six packages' own pre-PR implementation created — CREATE TABLE IF NOT EXISTS is a no-op against an already-populated ledger with that same shape. No drops, no truncates, anywhere in the diff. Ran all 376 tests across the six rollout packages plus migration-runner against the real local dev database (not just unit-level) — green, including the packages whose ledgers already had rows from prior db-setup runs.

Fixes pushed (two commits, both docs/tests only — no production code changed, since the runner itself was already correct):

  1. Add lock-release-on-failure test to migration-runner
  2. Update docs: state the hashtext collision risk plainly in migration-runner README

Verified locally, all foreground:

  • bunx prettier --check on every touched file — clean
  • bun run lint — 0 errors (pre-existing unrelated warnings only)
  • bun test packages/bench packages/insights packages/preferences packages/inference-catalog packages/evals packages/access-policy packages/migration-runner — 376 pass, 0 fail, against a real DB
  • bun run check:structural — clean, including check:licenses (the new package's LICENSE is byte-identical to an existing packages/*/LICENSE) and check:packages
  • typecheck run locally; also left to CI given current shared-machine load

Not merging — leaving for CI and final review.

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Observation from PR #504 (CL-7241), which stacks on this branch to convert three more packages.

That PR added a per-package concurrent-boot test to each of its three packages, each verified to fail against the pre-conversion racy code and pass after. None of this PR's six packages have an equivalent — they rely solely on @corbits/migration-runner's own shared test.

That is defensible: the primitive is the thing with the interesting concurrency behaviour, and testing it once is not obviously wrong. But it does mean a package that is mis-wired to the primitive — wrong ledger table name, wrong schema, a call that silently no-ops — would not be caught by anything here. The shared test proves the lock works; it does not prove each caller uses it correctly.

Not asking for a change to this PR. Recording it so the asymmetry between the six and the three is a deliberate choice rather than an accident, and so nobody later concludes from this PR that per-package tests are unnecessary.

Also disclosed on #504: converting webhook-triggers and routines drops JSON.stringify quoting around the migration name in their failure error messages. Confirmed nothing outside migration-runner's own tests parses that string.

Covers the shared runner that will replace six identical migration
runners (bench, insights, preferences, inference-catalog, evals,
access-policy): idempotent apply, two callers racing the same
migration set, and recovery after a lock holder's connection dies
without releasing it.
A session-level pg_advisory_lock held across the check-and-insert
closes the race where two hub replicas booting together both see a
migration as unapplied and one crashes on the ledger's primary-key
violation. Six package migration runners delegate to this next.
bench, insights, preferences, inference-catalog, evals, and
access-policy each carried their own copy of the migration-apply loop
without a lock, so two hub replicas booting together could both see a
migration as unapplied and race the ledger insert. Point all six at
@corbits/migration-runner's applyPackageMigrations instead of their
own copy; schema and ledger table names are unchanged so existing
ledgers keep working.
Document the advisory lock @corbits/migration-runner adds around the
bootstrap-and-apply run, and point the self-contained-shape reference
implementation at access-policy now that all six packages route
through applyPackageMigrations instead of a hand-rolled loop.
Existing coverage proved the lock releases on success, under concurrent
replicas, and after the holding connection dies, but not on the path
this PR's own finally-block guards: a migration whose SQL fails. Proves
a second run reaches the same lock immediately instead of hanging
behind the first run's connection.
…unner README

The prior wording read as a guarantee ("distinct packages never
contend") when what's actually true is narrower: the lock key is a
32-bit hash of the ledger table name, not the name itself, so two
distinct names could in principle collide — a liveness cost (one
package's boot waits on another's), never a correctness one. Verified
against a live hashtext() that today's six ledger table names don't
collide, and says so, with the reminder to reverify for a seventh.
@TheGreatAxios
TheGreatAxios force-pushed the cl-7211-migration-advisory-lock branch from fc42d66 to a7bc4a0 Compare August 30, 2026 21:33
TheGreatAxios added a commit that referenced this pull request Aug 30, 2026
…ploy-source migrations

Two hub replicas calling these packages' migration runners at once each
see no ledger row and race the check-then-insert loop, crashing the
loser on a duplicate-key violation (CL-7241). These tests reproduce
that race directly against the current implementation and currently
fail, matching CL-7211/PR #480's own before/after pattern for the six
packages already converted to the shared advisory-lock guard.
@TheGreatAxios
TheGreatAxios merged commit 86adec2 into main Aug 30, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant