Close the GitHub connect grant/webhook-trigger duplication race - #508
Conversation
|
Merge-order hazard between this PR and the other one touching PR #503 (CL-7226) recomputes the content hash for all 14 PR #508 (CL-7242) adds a new migration and a schema change under Each PR computed its hash from a tree that does not contain the other's changes. They will conflict textually on the Whichever of these merges second must recompute rather than resolve by hand: then verify with Recording this rather than leaving it to be discovered as a red check after merge. |
d9525df to
ad7f432
Compare
startReviewingRepos's hasRepoGrant/hasWebhookTrigger checks are a fast path for a sequential retry (CL-7134), not a lock: two concurrent "start reviewing" calls for the same repo (a double-click, or a client retrying an in-flight request) can both read "not set up yet" before either write lands, minting two repo:<owner/name> grants and two live webhook triggers with different secrets for one repo. A duplicate grant is a permissions defect, not an untidy row -- the room's access to that repo now depends on which of two indistinguishable rows a later revoke happens to touch, so "disconnect this repo" can silently fail to actually revoke access. A duplicate trigger risks a code-review run firing twice, or the wrong secret being shown for the one GitHub actually has configured. Interchange's own `grant` table is never modified to fix this: no index, no constraint, not even a workbench-owned migration issuing DDL against it, and `mintRepoGrant` stays a plain insert -- the same shape Interchange's own POST /grants route writes. The concurrency guard lives entirely in a new workbench-owned table instead: `webhook_triggers.repo_review_lease` (own schema, no relationship to any Interchange table -- every workbench package already stores a platform id as a plain text column, never a cross-schema FK, and "repo" is a GitHub identity Interchange has no row for at all). `startReviewingRepos` acquires this lease first, per repo; only the winner runs the rest of that repo's body (hasRepoGrant/mintRepoGrant/hasWebhookTrigger/createWebhookTrigger, otherwise unchanged from CL-7134), so two concurrent callers can never both reach it for the same repo. The lease is released in a `finally` as soon as that body finishes, success or failure, so a legitimate retry is never blocked by its own prior attempt; a database-side compare-and-swap (`ON CONFLICT ... DO UPDATE ... WHERE`) also lets a lease older than two minutes be stolen, so a crash mid-work self-heals instead of permanently stranding that repo's setup (the actual claim a stranded lease makes is only ever "someone claimed this as of time T", never "the grant/trigger exist" -- CL-7213's own precedent). The paired webhook_trigger fix stays in our own schema too: a unique index on webhook_trigger(tenant_id, workflow_definition_id, name), reconciling any pre-existing duplicates first (keeps the oldest row per group). The store gains ensure() -- insert-first with onConflictDoNothing and a re-select on conflict, returning the real persisted secret -- as a method distinct from create(), which keeps its plain-insert contract so the generic management API never hands a caller someone else's trigger under a different name; a duplicate create() now 409s. apps/hub's createWebhookTrigger port binds to ensure() as defense-in-depth even though the lease already makes this call-site single-flight per repo. Regression tests, confirmed red before each fix existed: - packages/webhook-triggers/test/repo-review-lease.drizzle.test.ts: two concurrent acquire() calls settle on exactly one winner, a stale lease can be stolen, a fresh one cannot, and release() lets an immediate reacquire succeed. - packages/webhook-triggers/test/store.drizzle.test.ts: two concurrent ensure() calls settle on exactly one trigger, same id and secret. - packages/webhook-triggers/test/management-routes.test.ts: a second create() for an existing name/definition 409s. - packages/workflow-catalog/src/connect-github-setup.test.ts: two concurrent startReviewingRepos calls sharing lease-backed fakes mint exactly one grant and one trigger for the same repo -- the pure reconstruction of the audit's own (deleted) reproduction.
…ange's schema CL-7242's lease prevents new repo:<owner/name> grant duplicates, but a database that already carries duplicates from before this fix ships needs them cleaned up too -- and never so a repo ends up with fewer grants than it had, only ever down to exactly one. reconcileDuplicateRepoGrants is ordinary application code, not a migration: a plain DELETE against Interchange's own `grant` table issued through @intx/db's published `grant` export -- the exact same table object apps/hub/src/index.ts already reads and writes through for every other grant operation in this codebase. No schema is touched (no ALTER TABLE, no index, no constraint), so this creates no re-pin debt the way a DDL delta would (a schema change must be hand-reapplied at every future vendor/intx/* re-pin; a DML statement runs once and leaves nothing behind to reconcile). Scoped to exactly the shape mintRepoGrant writes (system-origin, repo:-prefixed resource), keeping the lowest-id row per (tenant, resource, action) group and deleting the rest -- safe to re-run, since a database with no duplicates left just deletes nothing. scripts/db-setup.ts calls it once per setup run, alongside the installed-package migrations it already sequences, and logs what it removed.
The lease store's own test proves the underlying compare-and-swap works in isolation; this proves startReviewingRepos itself settles on exactly one grant and one trigger when its ports are bound the way apps/hub/src/index.ts actually binds them -- a bare grant insert, a real RepoReviewLeaseStore, a real WebhookTriggerStore.ensure() -- racing two calls for the same repo the way a double-click or a client retrying an in-flight request would, against a scratch database.
ad7f432 to
92b3012
Compare
describeIfDb across the repo hand-rolled its own DATABASE_URL check, so a locally-missing database skipped every DB-gated suite in total silence — a run reported green whether the suites passed or never ran at all. That hid a real migration-list bug in PR #508 for hours. dbGate centralizes the check: it still skips quietly by default, but prints an unmissable summary naming every skipped suite, and honors E2E_REQUIRED=1 (already CI's convention for the e2e suite) by throwing instead of skipping. Every describeIfDb definition now goes through it. docker-compose.test.yml mirrors the walking-skeleton job's postgres service in ci.yml exactly (pgvector/pgvector:pg17, postgres/postgres, same healthcheck), so a local DB-gated run uses the same database CI does instead of a hand-rolled substitute.
* Add tests for dbGate, the shared DB-skip gate Covers: a configured DATABASE_URL returns describe; an absent one returns describe.skip; E2E_REQUIRED=1 turns the skip into a throw naming the suite, and stops doing so once a database is configured. * Add dbGate and a Postgres compose file matching CI describeIfDb across the repo hand-rolled its own DATABASE_URL check, so a locally-missing database skipped every DB-gated suite in total silence — a run reported green whether the suites passed or never ran at all. That hid a real migration-list bug in PR #508 for hours. dbGate centralizes the check: it still skips quietly by default, but prints an unmissable summary naming every skipped suite, and honors E2E_REQUIRED=1 (already CI's convention for the e2e suite) by throwing instead of skipping. Every describeIfDb definition now goes through it. docker-compose.test.yml mirrors the walking-skeleton job's postgres service in ci.yml exactly (pgvector/pgvector:pg17, postgres/postgres, same healthcheck), so a local DB-gated run uses the same database CI does instead of a hand-rolled substitute. * Update docs: local Postgres compose file and loud DB-skip summary Points developers at docker-compose.test.yml for a Postgres matching CI, and documents dbGate's skip summary and E2E_REQUIRED for the package/hub DB-gated suites alongside the existing e2e docs. * Add tests for check:db-gate, the hand-rolled-gate check Covers: a hand-rolled describeIfDb ternary (both the undefined and "" variants) is a violation naming the file; a suite already routed through dbGate, or with no DB gate at all, passes. * Sweep the 9 hand-rolled gates main grew, add check:db-gate Nine PRs merged since the first CL-7279 sweep, adding 9 more (10 counting a pre-existing one the original grep missed) files that still hand-rolled their own describeIfDb ternary instead of going through dbGate — proving a one-time sweep doesn't hold. Repointed all of them at dbGate; vendor/intx/hub-api's copy is left alone, since editing inside a vendored tree carries re-pin tax. check:db-gate makes the invariant self-enforcing: it fails on any `databaseUrl === undefined/"" ? describe.skip : describe` ternary outside scripts/e2e/db-gate.ts itself, in the same structural job the other checks already run in.
Summary
Fixes CL-7242 (High):
startReviewingRepos'shasRepoGrant/mintRepoGrantandhasWebhookTrigger/createWebhookTriggerchecks are a fast path for a sequential retry (CL-7134's guards), not a lock. Two truly concurrent calls for the same repo — a double-click on "Start reviewing," or a client retrying an in-flight request rather than a failed one — can both read "not set up yet" before either write lands, so both mint arepo:<owner/name>grant and both create a live webhook trigger for the same repo. Audited and verified directly.What a duplicate grant actually means for the user: not an untidy row. Two identically-scoped
repo:<owner/name>allow grants means the room's access to that repo depends on which of two indistinguishable rows a later revoke happens to touch — revoking one leaves the other standing, so "disconnect this repo" can silently fail to actually revoke access. A duplicate trigger is the paired failure: two live triggers with different secrets for the same repo can double-launch the code-review workflow on every PR, or intermittently fail signature verification if GitHub's configured webhook secret doesn't match the one the person was shown.Design note: Interchange's
granttable is never touchedAn earlier revision of this PR added a partial unique index directly to Interchange's own
granttable (first as avendor/intx/dbmigration, then as a workbench-owned migration issuing DDL against it). Both were wrong for the same reason: grants are Interchange's — we use them, we do not modify their schema, and not even a workbench-owned migration should issue DDL against a table we don't own. This revision reverts every trace of that:vendor/has zero diff againstorigin/main(check:killdates's recorded hash forvendor/intx/dbis unchanged), andmintRepoGrantis back to the exact plain insert Interchange's ownPOST /grantsroute uses.The concurrency guard now lives entirely in a new workbench-owned table:
webhook_triggers.repo_review_lease(our own schema, no relationship to any Interchange table — every workbench package already stores a platform id as a plain text column, never a cross-schema FK; seepackages/chat/src/schema.ts's own header,packages/bench/src/schema.ts's,packages/routines/src/schema.ts's, etc., all explicitly documenting "plain text identifier, not a foreign key" — I confirmed viagrep -rn ".references(" packages/ apps/that zero workbench-owned tables anywhere in this repo declare a cross-schema FK to Interchange core. A cross-schema FK from our lease table to Interchange's tenant table would itself be the kind of hard schema coupling to a re-pinned vendor tree the "don't touch their schema" rule exists to avoid — closer to the thing being banned than the thing being asked for.)startReviewingReposacquires this lease first, per repo — only the winner runs the rest of that repo's body (hasRepoGrant/mintRepoGrant/hasWebhookTrigger/createWebhookTrigger, otherwise byte-for-byte what CL-7134 left them as), so two concurrent callers can never both reach it for the same repo. The lease is released in afinallyas soon as that body finishes — success or failure — so a legitimate retry is never blocked by its own prior attempt. A database-side compare-and-swap (INSERT ... ON CONFLICT (tenant_id, repo) DO UPDATE ... WHERE leased_at < now() - interval '2 minutes') also lets a lease older than two minutes be stolen, so a process that crashes mid-work self-heals on the next attempt instead of permanently stranding that repo's setup. This is deliberately answering the CL-7213 precedent ("do not leave a record asserting something that is not true"): the lease never asserts "the grant/trigger exist" — only ever "someone claimed this repo's setup as ofleased_at" — so it can never lie about completion, only about currency, and staleness reclaim bounds how long a lie about currency can matter.The paired
webhook_triggerfix stays in our own schema and is unaffected by any of the above: a unique index onwebhook_trigger(tenant_id, workflow_definition_id, name)(our table, uncontroversial), reconciling any pre-existing duplicates first. The store gainsensure()— insert-first withonConflictDoNothingand a re-select on conflict, returning the real persisted secret — distinct fromcreate(), which keeps its plain-insert contract so the generic management API never hands a caller someone else's trigger under a differentinputTemplate; a duplicatecreate()now 409s.apps/hub'screateWebhookTriggerport binds toensure()as defense-in-depth even though the lease already makes this call-site single-flight per repo.Reconciling pre-existing duplicate grants (a database that already has them from before this fix ships) is
packages/workflow-catalog/src/reconcile-duplicate-repo-grants.ts— ordinary application code, not a migration: a one-time, always-safe-to-re-runDELETEagainst Interchange'sgranttable issued through@intx/db's publishedgrantexport, the exact same table objectapps/hub/src/index.tsalready reads and writes through for every other grant operation in this codebase. No schema is touched (noALTER TABLE, no index, no constraint) — a DDL delta would need hand-reapplying at every futurevendor/intx/*re-pin forever; this DML statement runs once and leaves nothing behind to reconcile. Keeps the lowest-id row per(tenant, resource, action)group, so a repo never ends up with fewer grants than it had.scripts/db-setup.tscalls it once per setup run and logs what it removed.Should
mintRepoGrantroute through Interchange's own grants API instead of a direct insert?Assessed, not done here — this would be a larger change than CL-7242 should carry, and I'd rather report it than scope-creep this PR:
vendor/intx/hub-api/src/routes/grants.ts'sPOST /is an HTTP route, not an importable function — there's no in-process API surface to call directly; using it would mean apps/hub calling itself over loopback HTTP or extracting a new reusable non-HTTP grant-creation function from hub-api (neither exists today).requireGrant("grant:*", "create")— the calling principal must already hold agrant:*/creategrant. Our GitHub connect flow's calling context doesn't have (or need) that today; routing through this route would mean granting the connect flow (or some internal system caller) the ability to create any grant in the tenant, which is an authorization-model change with its own blast radius, not a mechanical swap.onConflictDoNothing, nothing) — routing through it would not make duplication impossible on its own; the lease is still what would have to serialize it, so this doesn't remove any of the work in this PR, only relocates one insert.mintRepoGrantisn't the only hand-rolleddb.insert(grantTable)in this codebase —packages/chat/src/workbench-tenancy.tsandpackages/folded-runs/src/launch.tsdo the same thing. A real fix here would be systemic, not scoped to one call site.Recommend filing this as its own ticket if it's worth pursuing.
CL-7134 overlap
No contradiction, no duplication. CL-7134 (Done, PR #445) added the
hasRepoGrant/hasWebhookTriggercheck-then-act guards themselves, closing a mid-loop-failure-plus-retry gap. This PR adds the atomicity backstop that guard explicitly lacked against true concurrency. The checks stay in place, unchanged, as the fast path for a sequential retry; only the correctness guarantee under concurrency moved to the lease.Regression tests (confirmed red before each fix existed, per-file)
All against real local Postgres:
packages/webhook-triggers/test/repo-review-lease.drizzle.test.ts— two concurrentacquire()calls for the same(tenant, repo)settle on exactly one winner; a stale lease (backdated past 2 minutes) can be stolen; a fresh one cannot;release()lets an immediate reacquire succeed.packages/webhook-triggers/test/store.drizzle.test.ts— two concurrentensure()calls settle on exactly one trigger, same id and secret.packages/webhook-triggers/test/management-routes.test.ts— a secondcreate()for an existing name/definition 409s.packages/workflow-catalog/src/connect-github-setup.test.ts— two concurrentstartReviewingReposcalls sharing lease-backed fakes (synchronous check-and-set, same race shape as the real DB compare-and-swap) mint exactly one grant and one trigger for the same repo — the pure reconstruction of the audit's own deleted reproduction, manually confirmed red (removing the lease gate reproduces 2 grants) before restoring.packages/workflow-catalog/test/connect-github-setup.race.test.ts— the same reconstruction against real, production-shaped ports (the realRepoReviewLeaseStore, a bare grant insert,WebhookTriggerStore.ensure()) on a scratch database, stable across repeated runs.packages/workflow-catalog/test/reconcile-duplicate-repo-grants.test.ts— seeds duplicate + unrelated grants, confirms exactly the duplicates are removed (lowest id kept), unrelated rows untouched, and a second run is a no-op.Notes for reviewers
bun.lockpicks up 4 unrelated tool-package version bumps and one nested peer-resolution swap, already present onorigin/main's stale lockfile before this branch;bun install(needed for new devDependencies) synced that pre-existing drift. Not introduced here.scripts/checks/no-product-tenancy.ts's allowlist gains an explicit ruling forrepo_review_lease(webhook-triggers' schema file now declares 2 tables instead of 1) — required by the check, not incidental.@intx/db's published exports, the grants route) worked exactly as documented; the assessment above is a scope/authorization question, not a defect.Test plan
cd packages/webhook-triggers && bun run typecheck && bun testcd packages/workflow-catalog && bun run typecheck && bun testcd apps/hub && bun run typecheck && bun testcd vendor/intx/db && bun run typecheck && bun test(confirmed unaffected — zero diff against origin/main)bunx prettier --check .bun run lint(0 errors; pre-existing unrelated warnings only)bun run check:killdates(vendor/intx/db hash unchanged)bun run check:structural