Skip to content

Close the GitHub connect grant/webhook-trigger duplication race - #508

Merged
TheGreatAxios merged 3 commits into
mainfrom
cl-7242-github-connect-race
Aug 31, 2026
Merged

TheGreatAxios merged 3 commits into
mainfrom
cl-7242-github-connect-race

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes CL-7242 (High): startReviewingRepos's hasRepoGrant/mintRepoGrant and hasWebhookTrigger/createWebhookTrigger checks are a fast path for a sequential retry (CL-7134's guards), not a lock. Two truly concurrent calls for the same repo — a double-click on "Start reviewing," or a client retrying an in-flight request rather than a failed one — can both read "not set up yet" before either write lands, so both mint a repo:<owner/name> grant and both create a live webhook trigger for the same repo. Audited and verified directly.

What a duplicate grant actually means for the user: not an untidy row. Two identically-scoped repo:<owner/name> allow grants means the room's access to that repo depends on which of two indistinguishable rows a later revoke happens to touch — revoking one leaves the other standing, so "disconnect this repo" can silently fail to actually revoke access. A duplicate trigger is the paired failure: two live triggers with different secrets for the same repo can double-launch the code-review workflow on every PR, or intermittently fail signature verification if GitHub's configured webhook secret doesn't match the one the person was shown.

Design note: Interchange's grant table is never touched

An earlier revision of this PR added a partial unique index directly to Interchange's own grant table (first as a vendor/intx/db migration, then as a workbench-owned migration issuing DDL against it). Both were wrong for the same reason: grants are Interchange's — we use them, we do not modify their schema, and not even a workbench-owned migration should issue DDL against a table we don't own. This revision reverts every trace of that: vendor/ has zero diff against origin/main (check:killdates's recorded hash for vendor/intx/db is unchanged), and mintRepoGrant is back to the exact plain insert Interchange's own POST /grants route uses.

The concurrency guard now lives entirely in a new workbench-owned table: webhook_triggers.repo_review_lease (our own schema, no relationship to any Interchange table — every workbench package already stores a platform id as a plain text column, never a cross-schema FK; see packages/chat/src/schema.ts's own header, packages/bench/src/schema.ts's, packages/routines/src/schema.ts's, etc., all explicitly documenting "plain text identifier, not a foreign key" — I confirmed via grep -rn ".references(" packages/ apps/ that zero workbench-owned tables anywhere in this repo declare a cross-schema FK to Interchange core. A cross-schema FK from our lease table to Interchange's tenant table would itself be the kind of hard schema coupling to a re-pinned vendor tree the "don't touch their schema" rule exists to avoid — closer to the thing being banned than the thing being asked for.)

startReviewingRepos acquires this lease first, per repo — only the winner runs the rest of that repo's body (hasRepoGrant/mintRepoGrant/hasWebhookTrigger/createWebhookTrigger, otherwise byte-for-byte what CL-7134 left them as), so two concurrent callers can never both reach it for the same repo. The lease is released in a finally as soon as that body finishes — success or failure — so a legitimate retry is never blocked by its own prior attempt. A database-side compare-and-swap (INSERT ... ON CONFLICT (tenant_id, repo) DO UPDATE ... WHERE leased_at < now() - interval '2 minutes') also lets a lease older than two minutes be stolen, so a process that crashes mid-work self-heals on the next attempt instead of permanently stranding that repo's setup. This is deliberately answering the CL-7213 precedent ("do not leave a record asserting something that is not true"): the lease never asserts "the grant/trigger exist" — only ever "someone claimed this repo's setup as of leased_at" — so it can never lie about completion, only about currency, and staleness reclaim bounds how long a lie about currency can matter.

The paired webhook_trigger fix stays in our own schema and is unaffected by any of the above: a unique index on webhook_trigger(tenant_id, workflow_definition_id, name) (our table, uncontroversial), reconciling any pre-existing duplicates first. The store gains ensure() — insert-first with onConflictDoNothing and a re-select on conflict, returning the real persisted secret — distinct from create(), which keeps its plain-insert contract so the generic management API never hands a caller someone else's trigger under a different inputTemplate; a duplicate create() now 409s. apps/hub's createWebhookTrigger port binds to ensure() as defense-in-depth even though the lease already makes this call-site single-flight per repo.

Reconciling pre-existing duplicate grants (a database that already has them from before this fix ships) is packages/workflow-catalog/src/reconcile-duplicate-repo-grants.ts — ordinary application code, not a migration: a one-time, always-safe-to-re-run DELETE against Interchange's grant table issued through @intx/db's published grant export, the exact same table object apps/hub/src/index.ts already reads and writes through for every other grant operation in this codebase. No schema is touched (no ALTER TABLE, no index, no constraint) — a DDL delta would need hand-reapplying at every future vendor/intx/* re-pin forever; this DML statement runs once and leaves nothing behind to reconcile. Keeps the lowest-id row per (tenant, resource, action) group, so a repo never ends up with fewer grants than it had. scripts/db-setup.ts calls it once per setup run and logs what it removed.

Should mintRepoGrant route through Interchange's own grants API instead of a direct insert?

Assessed, not done here — this would be a larger change than CL-7242 should carry, and I'd rather report it than scope-creep this PR:

  • vendor/intx/hub-api/src/routes/grants.ts's POST / is an HTTP route, not an importable function — there's no in-process API surface to call directly; using it would mean apps/hub calling itself over loopback HTTP or extracting a new reusable non-HTTP grant-creation function from hub-api (neither exists today).
  • It's gated by requireGrant("grant:*", "create") — the calling principal must already hold a grant:*/create grant. Our GitHub connect flow's calling context doesn't have (or need) that today; routing through this route would mean granting the connect flow (or some internal system caller) the ability to create any grant in the tenant, which is an authorization-model change with its own blast radius, not a mechanical swap.
  • Its handler does no conflict handling of its own (no onConflictDoNothing, nothing) — routing through it would not make duplication impossible on its own; the lease is still what would have to serialize it, so this doesn't remove any of the work in this PR, only relocates one insert.
  • mintRepoGrant isn't the only hand-rolled db.insert(grantTable) in this codebase — packages/chat/src/workbench-tenancy.ts and packages/folded-runs/src/launch.ts do the same thing. A real fix here would be systemic, not scoped to one call site.

Recommend filing this as its own ticket if it's worth pursuing.

CL-7134 overlap

No contradiction, no duplication. CL-7134 (Done, PR #445) added the hasRepoGrant/hasWebhookTrigger check-then-act guards themselves, closing a mid-loop-failure-plus-retry gap. This PR adds the atomicity backstop that guard explicitly lacked against true concurrency. The checks stay in place, unchanged, as the fast path for a sequential retry; only the correctness guarantee under concurrency moved to the lease.

Regression tests (confirmed red before each fix existed, per-file)

All against real local Postgres:

  • packages/webhook-triggers/test/repo-review-lease.drizzle.test.ts — two concurrent acquire() calls for the same (tenant, repo) settle on exactly one winner; a stale lease (backdated past 2 minutes) can be stolen; a fresh one cannot; release() lets an immediate reacquire succeed.
  • packages/webhook-triggers/test/store.drizzle.test.ts — two concurrent ensure() calls settle on exactly one trigger, same id and secret.
  • packages/webhook-triggers/test/management-routes.test.ts — a second create() for an existing name/definition 409s.
  • packages/workflow-catalog/src/connect-github-setup.test.ts — two concurrent startReviewingRepos calls sharing lease-backed fakes (synchronous check-and-set, same race shape as the real DB compare-and-swap) mint exactly one grant and one trigger for the same repo — the pure reconstruction of the audit's own deleted reproduction, manually confirmed red (removing the lease gate reproduces 2 grants) before restoring.
  • packages/workflow-catalog/test/connect-github-setup.race.test.ts — the same reconstruction against real, production-shaped ports (the real RepoReviewLeaseStore, a bare grant insert, WebhookTriggerStore.ensure()) on a scratch database, stable across repeated runs.
  • packages/workflow-catalog/test/reconcile-duplicate-repo-grants.test.ts — seeds duplicate + unrelated grants, confirms exactly the duplicates are removed (lowest id kept), unrelated rows untouched, and a second run is a no-op.

Notes for reviewers

  • bun.lock picks up 4 unrelated tool-package version bumps and one nested peer-resolution swap, already present on origin/main's stale lockfile before this branch; bun install (needed for new devDependencies) synced that pre-existing drift. Not introduced here.
  • scripts/checks/no-product-tenancy.ts's allowlist gains an explicit ruling for repo_review_lease (webhook-triggers' schema file now declares 2 tables instead of 1) — required by the check, not incidental.
  • No Interchange defects to report — Interchange's own API surface (@intx/db's published exports, the grants route) worked exactly as documented; the assessment above is a scope/authorization question, not a defect.

Test plan

  • cd packages/webhook-triggers && bun run typecheck && bun test
  • cd packages/workflow-catalog && bun run typecheck && bun test
  • cd apps/hub && bun run typecheck && bun test
  • cd vendor/intx/db && bun run typecheck && bun test (confirmed unaffected — zero diff against origin/main)
  • bunx prettier --check .
  • bun run lint (0 errors; pre-existing unrelated warnings only)
  • bun run check:killdates (vendor/intx/db hash unchanged)
  • bun run check:structural
  • CI (pushed, not polling — do not merge)

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Merge-order hazard between this PR and the other one touching scripts/checks/kill-dates.txt.

PR #503 (CL-7226) recomputes the content hash for all 14 vendor/intx/* rows, because it fixed hashDirectory to exclude dist/ — the previously recorded hashes were computed on a tree where local build output existed, and a fresh CI checkout has none, so 14 rows mismatched.

PR #508 (CL-7242) adds a new migration and a schema change under vendor/intx/db, which changes that package's tree content and therefore its hash.

Each PR computed its hash from a tree that does not contain the other's changes. They will conflict textually on the vendor/intx/db row, and — more importantly — resolving that conflict by picking either side's literal hash will be wrong, because the correct post-merge hash is of a tree containing both the dist/ exclusion fix and the new migration.

Whichever of these merges second must recompute rather than resolve by hand:

bun -e 'import { hashDirectory } from "./scripts/checks/lib/tree-hash"; console.log(hashDirectory("vendor/intx/db"));'

then verify with bun run check:killdates. Do this on a tree with no dist/ present, or after #503's exclusion fix has landed — otherwise the value is environment-dependent, which is the exact bug #503 fixed.

Recording this rather than leaving it to be discovered as a red check after merge.

startReviewingRepos's hasRepoGrant/hasWebhookTrigger checks are a fast
path for a sequential retry (CL-7134), not a lock: two concurrent
"start reviewing" calls for the same repo (a double-click, or a client
retrying an in-flight request) can both read "not set up yet" before
either write lands, minting two repo:<owner/name> grants and two live
webhook triggers with different secrets for one repo. A duplicate
grant is a permissions defect, not an untidy row -- the room's access
to that repo now depends on which of two indistinguishable rows a
later revoke happens to touch, so "disconnect this repo" can silently
fail to actually revoke access. A duplicate trigger risks a
code-review run firing twice, or the wrong secret being shown for the
one GitHub actually has configured.

Interchange's own `grant` table is never modified to fix this: no
index, no constraint, not even a workbench-owned migration issuing DDL
against it, and `mintRepoGrant` stays a plain insert -- the same shape
Interchange's own POST /grants route writes. The concurrency guard
lives entirely in a new workbench-owned table instead:
`webhook_triggers.repo_review_lease` (own schema, no relationship to
any Interchange table -- every workbench package already stores a
platform id as a plain text column, never a cross-schema FK, and
"repo" is a GitHub identity Interchange has no row for at all).
`startReviewingRepos` acquires this lease first, per repo; only the
winner runs the rest of that repo's body
(hasRepoGrant/mintRepoGrant/hasWebhookTrigger/createWebhookTrigger,
otherwise unchanged from CL-7134), so two concurrent callers can never
both reach it for the same repo. The lease is released in a `finally`
as soon as that body finishes, success or failure, so a legitimate
retry is never blocked by its own prior attempt; a database-side
compare-and-swap (`ON CONFLICT ... DO UPDATE ... WHERE`) also lets a
lease older than two minutes be stolen, so a crash mid-work self-heals
instead of permanently stranding that repo's setup (the actual claim
a stranded lease makes is only ever "someone claimed this as of time
T", never "the grant/trigger exist" -- CL-7213's own precedent).

The paired webhook_trigger fix stays in our own schema too: a unique
index on webhook_trigger(tenant_id, workflow_definition_id, name),
reconciling any pre-existing duplicates first (keeps the oldest row
per group). The store gains ensure() -- insert-first with
onConflictDoNothing and a re-select on conflict, returning the real
persisted secret -- as a method distinct from create(), which keeps
its plain-insert contract so the generic management API never hands a
caller someone else's trigger under a different name; a duplicate
create() now 409s. apps/hub's createWebhookTrigger port binds to
ensure() as defense-in-depth even though the lease already makes this
call-site single-flight per repo.

Regression tests, confirmed red before each fix existed:
- packages/webhook-triggers/test/repo-review-lease.drizzle.test.ts:
  two concurrent acquire() calls settle on exactly one winner, a stale
  lease can be stolen, a fresh one cannot, and release() lets an
  immediate reacquire succeed.
- packages/webhook-triggers/test/store.drizzle.test.ts: two concurrent
  ensure() calls settle on exactly one trigger, same id and secret.
- packages/webhook-triggers/test/management-routes.test.ts: a second
  create() for an existing name/definition 409s.
- packages/workflow-catalog/src/connect-github-setup.test.ts: two
  concurrent startReviewingRepos calls sharing lease-backed fakes mint
  exactly one grant and one trigger for the same repo -- the pure
  reconstruction of the audit's own (deleted) reproduction.
…ange's schema

CL-7242's lease prevents new repo:<owner/name> grant duplicates, but a
database that already carries duplicates from before this fix ships
needs them cleaned up too -- and never so a repo ends up with fewer
grants than it had, only ever down to exactly one.

reconcileDuplicateRepoGrants is ordinary application code, not a
migration: a plain DELETE against Interchange's own `grant` table
issued through @intx/db's published `grant` export -- the exact same
table object apps/hub/src/index.ts already reads and writes through
for every other grant operation in this codebase. No schema is
touched (no ALTER TABLE, no index, no constraint), so this creates no
re-pin debt the way a DDL delta would (a schema change must be
hand-reapplied at every future vendor/intx/* re-pin; a DML statement
runs once and leaves nothing behind to reconcile). Scoped to exactly
the shape mintRepoGrant writes (system-origin, repo:-prefixed
resource), keeping the lowest-id row per (tenant, resource, action)
group and deleting the rest -- safe to re-run, since a database with
no duplicates left just deletes nothing. scripts/db-setup.ts calls it
once per setup run, alongside the installed-package migrations it
already sequences, and logs what it removed.
The lease store's own test proves the underlying compare-and-swap
works in isolation; this proves startReviewingRepos itself settles on
exactly one grant and one trigger when its ports are bound the way
apps/hub/src/index.ts actually binds them -- a bare grant insert, a
real RepoReviewLeaseStore, a real WebhookTriggerStore.ensure() --
racing two calls for the same repo the way a double-click or a client
retrying an in-flight request would, against a scratch database.
@TheGreatAxios
TheGreatAxios force-pushed the cl-7242-github-connect-race branch from ad7f432 to 92b3012 Compare August 31, 2026 02:54
@TheGreatAxios
TheGreatAxios merged commit 4a4122f into main Aug 31, 2026
7 checks passed
TheGreatAxios added a commit that referenced this pull request Aug 31, 2026
describeIfDb across the repo hand-rolled its own DATABASE_URL check,
so a locally-missing database skipped every DB-gated suite in total
silence — a run reported green whether the suites passed or never ran
at all. That hid a real migration-list bug in PR #508 for hours.

dbGate centralizes the check: it still skips quietly by default, but
prints an unmissable summary naming every skipped suite, and honors
E2E_REQUIRED=1 (already CI's convention for the e2e suite) by throwing
instead of skipping. Every describeIfDb definition now goes through it.

docker-compose.test.yml mirrors the walking-skeleton job's postgres
service in ci.yml exactly (pgvector/pgvector:pg17, postgres/postgres,
same healthcheck), so a local DB-gated run uses the same database CI
does instead of a hand-rolled substitute.
TheGreatAxios added a commit that referenced this pull request Aug 31, 2026
* Add tests for dbGate, the shared DB-skip gate

Covers: a configured DATABASE_URL returns describe; an absent one
returns describe.skip; E2E_REQUIRED=1 turns the skip into a throw
naming the suite, and stops doing so once a database is configured.

* Add dbGate and a Postgres compose file matching CI

describeIfDb across the repo hand-rolled its own DATABASE_URL check,
so a locally-missing database skipped every DB-gated suite in total
silence — a run reported green whether the suites passed or never ran
at all. That hid a real migration-list bug in PR #508 for hours.

dbGate centralizes the check: it still skips quietly by default, but
prints an unmissable summary naming every skipped suite, and honors
E2E_REQUIRED=1 (already CI's convention for the e2e suite) by throwing
instead of skipping. Every describeIfDb definition now goes through it.

docker-compose.test.yml mirrors the walking-skeleton job's postgres
service in ci.yml exactly (pgvector/pgvector:pg17, postgres/postgres,
same healthcheck), so a local DB-gated run uses the same database CI
does instead of a hand-rolled substitute.

* Update docs: local Postgres compose file and loud DB-skip summary

Points developers at docker-compose.test.yml for a Postgres matching
CI, and documents dbGate's skip summary and E2E_REQUIRED for the
package/hub DB-gated suites alongside the existing e2e docs.

* Add tests for check:db-gate, the hand-rolled-gate check

Covers: a hand-rolled describeIfDb ternary (both the undefined and ""
variants) is a violation naming the file; a suite already routed
through dbGate, or with no DB gate at all, passes.

* Sweep the 9 hand-rolled gates main grew, add check:db-gate

Nine PRs merged since the first CL-7279 sweep, adding 9 more (10
counting a pre-existing one the original grep missed) files that
still hand-rolled their own describeIfDb ternary instead of going
through dbGate — proving a one-time sweep doesn't hold. Repointed all
of them at dbGate; vendor/intx/hub-api's copy is left alone, since
editing inside a vendored tree carries re-pin tax.

check:db-gate makes the invariant self-enforcing: it fails on any
`databaseUrl === undefined/"" ? describe.skip : describe` ternary
outside scripts/e2e/db-gate.ts itself, in the same structural job the
other checks already run in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant