fix(maintenance): skip a schema under concurrent DDL instead of deadlocking

[nicoloboschi]

The bug

The recurring DeadlockDetectedError in test-api. It cost four reruns on #3538 alone, and it is not test-only — the background maintenance loop hits the same race against tenant deletion and migration in production.

banks_needing_consolidation() and its three siblings snapshot the schemas owning a target table from pg_class, then query each schema in turn inside one transaction, holding AccessShareLock on two or three relations per schema until the caller commits. c7e9f1a3b5d2 already handles a schema vanishing mid-scan. The same race has a second outcome: the schema is not gone, it is being rewritten, and its DDL holds — or has queued — AccessExclusiveLock.

A queued AccessExclusiveLock blocks later AccessShareLock requests, so:

routine  holds AccessShare(memory_units)  ->  wants AccessShare(banks)
dropper  queued AccessExclusive(banks)    ->  wants AccessExclusive(memory_units)

is a cycle, and PostgreSQL breaks it by killing one side. When it picks the routine, one tenant being dropped aborts an entire maintenance pass.

Observed signature, from a real failure:

DETAIL:  Process 3873 waits for AccessShareLock on relation 23228; blocked by process 3839.
         Process 3839 waits for AccessExclusiveLock on relation 23244; blocked by process 3873.

Note this is not the DROP INDEX CONCURRENTLY storm previously blamed for these flakes — that one takes ShareUpdateExclusive. The lock modes here are AccessShare vs AccessExclusive, which is what led to the actual mechanism.

The fix

One more arm on the handler that already skips vanished schemas: give each per-schema query a short lock_timeout so it abandons the wait well before the deadlock detector runs, then skip that schema. A schema mid-DDL has nothing useful to report, and the maintenance loop runs on a ticker, so it is picked up on the next pass. Locks already held from earlier schemas stay until the caller commits — that is fine; the point is only that this routine stops waiting on the other party, which is what breaks the cycle.

Applied to all four cross-schema routines — banks_needing_consolidation, mental_models_with_cron, schemas_with_expired_rows, schemas_with_expired_operations — so the next deadlock does not simply move to a sibling.

Two implementation notes:

• No advisory lock. The earlier investigation of this flake recommended a session-scoped pg_advisory_lock. Project standards now forbid advisory locks outright, so this designs the wait out rather than locking around it — which is what that standard asks for.
• lock_timeout goes through set_config(..., is_local => true) rather than SET LOCAL: PL/pgSQL rejects the SET command inside a non-volatile function, and all four routines are STABLE. The previous value is restored before returning, so the caller's transaction is left as it was found. Only conflicting DDL can trigger it — AccessShareLock does not conflict with ordinary DML — so it never fires on a merely busy table.

Downgrade is a no-op by design: these bodies are the previous ones plus a strictly-additive skip arm, with identical signatures and results, so leaving them in place is harmless. Downgrading past b6d2f8a4c1e7 / d7b2f8a1c934 restores or drops them as those migrations define.

Test

test_banks_needing_consolidation_skips_schema_locked_by_ddl holds ACCESS EXCLUSIVE on a schema's banks table for the duration of the call, then asserts the routine still returns, skips that schema, and still reports the eligible bank in public — i.e. the scan was not aborted, only the locked schema was skipped.

The asyncio.wait_for is the actual guard. Verified failing against the pre-fix body: the routine was reverted in a live database and the test failed with TimeoutError; with the fix it passes. 106 tests pass on a from-scratch database with the migration applied.

Split out of #3538 — it was fixing that PR's own CI failures, but it is an unrelated subsystem and its own migration.

🤖 Generated with Claude Code