Skip to content

feat(store): generalise schema to multi-chain - #232

Open
Manuel1234477 wants to merge 1 commit into
Octo-Protocol-org:mainfrom
Manuel1234477:feat/multi-chain-schema
Open

feat(store): generalise schema to multi-chain#232
Manuel1234477 wants to merge 1 commit into
Octo-Protocol-org:mainfrom
Manuel1234477:feat/multi-chain-schema

Conversation

@Manuel1234477

Copy link
Copy Markdown

Summary

Closes #214.

Generalises the schema from Stellar-only to multi-chain: a chains registry (CAIP-2-shaped
chain_id slugs), plus chain_id threaded through wallets, addresses, and transactions,
and a chain-scoped replacement for the anti-double-credit unique index
(uq_tx_onchainuq_tx_onchain_chain on (chain_id, tx_hash, operation_index)).

An EVM tx hash is not globally unique across chains — the same (tx_hash, operation_index) pair
can legitimately occur on two different chains. The old two-column index would have silently
rejected the second chain's deposit as a duplicate (a dropped-deposit bug). This is fixed by
re-scoping the guard to include chain_id, built CONCURRENTLY and swapped in only after it's
confirmed live — the anti-double-credit invariant is never unenforced, even momentarily.

Backward-compatibility strategy (the main reviewable decision)

Chosen: keep every legacy column, add generic ones alongside (option (b) in the issue).

network, stellar_tx_hash, and muxed_address all still exist unchanged; chain_id, tx_hash,
and deposit_address are new columns kept in lockstep by every Store write path
(octo_store::stellar_chain_id_for_network is the single place that derives one from the other).

Rejected the two-release rename-behind-a-view approach because: this crate's queries lean on
SELECT * / RETURNING * with sqlx::FromRow, which a compatibility view or generated column
complicates for comparatively little benefit at this schema's size; and keeping both names live
throughout the transition is simpler to reason about than sequencing two coordinated releases
against one running production binary. The cost is duplicated data (tx_hash == stellar_tx_hash
for every Stellar row today) — acceptable short-term; a follow-up migration can drop the legacy
columns once every caller has moved onto the octo-chain adapter (#213/#215/#220/#223).

Migration shape (online-safe)

Additive → backfill (batched, resumable) → NOT VALID check → VALIDATE CONSTRAINT
SET NOT NULL → new indexes built CONCURRENTLY → old index dropped only once the new one is
confirmed live. Full step-by-step rationale is in docs/architecture.md ("Data model:
multi-chain (#214)"), including an ER diagram.

No step takes a long ACCESS EXCLUSIVE lock on transactions:

  • ADD COLUMN ... NULL (0022) is metadata-only on PG11+.
  • The batched backfills (0024/0025) commit every 5,000 rows and are resumable.
  • ADD CONSTRAINT ... NOT VALID (0026) registers instantly; the actual scan runs under
    VALIDATE CONSTRAINT's SHARE UPDATE EXCLUSIVE (0027), which doesn't block reads/writes.
  • SET NOT NULL (0028) reuses the already-validated CHECK on PG12+ and skips its own scan.
  • CREATE INDEX CONCURRENTLY / DROP INDEX CONCURRENTLY (0029-0033) never take a blocking lock.

EXPLAIN output and timing, against a production-sized transactions table

Measured end-to-end by applying the full pre-#214 → post-#214 migration set (0001-0033) to a
scratch database seeded with 10 wallets, 200,000 addresses, and 2,000,000 transactions
(transactions at 842 MB, the largest table in the schema) on a resource-constrained 2-vCPU
dev container — i.e. a pessimistic baseline, not production-grade hardware. Reproduction script
available on request; not committed (throwaway docker exec/psql against a scratch DB, not
project code).

Phase Migration(s) What it does Measured time
Additive 0021-0022 Create chains, seed 2 rows; ADD COLUMN x6 (all nullable) < 5 ms total — metadata-only, confirms the "no table rewrite" claim
Backfill: wallets 0023 Single UPDATE, 10 rows 23 ms
Backfill: addresses 0024 Batched (5,000/commit), 200,000 rows 9.4 s (~21,300 rows/s)
Backfill: transactions 0025 Batched (5,000/commit), 2,000,000 rows sustained ~1,800-2,000 rows/s in the batched loop (measured across multiple 30-70k-row windows); a single unbatched UPDATE over the same join hit ~8,300 rows/s — the batched loop is ~4-5x slower by design, trading throughput for a bounded lock/WAL footprint per commit and full resumability
NOT VALID check add 0026 ADD CONSTRAINT ... NOT VALID x4 1.07 s total (1.0 s of that is the transactions constraint registering catalog metadata on an 842 MB table — no row scan)
VALIDATE CONSTRAINT 0027 Full-table scan, non-blocking lock 1.9 s for all 2,000,000 transactions rows (wallets/addresses: 7-173 ms)
SET NOT NULL 0028 Promote validated CHECK, drop it ~80 ms total for all 4 columns — confirms PG12's "reuse the validated CHECK, skip the scan" optimization
idx_addresses_chain 0029 CREATE INDEX CONCURRENTLY, 200,000 rows 470 ms
uq_addresses_chain_deposit 0030 CREATE UNIQUE INDEX CONCURRENTLY, 200,000 rows 906 ms
idx_tx_chain 0031 CREATE INDEX CONCURRENTLY, 2,000,000 rows 6.4 s
uq_tx_onchain_chain 0032 CREATE UNIQUE INDEX CONCURRENTLY, 2,000,000 rows — the anti-double-credit guard 19.4 s
Drop legacy index 0033 DROP INDEX CONCURRENTLY uq_tx_onchain 1.0 s

Total wall-clock for the whole 13-migration set against the 2M-row table: on the order of a few
minutes
, dominated by the transactions backfill (0025) — and every step in that dominant cost is
non-blocking (batched + committed incrementally) or has a bounded, sub-20-second blocking-lock-free
window (the CONCURRENTLY builds). At no point does any step hold ACCESS EXCLUSIVE on
transactions for longer than the sub-second catalog update in 0026.

EXPLAIN (ANALYZE, BUFFERS) after the full migration, confirming the new index is actually used
and cheap:

-- The anti-double-credit lookup record_deposit's write path relies on:
EXPLAIN (ANALYZE, BUFFERS)
SELECT 1 FROM transactions
WHERE chain_id = 'stellar:pubnet' AND tx_hash = $1 AND operation_index = 0;

 Index Only Scan using uq_tx_onchain_chain on transactions
   (cost=0.55..8.57 rows=1 width=4) (actual time=1.391..1.392 rows=0 loops=1)
   Index Cond: ((chain_id = 'stellar:pubnet'::text) AND (tx_hash = $1) AND (operation_index = 0))
   Heap Fetches: 0
   Buffers: shared hit=2 read=2
 Execution Time: 1.420 ms
-- A real INSERT against the migrated schema (FK + unique-index overhead):
EXPLAIN (ANALYZE, BUFFERS)
INSERT INTO transactions (wallet_id, chain_id, ..., tx_hash, operation_index, ...) VALUES (...);

 Insert on transactions  (actual time=30.218..30.219 rows=0 loops=1)
   Trigger for constraint transactions_chain_id_fkey: time=5.171 calls=1
 Execution Time: 37.326 ms
-- The chain-scoped deposit-address lookup (uq_addresses_chain_deposit):
EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM addresses WHERE chain_id = 'stellar:pubnet' AND deposit_address = $1;

 Index Scan using uq_addresses_chain_deposit on addresses
   (cost=0.46..8.48 rows=1 width=154) (actual time=4.058..4.060 rows=0 loops=1)
 Execution Time: 4.193 ms

Index-only scan on uq_tx_onchain_chain at ~1.4 ms confirms the chain-scoped index serves the
double-credit check at least as cheaply as the old two-column index did — the added chain_id
predicate costs nothing extra since it's the leading index column.

Post-migration integrity, same scratch DB: 2,000,000 (+1 probe row) transactions, 200,000
addresses, 10 wallets — zero rows lost; uq_tx_onchain_chain present and indisvalid/
indisready; legacy uq_tx_onchain absent.

Testing

  • migration_round_trip_preserves_pre_migration_stellar_data_and_invariants (store_tests.rs):
    bootstraps a scratch DB, applies only the pre-feat(store): Multi-chain database schema migration #214 migrations (0001-0020) to reproduce
    production's exact schema shape, seeds representative Stellar rows (server- and client-custody
    wallets, confirmed deposits, a pending withdrawal with a NULL hash, null asset_issuer/ledger),
    applies 0021-0033, and asserts zero data loss, correct backfill, enforced NOT NULL, the new
    indexes present and the old one gone, and that the live Store API still works end-to-end
    against the migrated-in-place schema.
  • same_tx_hash_and_operation_index_is_accepted_across_chains_but_not_within_one
    (store_tests.rs): the regression test for the core fix — proves the same
    (tx_hash, operation_index) is accepted across two different chain_ids and still rejected
    within one.
  • migrate_applies_exactly_the_expected_version_set updated for the 13 new migrations (33 total).
  • crates/ingest tests updated to allocate a fresh base Stellar account per wallet, since
    uq_addresses_chain_deposit (chain_id, deposit_address) now enforces uniqueness across all
    wallets on a chain (previously only per-wallet (wallet_id, muxed_id) was enforced) — this
    matches on-chain reality (no two real Stellar accounts share a base key) and was already latent,
    not a new restriction introduced by this PR.
  • Full workspace suite (cargo test --workspace --locked), cargo clippy --workspace --all-targets --locked -- -D warnings, and cargo fmt --all -- --check all pass.

Guarantees preserved

  • Atomic address allocation ([Store::allocate_address]) — now also reads and stamps the wallet's
    chain_id inside the same row-locked transaction.
  • Idempotent deposit recording ([Store::record_deposit]) — idempotent on the chain-scoped
    (chain_id, tx_hash, operation_index) index.
  • Idempotent withdrawal creation ([Store::create_withdrawal]) — unchanged, (wallet_id, idempotency_key).

Refs #214. Depends on #213 (unblocked; this PR doesn't require the octo-chain adapter trait to
land — it only prepares the schema). Blocks #215, #220, #223.

Adds a chains registry and chain_id to wallets/addresses/transactions,
and re-scopes the anti-double-credit unique index to
(chain_id, tx_hash, operation_index) — an EVM tx hash is not unique
across chains, so the old two-column index would have dropped
legitimate cross-chain deposits.

Migration is online-safe: additive columns, backfill, CONCURRENTLY-built
index, old index dropped only after the new one is live.

Backward-compat strategy: keep every legacy column (network,
stellar_tx_hash, muxed_address) and add generic ones alongside
(chain_id, tx_hash, deposit_address), backfilled and kept in lockstep
by every Store write path. Chosen over a two-release rename-behind-a-view
because this crate leans on SELECT */RETURNING * with sqlx::FromRow,
which a view/generated-column indirection complicates for little benefit
at this schema's size.

Closes Octo-Protocol-org#214
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(store): Multi-chain database schema migration

1 participant