feat(store): generalise schema to multi-chain - #232
Open
Manuel1234477 wants to merge 1 commit into
Open
Conversation
Adds a chains registry and chain_id to wallets/addresses/transactions, and re-scopes the anti-double-credit unique index to (chain_id, tx_hash, operation_index) — an EVM tx hash is not unique across chains, so the old two-column index would have dropped legitimate cross-chain deposits. Migration is online-safe: additive columns, backfill, CONCURRENTLY-built index, old index dropped only after the new one is live. Backward-compat strategy: keep every legacy column (network, stellar_tx_hash, muxed_address) and add generic ones alongside (chain_id, tx_hash, deposit_address), backfilled and kept in lockstep by every Store write path. Chosen over a two-release rename-behind-a-view because this crate leans on SELECT */RETURNING * with sqlx::FromRow, which a view/generated-column indirection complicates for little benefit at this schema's size. Closes Octo-Protocol-org#214
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #214.
Generalises the schema from Stellar-only to multi-chain: a
chainsregistry (CAIP-2-shapedchain_idslugs), pluschain_idthreaded throughwallets,addresses, andtransactions,and a chain-scoped replacement for the anti-double-credit unique index
(
uq_tx_onchain→uq_tx_onchain_chainon(chain_id, tx_hash, operation_index)).An EVM tx hash is not globally unique across chains — the same
(tx_hash, operation_index)paircan legitimately occur on two different chains. The old two-column index would have silently
rejected the second chain's deposit as a duplicate (a dropped-deposit bug). This is fixed by
re-scoping the guard to include
chain_id, builtCONCURRENTLYand swapped in only after it'sconfirmed live — the anti-double-credit invariant is never unenforced, even momentarily.
Backward-compatibility strategy (the main reviewable decision)
Chosen: keep every legacy column, add generic ones alongside (option (b) in the issue).
network,stellar_tx_hash, andmuxed_addressall still exist unchanged;chain_id,tx_hash,and
deposit_addressare new columns kept in lockstep by everyStorewrite path(
octo_store::stellar_chain_id_for_networkis the single place that derives one from the other).Rejected the two-release rename-behind-a-view approach because: this crate's queries lean on
SELECT */RETURNING *withsqlx::FromRow, which a compatibility view or generated columncomplicates for comparatively little benefit at this schema's size; and keeping both names live
throughout the transition is simpler to reason about than sequencing two coordinated releases
against one running production binary. The cost is duplicated data (
tx_hash == stellar_tx_hashfor every Stellar row today) — acceptable short-term; a follow-up migration can drop the legacy
columns once every caller has moved onto the
octo-chainadapter (#213/#215/#220/#223).Migration shape (online-safe)
Additive → backfill (batched, resumable) →
NOT VALIDcheck →VALIDATE CONSTRAINT→SET NOT NULL→ new indexes builtCONCURRENTLY→ old index dropped only once the new one isconfirmed live. Full step-by-step rationale is in
docs/architecture.md("Data model:multi-chain (#214)"), including an ER diagram.
No step takes a long
ACCESS EXCLUSIVElock ontransactions:ADD COLUMN ... NULL(0022) is metadata-only on PG11+.ADD CONSTRAINT ... NOT VALID(0026) registers instantly; the actual scan runs underVALIDATE CONSTRAINT'sSHARE UPDATE EXCLUSIVE(0027), which doesn't block reads/writes.SET NOT NULL(0028) reuses the already-validated CHECK on PG12+ and skips its own scan.CREATE INDEX CONCURRENTLY/DROP INDEX CONCURRENTLY(0029-0033) never take a blocking lock.EXPLAIN output and timing, against a production-sized
transactionstableMeasured end-to-end by applying the full pre-#214 → post-#214 migration set (0001-0033) to a
scratch database seeded with 10 wallets, 200,000 addresses, and 2,000,000 transactions
(
transactionsat 842 MB, the largest table in the schema) on a resource-constrained 2-vCPUdev container — i.e. a pessimistic baseline, not production-grade hardware. Reproduction script
available on request; not committed (throwaway
docker exec/psqlagainst a scratch DB, notproject code).
chains, seed 2 rows;ADD COLUMNx6 (all nullable)UPDATE, 10 rowsUPDATEover the same join hit ~8,300 rows/s — the batched loop is ~4-5x slower by design, trading throughput for a bounded lock/WAL footprint per commit and full resumabilityNOT VALIDcheck addADD CONSTRAINT ... NOT VALIDx4transactionsconstraint registering catalog metadata on an 842 MB table — no row scan)VALIDATE CONSTRAINTtransactionsrows (wallets/addresses: 7-173 ms)SET NOT NULLidx_addresses_chainCREATE INDEX CONCURRENTLY, 200,000 rowsuq_addresses_chain_depositCREATE UNIQUE INDEX CONCURRENTLY, 200,000 rowsidx_tx_chainCREATE INDEX CONCURRENTLY, 2,000,000 rowsuq_tx_onchain_chainCREATE UNIQUE INDEX CONCURRENTLY, 2,000,000 rows — the anti-double-credit guardDROP INDEX CONCURRENTLY uq_tx_onchainTotal wall-clock for the whole 13-migration set against the 2M-row table: on the order of a few
minutes, dominated by the transactions backfill (0025) — and every step in that dominant cost is
non-blocking (batched + committed incrementally) or has a bounded, sub-20-second blocking-lock-free
window (the
CONCURRENTLYbuilds). At no point does any step holdACCESS EXCLUSIVEontransactionsfor longer than the sub-second catalog update in 0026.EXPLAIN (ANALYZE, BUFFERS)after the full migration, confirming the new index is actually usedand cheap:
Index-only scan on
uq_tx_onchain_chainat ~1.4 ms confirms the chain-scoped index serves thedouble-credit check at least as cheaply as the old two-column index did — the added
chain_idpredicate costs nothing extra since it's the leading index column.
Post-migration integrity, same scratch DB:
2,000,000(+1 probe row)transactions,200,000addresses,10wallets— zero rows lost;uq_tx_onchain_chainpresent andindisvalid/indisready; legacyuq_tx_onchainabsent.Testing
migration_round_trip_preserves_pre_migration_stellar_data_and_invariants(store_tests.rs):bootstraps a scratch DB, applies only the pre-feat(store): Multi-chain database schema migration #214 migrations (0001-0020) to reproduce
production's exact schema shape, seeds representative Stellar rows (server- and client-custody
wallets, confirmed deposits, a pending withdrawal with a NULL hash, null
asset_issuer/ledger),applies 0021-0033, and asserts zero data loss, correct backfill, enforced NOT NULL, the new
indexes present and the old one gone, and that the live
StoreAPI still works end-to-endagainst the migrated-in-place schema.
same_tx_hash_and_operation_index_is_accepted_across_chains_but_not_within_one(
store_tests.rs): the regression test for the core fix — proves the same(tx_hash, operation_index)is accepted across two differentchain_ids and still rejectedwithin one.
migrate_applies_exactly_the_expected_version_setupdated for the 13 new migrations (33 total).crates/ingesttests updated to allocate a fresh base Stellar account per wallet, sinceuq_addresses_chain_deposit (chain_id, deposit_address)now enforces uniqueness across allwallets on a chain (previously only per-wallet
(wallet_id, muxed_id)was enforced) — thismatches on-chain reality (no two real Stellar accounts share a base key) and was already latent,
not a new restriction introduced by this PR.
cargo test --workspace --locked),cargo clippy --workspace --all-targets --locked -- -D warnings, andcargo fmt --all -- --checkall pass.Guarantees preserved
Store::allocate_address]) — now also reads and stamps the wallet'schain_idinside the same row-locked transaction.Store::record_deposit]) — idempotent on the chain-scoped(chain_id, tx_hash, operation_index)index.Store::create_withdrawal]) — unchanged,(wallet_id, idempotency_key).Refs #214. Depends on #213 (unblocked; this PR doesn't require the
octo-chainadapter trait toland — it only prepares the schema). Blocks #215, #220, #223.