fix(persistence): version-guarded restore_deleted, PUT race retry, and the two-handle cluster harness (Phase 0.1)

[aacruzgon]

F5: version-guarded restore + PUT race retry, and the two-handle cluster harness

First PR of the cluster-capable-state rebuild (discussion #223, Phase 0.1). It fixes the one live single-instance data bug left in the F5 finding and lands the two-instance test harness every later cluster PR is built on. No migration, no config change, no behaviour change for non-racing callers.

Why

Postgres restore_deleted — the path an unconditional PUT takes onto a soft-deleted id — was the last read-then-write on the write path: SELECT version_id, compute +1 in Rust, UPDATE with no version predicate, then a separate history INSERT. Two PUTs racing onto one tombstone both computed the same version; the second history insert hit the (tenant_id, resource_type, id, version_id) primary key (a 500), and the live row and history v3 carried different bodies. update and delete had already been made compare-and-swaps on main; this closes the gap. Mongo's restore was already a CAS, but a lost race surfaced as NotFound, so a PUT racing another PUT returned 404.

Every later phase needs a test shape that proves a protocol across two instances. A cloned Arc shares the in-process heap and proves nothing, so the harness constructs two backends independently over one shared database, races them from a barrier, and asserts the definition-of-done rows.

Changes

Persistence — Postgres

• restore_deleted lands the row update and the history insert in one statement guarded by is_deleted = TRUE AND version_id = $expected. Zero rows re-selects (deliberately without an is_deleted filter: "someone restored" and "someone restored then re-deleted" are the same fact) and reports VersionConflict with the version actually stored, or NotFound. The pre-read stays: it is what lets the loser report a truthful expected_version, matching S3's restore contract.
• create_or_update retries up to three times from a fresh read when the write reports VersionConflict, AlreadyExists, or NotFound. None of those is a legitimate answer from a method that decides existence itself, so unconditional PUT stays last-writer-wins at the API surface. NotFound is in the set because update's zero-row branch returns it when the row was deleted underneath it.

Persistence — Mongo

• Same bounded retry in create_or_update.
• restore_deleted's zero-match branch re-finds without the is_deleted filter and reports VersionConflict or NotFound, so the taxonomy matches Postgres and S3.

Trait contract

• ResourceStorage::create_or_update gains a # Concurrency paragraph and an # Errors section (it had none).

Test harness

• New crates/persistence/tests/common/cluster_harness.rs, #[path]-included from postgres_tests.rs and mongodb_tests.rs: two_handles, barrier-started race2, assert_exactly_one, assert_visible, assert_wrong_tenant_hidden.
• Calibration suite against the already-cluster-safe bulk-export job store (visibility, wrong-tenant isolation, claim exclusivity, fencing after release, durability across a handle drop). These pass on the unchanged tree: they prove the harness, not the product.

Hygiene

• .gitignore covers the local cluster planning docs under docs/ (separate commit).

Testing

Red first, on the unchanged tree (3 of 3 runs):

postgres_integration_cluster_resource_create_or_update_race_both_succeed
  create_or_update must absorb CAS races: Concurrency(VersionConflict { resource_type: "Patient", id: "…", expected_version: "1", actual_version: "2" })

postgres_integration_cluster_restore_deleted_race_keeps_history_coherent
  round 0: PUT via A must succeed: Backend(Internal { backend_name: "postgres", message: "Failed to insert restore history: db error", source: None })

mongodb_integration_cluster_restore_deleted_race_keeps_history_coherent
  round 0: PUT via A must succeed: Resource(NotFound { resource_type: "Patient", id: "…" })

Green after the fix (two runs each, five race rounds per run):

• cargo test -p helios-persistence --features postgres --test postgres_tests -- cluster_ — 9 passed (4 calibration, 3 F5 update/PUT/delete races, restore race, restore isolation).
• cargo test -p helios-persistence --features mongodb --test mongodb_tests -- cluster_restore — 2 passed.
• Full binaries: postgres_tests 165 passed, mongodb_tests 80 passed.
• cargo fmt --all; CI-exact clippy (--all-targets --all-features -D warnings with CI's -A set) clean.

Definition-of-done rows for the restore path: isolation ✓ (wrong-tenant PUT creates a fresh v1 and leaves the tombstone), exclusivity ✓ (exactly one racer restores; the other converges through update), durability ✓ (history contiguous 1..=4, newest entry equals the live body). Visibility and fencing do not apply: a resource row has no lease, and cross-handle reads are plain CRUD already covered by the calibration suite.

Notes

• The retry clones the request body once per attempt except the last. A follow-up to take &Value in the write paths would remove it.
• SQLite's restore_deleted has the same unguarded shape; out of scope here (SQLite cannot be clustered) and tracked as a follow-up.
• Search-index writes still run as separate statements after the row statement on every write path, so tests must not assert search results after a race; unchanged by this PR.
• Existing create_or_update_restores_deleted tests on both backends are untouched and still expect a three-entry history.

[codecov]

Codecov Report

❌ Patch coverage is 81.25000% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/persistence/src/backends/mongodb/storage.rs	70.37%	8 Missing ⚠️
...rates/persistence/src/backends/postgres/storage.rs	89.18%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!