Payments 7/7: End-to-End Payment Orchestration, Observability & Production-Readiness Demo

[yusuftomilola]

Problem / Context

Issues #1570–#1575 build the individual engineering pieces — domain model, confirmation pipeline, failure handling/reconciliation, wallets, on-chain rail, micropayment ledger — but a coworking-space member needs one coherent journey, and the team needs confidence the whole system is operable in production, not just individually correct.

This is issue 7 of 7 in the payment track — the capstone that ties everything together.

Objective

Wire every prior piece into one demonstrable end-to-end payment flow (fiat or Stellar/Soroban, real-time confirmation, booking activation, credit funding and revenue split where relevant), plus the observability and admin tooling needed to run it in production.

Technical approach

• A PaymentOrchestrator sitting above the rail adapters from Payments 1/7: Payment Domain Model, Initiation Flow & Idempotent Transaction Lifecycle #1570, so checkout/booking code calls one interface regardless of which rail the user chose, and every rail reports through the same status/event model from Payments 2/7: Real-Time Confirmation Pipeline & the App/Provider State Boundary #1571.
• Observability: the structured payment event log from Payments 2/7: Real-Time Confirmation Pipeline & the App/Provider State Boundary #1571, exposed as an admin timeline per payment/booking; metrics (initiated/confirmed/failed counts per rail, p50/p95 confirmation latency per rail, reconciliation-resolved count, manual-review queue depth) on a dashboard; alerts for manual-review queue depth, reconciliation job failures, and elevated failure rate per rail.
• Operational recovery: a runbook covering "payment stuck," "webhook outage," "RPC outage," and "reconcile a specific payment now," each backed by the admin actions already built in Payments 3/7: Failure Taxonomy, Retries, Partial Outcomes & Payment Reconciliation Engine #1572 and Payments 5/7: On-Chain Escrow Rail — Soroban Integration & Chain-State Reconciliation #1574.
• A scripted, repeatable demo (automated in CI or documented as manual QA): create a booking → pay via the fiat rail → observe realtime confirmation → booking auto-activates → invoice/notification fires; repeat via a custodial Stellar wallet (Payments 4/7: Custodial Wallet Onboarding & Non-Custodial Upgrade Path #1573) → on-chain escrow create/release (Payments 5/7: On-Chain Escrow Rail — Soroban Integration & Chain-State Reconciliation #1574) → same outcome; deliberately induce a failure (simulated webhook loss) and show reconciliation (Payments 3/7: Failure Taxonomy, Retries, Partial Outcomes & Payment Reconciliation Engine #1572) recovering it without manual intervention.

Detailed scope

• PaymentOrchestrator consolidating rail selection and a single confirmation/status contract for calling code.
• Admin payment timeline (per-payment event log) and a metrics dashboard.
• Alerting wired to the thresholds above.
• Runbook document.
• Automated E2E test or scripted demo covering both rails plus one induced-failure recovery path.

Important edge cases and failure scenarios

• A booking's payment succeeds on-chain but the booking-activation step itself fails (app bug, DB hiccup) — the orchestrator must not silently strand a paid-but-unactivated booking; needs a compensating retry and an alert, never a lost state.
• A user abandons the fiat rail mid-flow and retries with a wallet — the orchestrator must guarantee only one Payment can end up CONFIRMED per booking, reusing Payments 1/7: Payment Domain Model, Initiation Flow & Idempotent Transaction Lifecycle #1570's per-booking uniqueness guarantee.
• The dashboard/metrics pipeline itself fails or lags — it must visibly degrade to "no data," never present stale numbers as current.
• A runbook action is invoked against a payment already resolved by the time an admin acts — actions must be safe/no-op against already-terminal payments, reusing every prior issue's idempotency guarantees.

Dependencies

Depends on #1570, #1571, #1572, #1573, #1574, #1575. This is the integration/capstone issue and cannot meaningfully start before #1574 and #1575 exist, since it demonstrates both rails and the ledger/split path.

Acceptance criteria

• The scripted demo completes successfully end-to-end for both rails in staging, with real-time UI confirmation observed for each.
• The induced-failure scenario (webhook loss) is shown recovering via reconciliation without manual admin action.
• The admin dashboard shows accurate, live counts for at least: payments by status, by rail, and manual-review queue depth.
• The runbook is reviewed and usable by someone other than the implementer (a cold-read check).

Definition of done

The orchestrator is merged and used by the real booking checkout path, not just the demo script; dashboard and alerts are live; the runbook is published in the repo/wiki; the scripted E2E demo passes in CI or is documented as a repeatable manual QA procedure; a walkthrough (recording or live) shows the full journey for both rails.