Event-driven order processing system (saga, idempotency, outbox, tracing)

[ak1oo3]

What this is

A simulated e-commerce backend where placing an order triggers a chain of async events across 5 independent services coordinated over RabbitMQ instead of one synchronous function call. The order service owns a saga: it issues each command, consumes every reply, and on failure walks the completed steps in reverse issuing compensating transactions one hop at a time.

Client ──POST /orders──► order-service ──► RabbitMQ ──► inventory ──► payment ──► shipping
                              ▲                                                      │
                              └──────────── replies advance the saga ◄───────────────┘
                                     failure ──► compensations, in reverse

The three problems it's built around

• Saga with compensating transactions — on failure the orchestrator undoes completed steps in strict reverse order. A carrier failure after the card is charged refunds first, then releases stock.
• Idempotent consumers — every handler claims the message's event_id in a processed_events ledger in the same transaction as its business write, turning at-least-once delivery into effectively-once processing.
• Transactional outbox — domain writes and outgoing messages commit together, so a crash between "write" and "publish" can't strand a saga.

Plus broker-side exponential-backoff retries with a dead-letter parking lot, per-service PostgreSQL databases and roles (the only cross-boundary state is a message), and OpenTelemetry tracing where one order renders as a single Jaeger trace across all five services.

Changes

• packages/shared/ — platform library: broker (publish-with-confirms, consume-with-retry/DLQ, hand-built producer/consumer spans), topology (exchanges + backoff delay tiers), transactional outbox + relay, idempotency claim, service bootstrap, OTel + Prometheus.
• services/{order,inventory,payment,shipping,notification}-service/ — the five services, each with its own migrations and database. order-service/src/saga.js is a pure, unit-tested state machine.
• infra/ + docker-compose.yml + Dockerfile — RabbitMQ, PostgreSQL (one DB/role per service), Jaeger, Prometheus.
• scripts/ — smoke, chaos, bench, reconcile, verify-tracing, dev runner.
• tests/ — 19 unit tests (saga transitions, backoff tiers, validation).
• docs/ — architecture (Mermaid sequence diagrams + state machine + retry topology), design decisions with trade-offs, measured benchmarks, repo summary.
• .github/workflows/ci.yml — unit + end-to-end + chaos + reconcile suites.

Verification (run against a live stack)

• 19 unit tests, 29 end-to-end assertions (every saga path), 11 failure-injection drills (duplicate command, replayed reply, poison message), and a cross-service consistency audit — all pass from a clean database.
• Under simultaneous chaos (25% payment declines, 25% carrier failures, 5% handler crashes, every message published twice): 200/200 orders reached a consistent terminal state, 1,922 duplicates dropped, 0 messages parked, no stranded payments or reservations, stock conserved exactly.
• Measured optimization: batching publisher confirms in the outbox relay cut saga p50 latency 9.3s → 1.9s (5×) and raised throughput 2.8×. Full numbers and methodology in docs/benchmarks.md.

Notes for the reviewer

• The repository was empty, so this PR targets a newly created empty-root main as its base; the diff is the entire initial system.
• Docker images couldn't be pulled in the build environment (registry blocked by the network policy), so the stack was validated by running RabbitMQ and PostgreSQL natively — all measured numbers come from real runs, not estimates.