AsyncCheck — event-driven order processing

Placing an order here does not call four functions. It publishes one message, and an orchestrated saga across five independent services takes it from there: reserve stock, charge the card, book the carrier, notify the customer — and, when any step fails, undo the steps that already succeeded, in reverse order.

Every service has its own database, every consumer is idempotent because the broker only promises at-least-once delivery, and every order is one distributed trace you can read end to end.

Client ──POST /orders──► order-service ──► RabbitMQ ──► inventory ──► payment ──► shipping
                              ▲                                                      │
                              └──────────── replies advance the saga ◄───────────────┘
                                     failure ──► compensations, in reverse

Services	order (API + saga orchestrator), inventory, payment, shipping, notification
Broker	RabbitMQ — topic exchange, exponential-backoff retry tiers, dead-letter parking
Storage	PostgreSQL, one database and one role per service
Patterns	Saga with compensating transactions · transactional outbox · idempotent consumers
Observability	OpenTelemetry → Jaeger (one trace per order, all five services) · Prometheus metrics
Verified	19 unit tests · 29 end-to-end assertions · 11 failure-injection drills · a cross-service consistency audit

---

The three problems this project is actually about

1. What happens when the payment service fails halfway through an order?

Nothing is left half-done. The orchestrator tracks each saga step in saga_steps and, on failure, walks the completed steps in reverse and issues one compensating command at a time. A carrier failure after the card was charged triggers payment.refund first and only then inventory.release. Verified end to end — see the drills below.

2. What happens when a message is delivered twice?

It gets dropped. RabbitMQ guarantees at-least-once delivery, so every consumer claims the envelope's event_id in a processed_events ledger in the same transaction as its business write. Zero rows affected means "already handled". Publishing every single message twice under load produced 1,922 dropped duplicates and zero double-charges.

3. What happens when a service crashes between writing and publishing?

It can't lose the message. Domain writes and outgoing messages commit together via a transactional outbox; a relay publishes afterwards. Worst case a message is published twice — which problem 2 already makes harmless.

Full reasoning, with the trade-offs each choice costs: docs/decisions.md.

---

Quick start

cp .env.example .env
npm run up            # RabbitMQ, PostgreSQL, Jaeger, Prometheus + the five services
npm run smoke         # 29 end-to-end assertions across every saga path

Place an order:

curl -sX POST localhost:3000/orders -H 'content-type: application/json' -d '{
  "customerId": "cust-1",
  "items": [{ "sku": "SKU-KEYBOARD", "quantity": 1, "unitPriceCents": 12900 }]
}' | jq

curl -s localhost:3000/orders/<id> | jq '{status, failureReason, saga}'

{
  "status": "CONFIRMED",
  "failureReason": null,
  "saga": [
    { "step": "RESERVE_INVENTORY", "seq": 1, "status": "COMPLETED", "detail": { "reservationId": "…" } },
    { "step": "CHARGE_PAYMENT",    "seq": 2, "status": "COMPLETED", "detail": { "providerRef": "ch_77a63ebdf704…" } },
    { "step": "CREATE_SHIPMENT",   "seq": 3, "status": "COMPLETED", "detail": { "trackingNo": "1ZD70963504E234E0C" } }
  ]
}

Watch it happen:

Jaeger — one trace per order	http://localhost:16686
RabbitMQ management (asynccheck/asynccheck)	http://localhost:15672
Prometheus	http://localhost:9090
Service logs (JSON, with trace_id on every line)	npm run logs

Break it on purpose

# the card declines -> stock is released, nothing ships
curl -sX POST localhost:3000/orders -H 'content-type: application/json' \
  -d '{"customerId":"c","items":[{"sku":"SKU-KEYBOARD","quantity":1,"unitPriceCents":12900}],
       "simulate":"payment_failure"}' | jq -r .id

# the carrier fails after the card was charged -> refund, then release
#   "simulate": "shipping_failure"
# the warehouse is short -> saga fails at step 1, card never touched
#   "items": [{"sku":"SKU-RARE","quantity":999,"unitPriceCents":24900}]

The order ends FAILED, and the compensations show up in the saga ledger:

{
  "status": "FAILED",
  "failureReason": "no_carrier_capacity",
  "saga": [
    { "step": "RESERVE_INVENTORY", "status": "COMPENSATED" },   // stock released
    { "step": "CHARGE_PAYMENT",    "status": "COMPENSATED" },   // money refunded
    { "step": "CREATE_SHIPMENT",   "status": "FAILED" }
  ]
}

---

Results

Measured on 4 vCPU / 15 GB, everything on one box, durability on (persistent messages, publisher confirms, fsync). Full methodology and caveats: docs/benchmarks.md.

One order, idle system	56 ms end-to-end across 5 services, 9 messages, 12 transactions
Sustained (200 @ 5 concurrent)	74.5 orders/s, saga p50 484 ms, p95 662 ms
Burst (500 @ 25 concurrent)	500/500 confirmed, ingest 104 orders/s absorbed by the broker
Optimising the outbox relay	saga p50 9,326 ms → 1,856 ms (5.0×), throughput 2.8×

Under simultaneous chaos — 25% payment declines, 25% carrier failures, 5% handler crashes, and every message published twice:

Orders reaching a terminal state	200 / 200 — none stuck
Duplicates detected and dropped	1,922
Messages retried and recovered	223
Messages parked in a dead-letter queue	0
Cross-service consistency audit	passed — no stranded money, no stranded stock

Stock conservation held exactly: on_hand + reserved equalled the seeded level for every SKU, and all 82 failed orders were refunded (or never charged) with their stock returned.

---

How it works

Sequence diagrams for the happy path and both compensation paths, the saga state machine, the retry/DLQ topology and the message contracts are in docs/architecture.md.

The short version:

• The order service owns the saga. It issues each command, consumes every reply, and keeps one saga_steps row per step (with the seq that compensation runs in reverse, and the detail a compensation will need — e.g. the payment reference a refund requires). Concurrent replies for one order are serialised with SELECT … FOR UPDATE on the order row.
• decide(snapshot, event) is a pure function of (saga state, inbound event). It returns null for any event that must not move the saga — a reply for a step already settled, or anything at all after a terminal state. That is a second line of defence behind the idempotency ledger, and it is what the unit tests exercise.
• Retries live in the broker. A failed handler republishes the message into a delay tier (<queue>.retry.1000ms, .2000ms, .4000ms, … capped at 60s) whose x-message-ttl dead-letters it back onto the live queue, then acks the original. Nothing sleeps in application code, and a restart doesn't lose in-flight retries. After AMQP_MAX_ATTEMPTS the message is parked in <queue>.parking so one poison message never blocks the queue.
• Trace context rides on the message. W3C traceparent goes into the AMQP headers on publish and is extracted on consume, so the consumer span is a child of the publisher span and one order is one Jaeger trace across all five services.

---

API

order-service — :3000

POST /orders	Start a saga. Honours Idempotency-Key; a retried request returns the original order instead of starting a second saga. 202 Accepted.
GET /orders/:id	Order, line items and the full saga step ledger.
GET /orders?status=&limit=	List orders.
GET /stats	Orders by status, saga latency avg/p95, processed-event count, outbox backlog.

Every service also exposes GET /healthz, GET /readyz and GET /metrics.

GET :3001/inventory · POST :3001/inventory/restock	Stock levels; top up for demos and load tests.
GET :3001/inventory/reservations/:orderId	Per-order reservations and their release state.
GET :3002/payments/:orderId · GET :3002/payments	Payment record; totals by status.
GET :3003/shipments/:orderId · GET :3003/shipments	Shipment record; totals by status.
GET :3004/notifications?orderId=	The "emails" the customer would have received.

---

Verifying it

npm test                  # 19 unit tests: saga transitions, backoff tiers, validation
npm run smoke             # 29 end-to-end assertions against a live stack
npm run chaos             # duplicate delivery, replayed replies, poison messages
npm run reconcile         # cross-service audit: no stranded money, no stranded stock
node scripts/bench.js 500 25 # load test with latency percentiles and message counters
npm run verify:tracing    # asserts one trace spans all five services (native mode)

npm run chaos runs three drills against a live stack:

1. Duplicate command — reads the exact inventory.reserve envelope the order service published and puts a byte-identical copy back on the exchange. Stock must not move twice.
2. Replayed reply — replays payment.completed at the orchestrator for an order that is already CONFIRMED. The saga must not re-run or emit a second shipping command.
3. Poison message — injects a message that always throws. It must exhaust its retries, land in inventory.commands.parking, and leave the live queue draining normally.

Run it with short retries so the third drill doesn't take a minute:

AMQP_MAX_ATTEMPTS=3 AMQP_RETRY_DELAY_MS=1000 npm run up && npm run chaos

---

Configuration

Everything is environment-driven; see .env.example. The knobs that matter:

Variable	Default	Purpose
AMQP_MAX_ATTEMPTS	5	Attempts before a message is parked in the DLQ
AMQP_RETRY_DELAY_MS	5000	Base backoff; doubles per attempt, capped at 60s
AMQP_PREFETCH	16	Unacked messages per consumer
PAYMENT_FAILURE_RATE	0	Fraction of card authorisations that decline
SHIPPING_FAILURE_RATE	0	Fraction of carrier bookings that fail (forces refunds)
CHAOS_DUPLICATE_PUBLISH_RATE	0	Publish every message twice, same event id
CHAOS_HANDLER_FAILURE_RATE	0	Fraction of handler calls that throw
DB_POOL_MAX	10	PostgreSQL pool size per service
OTEL_ENABLED	true	Turn tracing off for clean benchmarks

Running without Docker

Useful for development, and required for npm run verify:tracing (which binds the OTLP port itself). Needs a local PostgreSQL and RabbitMQ:

psql -U postgres -f infra/postgres/init.sql     # per-service databases and roles
rabbitmqctl add_user asynccheck asynccheck && rabbitmqctl set_permissions -p / asynccheck '.*' '.*' '.*'
npm install
npm run dev                                     # all five services, prefixed logs

Layout

packages/shared/          the platform library every service is built on
  src/broker.js             connection, publish with confirms, consume with retry/DLQ, spans
  src/topology.js           exchanges, delay tiers, backoff, dead-letter parking
  src/outbox.js             transactional outbox + relay
  src/idempotency.js        the processed_events claim
  src/service.js            boot: db + migrations + broker + relay + http + shutdown
  src/tracing.js            OpenTelemetry bootstrap
services/order-service/     API, saga orchestrator (saga.js is pure and unit-tested)
services/inventory-service/ stock reservation and release
services/payment-service/   mocked gateway, capture and refund
services/shipping-service/  carrier booking and cancellation
services/notification-service/ leaf consumer; renders and records customer messages
scripts/                    smoke, chaos, bench, reconcile, verify-tracing, dev runner
docs/                       architecture, decisions, benchmarks

Limitations

Things this deliberately does not do, and what they'd take:

• The outbox relay assumes one instance per service. Ordering is preserved by ORDER BY id with a single relay; scaling out needs partitioning by correlation_id.
• processed_events grows forever. Production needs a retention job dropping rows older than the longest possible redelivery window.
• Parked messages need an operator. There is no replay-from-DLQ tooling yet; the parking queues hold the message plus an x-parked-reason header, which is the input such a tool would need.
• No authentication. The gateway is open — this is a systems demonstration, not a production storefront.
• Compensations are assumed to succeed. They retry and can be parked, but a permanently-failing refund needs a human; the design records enough state (saga_steps.detail) for one to finish the job by hand.