fix(infra): stop the old task before starting the new one on the singleton services

[robertleifke]

What

execution-service and market-maker-spot deploy with the ECS defaults — 100% minimum healthy, 200% maximum — so every rolling deploy runs two tasks at once. Both are singletons. The matcher already sets 0/100 for exactly this reason; this applies the same setting to the two that were missed.

service	before	after	why
markets	100/200	unchanged	HTTP API, no singleton constraint — rolling avoids downtime
matcher	0/100	unchanged	already correct
execution	100/200	0/100	shared executor EOA nonce sequence
market-maker	100/200	0/100	two makers quote the same book

Why it matters for execution-service

Two tasks sign with the same KMS key, and nothing coordinates the nonce across processes. serial-queue.ts serialises within one process and states the hazard itself:

"viem reads the next nonce from the RPC at send time, so two sends in flight at once can take the same nonce and one replaces or rejects the other."

There is no load balancer to drain — Cloud Map returns an A record for both tasks (MULTIVALUE, TTL 10s), so a settlement lands on whichever the matcher resolves. The deployment circuit breaker is also disabled, so nothing would have caught a collision.

Why it matters for market-maker

Not hypothetical. A draining task kept placing orders for seconds after its successor had finished startup reconciliation, leaving quotes resting on the live book under the previous configuration that nothing re-examined.

The cost, and why it's acceptable

A gap of roughly a minute or two per deploy on those two services. For settlement that's lag, not loss, and I checked the matcher rather than assuming:

• internal/matching/revert.go — revertUnknown is the zero value, documented as "treated like a transient failure". A connection error carries no revert selector, so it cannot be classified permanent and cannot park a pair.
• internal/matching/backoff.go — retries on a doubling backoff, 2s to a 5-minute cap, then every 30 minutes after twelve failures, until an order expires (the venue signs for a day). It is explicitly "non-destructive: it delays retries, it never cancels an order."

A ~2 minute gap accumulates ~6 failures, so settlement resumes within a couple of minutes of the service returning.

Applying this restarts nothing

The plan is two in-place service updates changing only deploymentConfiguration:

~ deployment_maximum_percent         = 200 -> 100
~ deployment_minimum_healthy_percent = 100 -> 0
Plan: 0 to add, 2 to change, 0 to destroy.

No task definition is registered and no new deployment is forced, so running tasks are untouched.

Context

Found while preparing the execution-service deploy of #69/#71 — the rollout would have been a rolling one over a shared nonce sequence. With this merged, that deploy becomes a single reviewed apply that stops before it starts, instead of a manual scale-to-zero dance that has to be remembered every time.

🤖 Generated with Claude Code