From c0ef8928b33b8f6cfce68b29436fc18251b04a13 Mon Sep 17 00:00:00 2001 From: Robert Leifke Date: Sun, 20 Sep 2026 09:09:21 -0400 Subject: [PATCH] fix(infra): stop the old task before starting the new one on the singleton services execution-service and market-maker-spot deployed with the ECS defaults, 100% minimum healthy and 200% maximum, so every rolling deploy ran two tasks at once. Both are singletons, and the matcher already sets 0/100 for exactly this reason. This applies the same setting to the two that were missed. execution-service: two tasks sign with the SAME KMS key, and nothing coordinates the executor EOA's nonce across processes. serial-queue.ts serialises within one process and states the hazard plainly -- "viem reads the next nonce from the RPC at send time, so two sends in flight at once can take the same nonce and one replaces or rejects the other". There is no load balancer to drain: Cloud Map returns an A record for BOTH tasks (MULTIVALUE, TTL 10s), so a settlement lands on whichever the matcher resolves. The deployment circuit breaker is disabled, so nothing would have caught it either. market-maker-spot: two market makers quote the same book. Already observed -- a draining task kept placing orders for seconds after its successor had finished its startup reconciliation, leaving quotes resting under the previous configuration that nothing re-examined. markets-service is deliberately left rolling. It is an HTTP API with no singleton constraint, so rolling avoids downtime and costs nothing. The cost is a gap of roughly a minute or two per deploy on those two services. For settlement that is lag, not loss: the matcher classifies an unrecognised executor error as transient (internal/matching/revert.go, revertUnknown is the zero value and documented as "treated like a transient failure") and retries on a doubling backoff -- 2s to a 5m cap, then every 30m after twelve failures -- that never cancels an order. A connection failure carries no revert selector, so it cannot be classified permanent and cannot park a pair. Applying this changes deploymentConfiguration only. It registers no task definition and forces no new deployment, so nothing restarts. Found while preparing the execution-service deploy of #71/#69: the rollout plan was a rolling one, and the nonce sequence is shared. Co-authored-by: Claude Opus 5 (1M context) --- infra/aws/ecs.tf | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/infra/aws/ecs.tf b/infra/aws/ecs.tf index f8a1511..d2c9ff3 100644 --- a/infra/aws/ecs.tf +++ b/infra/aws/ecs.tf @@ -548,6 +548,20 @@ resource "aws_ecs_service" "execution" { desired_count = var.desired_count_execution launch_type = "FARGATE" + # Stop the old task before starting the new one, as the matcher already does. Two tasks sign with + # the SAME KMS key, and nothing coordinates the executor EOA's nonce across processes -- + # serial-queue.ts serialises within one process and says so: "viem reads the next nonce from the + # RPC at send time, so two sends in flight at once can take the same nonce and one replaces or + # rejects the other." With the AWS defaults (100/200) every rolling deploy opened that window: + # there is no load balancer, so Cloud Map returns an A record for BOTH tasks (MULTIVALUE, TTL 10s) + # and a settlement lands on either. + # + # The cost is a settlement gap of roughly a minute or two per deploy. That is lag, not loss: the + # matcher treats an unrecognised executor error as transient (internal/matching/revert.go, + # revertUnknown) and retries on a doubling backoff that never cancels an order. + deployment_minimum_healthy_percent = 0 + deployment_maximum_percent = 100 + network_configuration { subnets = aws_subnet.app[*].id security_groups = [aws_security_group.app.id] @@ -567,6 +581,13 @@ resource "aws_ecs_service" "market_maker" { desired_count = var.desired_count_market_maker launch_type = "FARGATE" + # Stop before start: two market makers quote the same book at once. This is not hypothetical -- + # a draining task kept placing orders for seconds after its successor had finished its startup + # reconciliation, leaving quotes resting under the previous configuration that nothing + # re-examined. Going dark for a minute between tasks is the cheaper failure. + deployment_minimum_healthy_percent = 0 + deployment_maximum_percent = 100 + network_configuration { subnets = aws_subnet.app[*].id security_groups = [aws_security_group.app.id]