feat(rebalance): buy cNGN back on HyperFX, and share the KMS signer - #69
Merged
Merged
Conversation
The market maker's flow is one-directional -- it sells cNGN for USDC -- so
without a routine that buys cNGN back the bid side eventually goes dark.
This is that routine, proven end to end on Base tonight:
19.9 USDC -> 27,262.280616 cNGN -> sub 15 (448,634 -> 475,896.280616)
Filled at 1369.96 against a rate-picker mid of ~1372, so about 15bps to
rebalance -- cheaper and far faster than a CEX round trip. Same chain, same
cNGN the venue settles (0x46C85152...EF5F), no bridge.
kms-signer moves out of services/execution into @numo/kms-signer rather than
being copied. It is the code that signs; two copies of it would drift, and
this repo's recurring mistake is fixing one instance of a thing. The 6 signer
tests move with it: execution goes 75 -> 69, the package has 6.
What is not obvious, and cost a working evening to find:
- @hyperbridge/sdk 2.8.13 (latest) prices this pair off the V1
phantomOrderPriceSnapshots table, which FROZE on 2026-08-08 at 1393.0 --
1,617 identical rows -- and validates only that the timestamp parses. The
live feed is phantomOrderPriceSnapshotV2s, which the SDK never references.
Quoting off V1 asks ~1.8% above market and never fills. quote.ts reads V2
and refuses anything past MAX_SNAPSHOT_AGE_SECONDS.
- executeBest submits the winning bid as an ERC-4337 UserOperation, so it
needs a bundler URL. Without one it places the order, fails instantly with
"Bundler URL not configured", and leaves the input escrowed. Alchemy serves
its Rundler bundler on the same URL as the RPC.
- The gateway rewrites the order: it assigns the nonce (we send 0) and
deducts the 5bps protocol fee from the input. The commitment hashes the
canonical struct, so cancel rebuilds from the indexer and checks the hash
before sending.
- cancelOrder takes a second CancelOptions argument; a one-argument call
reverts with no reason. Same-chain needs no destination proof, and refunds
the escrow AND the solver fee -- only the 5bps is kept, ~1 cent on $20.
- Read replicas lag both ways. A read after a receipt can be served
pre-block, and one pinned to the receipt's block can answer "Unknown
block". This produced three wrong conclusions in one evening: a 19.92 USDC
refund printed as +0, an approve that had succeeded made the next simulate
revert "ERC20: insufficient allowance", and a landed deposit reported an
error. waitFor() in clients.ts is the fix; never read latest and believe it.
- Status updates carry circular objects (Bid -> Swap -> UniswapQuoteEngine
-> adapter), so JSON.stringify on one kills a process holding escrowed
funds. brief() cannot throw.
The staleness guard, the no-bids guard and gross-vs-net pricing are covered by
tests that assert the guard fires, not just that the happy path works. Both new
packages are wired into services-execution.yml, a new services-rebalance.yml,
and scripts/verify.sh -- ./scripts/verify.sh node is green.
Signs with KMS alias/numo-exchange-rebalance, deliberately not the executor
key: a compromise here costs the float, not the venue's settlement authority.
Still manual: the withdrawal leg (sub 15 -> signer), which needs the market
maker's key or a signed /v1/withdrawals request.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
robertleifke
added a commit
that referenced
this pull request
Sep 20, 2026
…e it (#72) The execution image has not been buildable since #69. That PR moved the KMS signer into @numo/kms-signer, but services/execution/Dockerfile copies and builds only @numo/abis, so the service's own tsc fails inside the image: src/executor.ts(15,34): error TS2307: Cannot find module '@numo/kms-signer' ERROR: failed to build: ... exit code: 1 Nothing was deployed with it, so the running venue is unaffected. But the breakage is not limited to shipping new work: execution-service could not be rebuilt for ANY reason, including a rollback. The cause is narrow -- a workspace package's main/types resolve to dist/, which is gitignored (.gitignore:5 **/dist/), so copied-but-unbuilt means "Cannot find module" -- and the fix is to copy and build it like @numo/abis. The reason it reached main matters more. CI runs pnpm install + tsc at the WORKSPACE ROOT, where every package resolves through the root node_modules whatever the Dockerfile does. The image only has what it COPYs and builds. Those are different claims, and only one of them was checked, so green CI and an unbuildable image were entirely compatible -- the same "green means two different things" this repo wrote scripts/verify.sh to remove. So CI now builds the image (services-execution.yml), and verify.sh mirrors it, skipping with a note when docker is absent rather than failing -- the same shape as the terraform section, and for the same reason. A future workspace dependency that is not added to the Dockerfile now fails in CI instead of at deploy. Swept the class rather than the instance: services/markets is Go and uses COPY . ., so it cannot have this bug. It is the only other Dockerfile. The local build that proved the failure is reproduced in the message above; the verifying build of the fix stalled on registry metadata and was killed rather than waited out, so CI running the new step is the proof this works. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
robertleifke
added a commit
that referenced
this pull request
Sep 20, 2026
…leton services (#73) execution-service and market-maker-spot deployed with the ECS defaults, 100% minimum healthy and 200% maximum, so every rolling deploy ran two tasks at once. Both are singletons, and the matcher already sets 0/100 for exactly this reason. This applies the same setting to the two that were missed. execution-service: two tasks sign with the SAME KMS key, and nothing coordinates the executor EOA's nonce across processes. serial-queue.ts serialises within one process and states the hazard plainly -- "viem reads the next nonce from the RPC at send time, so two sends in flight at once can take the same nonce and one replaces or rejects the other". There is no load balancer to drain: Cloud Map returns an A record for BOTH tasks (MULTIVALUE, TTL 10s), so a settlement lands on whichever the matcher resolves. The deployment circuit breaker is disabled, so nothing would have caught it either. market-maker-spot: two market makers quote the same book. Already observed -- a draining task kept placing orders for seconds after its successor had finished its startup reconciliation, leaving quotes resting under the previous configuration that nothing re-examined. markets-service is deliberately left rolling. It is an HTTP API with no singleton constraint, so rolling avoids downtime and costs nothing. The cost is a gap of roughly a minute or two per deploy on those two services. For settlement that is lag, not loss: the matcher classifies an unrecognised executor error as transient (internal/matching/revert.go, revertUnknown is the zero value and documented as "treated like a transient failure") and retries on a doubling backoff -- 2s to a 5m cap, then every 30m after twelve failures -- that never cancels an order. A connection failure carries no revert selector, so it cannot be classified permanent and cannot park a pair. Applying this changes deploymentConfiguration only. It registers no task definition and forces no new deployment, so nothing restarts. Found while preparing the execution-service deploy of #71/#69: the rollout plan was a rolling one, and the nonce sequence is shared. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The market maker's flow is one-directional — it sells cNGN for USDC — so without a routine that buys cNGN back, the bid side eventually goes dark. This adds that routine, and moves
kms-signerinto a shared package rather than copying it.Proven end to end on Base tonight:
Filled at ₦1,369.96 against a rate-picker mid of ~₦1,372, so about 15 bps to rebalance. Same chain, same cNGN the venue settles, no bridge. The whole exercise — including three failed attempts — cost about 9 cents.
Why a shared package for the signer
kms-signer.tsmoves fromservices/execution/srcto@numo/kms-signer(as agit mv, so history follows). It is the code that signs; a second copy would drift, and fixing one instance of a thing is this repo's most-repeated mistake. The 6 signer tests move with it — execution goes 75 → 69, the new package has 6.Things worth knowing before reviewing
@hyperbridge/sdk2.8.13 (latest) prices this pair off the V1phantomOrderPriceSnapshotstable, which froze 2026-08-08 at 1393.0 — 1,617 identical rows — and validates only that the timestamp parses. The live feed isphantomOrderPriceSnapshotV2s, which the SDK never references. Quoting off V1 asks ~1.8% above market and never fills.executeBestsubmits the winning bid as an ERC-4337 UserOperation. Without a bundler URL it places the order, fails instantly withBundler URL not configured, and leaves the input escrowed. Alchemy serves its Rundler bundler on the same URL as the RPC.cancelrebuilds from the indexer and verifies the hash before sending.cancelOrdertakes two argumentsUnknown block. This produced three wrong conclusions in one evening — a 19.92 USDC refund printed as+0, an approve that had succeeded made the next simulate revertinsufficient allowance, and a landed deposit reported an error.waitFor()is the fix.Bid -> Swap -> UniswapQuoteEngine -> adapter.JSON.stringifyon one killed a process that was holding escrowed funds.brief()cannot throw.Verification
./scripts/verify.sh node→ all checks passed300 USDC → 410,494.65 cNGN)services-execution.yml, a newservices-rebalance.yml, andscripts/verify.sh— a check in only one place is what that script exists to preventKeys
Signs with KMS
alias/numo-exchange-rebalance→0x1661AA54fA390cd916722F971e4A9Fe4c01889fB. Deliberately not the executor key: a compromise here costs the float, not the venue's settlement authority.Not done
The withdrawal leg (sub 15 → signer). Sub 15 is owned by the market maker's wallet, so it needs either that key or a signed
POST /v1/withdrawals. Until then the loop is manual on that one step.🤖 Generated with Claude Code