Skip to content

feat(rebalance): buy cNGN back on HyperFX, and share the KMS signer - #69

Merged
robertleifke merged 1 commit into
mainfrom
feat/rebalance-service
Sep 18, 2026
Merged

robertleifke merged 1 commit into
mainfrom
feat/rebalance-service

Conversation

@robertleifke

Copy link
Copy Markdown
Contributor

What

The market maker's flow is one-directional — it sells cNGN for USDC — so without a routine that buys cNGN back, the bid side eventually goes dark. This adds that routine, and moves kms-signer into a shared package rather than copying it.

Proven end to end on Base tonight:

19.9 USDC -> 27,262.280616 cNGN -> sub 15  (448,634 -> 475,896.280616)

Filled at ₦1,369.96 against a rate-picker mid of ~₦1,372, so about 15 bps to rebalance. Same chain, same cNGN the venue settles, no bridge. The whole exercise — including three failed attempts — cost about 9 cents.

Why a shared package for the signer

kms-signer.ts moves from services/execution/src to @numo/kms-signer (as a git mv, so history follows). It is the code that signs; a second copy would drift, and fixing one instance of a thing is this repo's most-repeated mistake. The 6 signer tests move with it — execution goes 75 → 69, the new package has 6.

Things worth knowing before reviewing

Never use the SDK's quote here @hyperbridge/sdk 2.8.13 (latest) prices this pair off the V1 phantomOrderPriceSnapshots table, which froze 2026-08-08 at 1393.0 — 1,617 identical rows — and validates only that the timestamp parses. The live feed is phantomOrderPriceSnapshotV2s, which the SDK never references. Quoting off V1 asks ~1.8% above market and never fills.
A bundler is required executeBest submits the winning bid as an ERC-4337 UserOperation. Without a bundler URL it places the order, fails instantly with Bundler URL not configured, and leaves the input escrowed. Alchemy serves its Rundler bundler on the same URL as the RPC.
The gateway rewrites the order It assigns the nonce (we send 0) and deducts the 5 bps protocol fee from the input. The commitment hashes the canonical struct, so cancel rebuilds from the indexer and verifies the hash before sending.
cancelOrder takes two arguments A one-argument call reverts with no reason. Same-chain needs no destination proof, and recovery refunds the escrow and the solver fee — only the 5 bps is kept.
Read replicas lag both ways A read after a receipt can be served pre-block; one pinned to the receipt's block can answer Unknown block. This produced three wrong conclusions in one evening — a 19.92 USDC refund printed as +0, an approve that had succeeded made the next simulate revert insufficient allowance, and a landed deposit reported an error. waitFor() is the fix.
Status updates are circular Bid -> Swap -> UniswapQuoteEngine -> adapter. JSON.stringify on one killed a process that was holding escrowed funds. brief() cannot throw.

Verification

  • ./scripts/verify.sh nodeall checks passed
  • Tests assert the guards fire, not just the happy path: the staleness limit, the no-bids case, a zero-rounding quote, and gross-vs-net pricing (pinned to the real on-chain fill of 300 USDC → 410,494.65 cNGN)
  • All four CLI commands exercised as dry runs against live state
  • Both new packages wired into services-execution.yml, a new services-rebalance.yml, and scripts/verify.sh — a check in only one place is what that script exists to prevent

Keys

Signs with KMS alias/numo-exchange-rebalance0x1661AA54fA390cd916722F971e4A9Fe4c01889fB. Deliberately not the executor key: a compromise here costs the float, not the venue's settlement authority.

Not done

The withdrawal leg (sub 15 → signer). Sub 15 is owned by the market maker's wallet, so it needs either that key or a signed POST /v1/withdrawals. Until then the loop is manual on that one step.

🤖 Generated with Claude Code

The market maker's flow is one-directional -- it sells cNGN for USDC -- so
without a routine that buys cNGN back the bid side eventually goes dark.
This is that routine, proven end to end on Base tonight:

  19.9 USDC -> 27,262.280616 cNGN -> sub 15 (448,634 -> 475,896.280616)

Filled at 1369.96 against a rate-picker mid of ~1372, so about 15bps to
rebalance -- cheaper and far faster than a CEX round trip. Same chain, same
cNGN the venue settles (0x46C85152...EF5F), no bridge.

kms-signer moves out of services/execution into @numo/kms-signer rather than
being copied. It is the code that signs; two copies of it would drift, and
this repo's recurring mistake is fixing one instance of a thing. The 6 signer
tests move with it: execution goes 75 -> 69, the package has 6.

What is not obvious, and cost a working evening to find:

  - @hyperbridge/sdk 2.8.13 (latest) prices this pair off the V1
    phantomOrderPriceSnapshots table, which FROZE on 2026-08-08 at 1393.0 --
    1,617 identical rows -- and validates only that the timestamp parses. The
    live feed is phantomOrderPriceSnapshotV2s, which the SDK never references.
    Quoting off V1 asks ~1.8% above market and never fills. quote.ts reads V2
    and refuses anything past MAX_SNAPSHOT_AGE_SECONDS.
  - executeBest submits the winning bid as an ERC-4337 UserOperation, so it
    needs a bundler URL. Without one it places the order, fails instantly with
    "Bundler URL not configured", and leaves the input escrowed. Alchemy serves
    its Rundler bundler on the same URL as the RPC.
  - The gateway rewrites the order: it assigns the nonce (we send 0) and
    deducts the 5bps protocol fee from the input. The commitment hashes the
    canonical struct, so cancel rebuilds from the indexer and checks the hash
    before sending.
  - cancelOrder takes a second CancelOptions argument; a one-argument call
    reverts with no reason. Same-chain needs no destination proof, and refunds
    the escrow AND the solver fee -- only the 5bps is kept, ~1 cent on $20.
  - Read replicas lag both ways. A read after a receipt can be served
    pre-block, and one pinned to the receipt's block can answer "Unknown
    block". This produced three wrong conclusions in one evening: a 19.92 USDC
    refund printed as +0, an approve that had succeeded made the next simulate
    revert "ERC20: insufficient allowance", and a landed deposit reported an
    error. waitFor() in clients.ts is the fix; never read latest and believe it.
  - Status updates carry circular objects (Bid -> Swap -> UniswapQuoteEngine
    -> adapter), so JSON.stringify on one kills a process holding escrowed
    funds. brief() cannot throw.

The staleness guard, the no-bids guard and gross-vs-net pricing are covered by
tests that assert the guard fires, not just that the happy path works. Both new
packages are wired into services-execution.yml, a new services-rebalance.yml,
and scripts/verify.sh -- ./scripts/verify.sh node is green.

Signs with KMS alias/numo-exchange-rebalance, deliberately not the executor
key: a compromise here costs the float, not the venue's settlement authority.

Still manual: the withdrawal leg (sub 15 -> signer), which needs the market
maker's key or a signed /v1/withdrawals request.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@robertleifke
robertleifke merged commit ce8d4db into main Sep 18, 2026
2 checks passed
@robertleifke
robertleifke deleted the feat/rebalance-service branch September 18, 2026 05:15
robertleifke added a commit that referenced this pull request Sep 20, 2026
…e it (#72)

The execution image has not been buildable since #69. That PR moved the KMS
signer into @numo/kms-signer, but services/execution/Dockerfile copies and
builds only @numo/abis, so the service's own tsc fails inside the image:

  src/executor.ts(15,34): error TS2307: Cannot find module '@numo/kms-signer'
  ERROR: failed to build: ... exit code: 1

Nothing was deployed with it, so the running venue is unaffected. But the
breakage is not limited to shipping new work: execution-service could not be
rebuilt for ANY reason, including a rollback.

The cause is narrow -- a workspace package's main/types resolve to dist/, which
is gitignored (.gitignore:5 **/dist/), so copied-but-unbuilt means "Cannot find
module" -- and the fix is to copy and build it like @numo/abis.

The reason it reached main matters more. CI runs pnpm install + tsc at the
WORKSPACE ROOT, where every package resolves through the root node_modules
whatever the Dockerfile does. The image only has what it COPYs and builds. Those
are different claims, and only one of them was checked, so green CI and an
unbuildable image were entirely compatible -- the same "green means two
different things" this repo wrote scripts/verify.sh to remove.

So CI now builds the image (services-execution.yml), and verify.sh mirrors it,
skipping with a note when docker is absent rather than failing -- the same shape
as the terraform section, and for the same reason. A future workspace dependency
that is not added to the Dockerfile now fails in CI instead of at deploy.

Swept the class rather than the instance: services/markets is Go and uses
COPY . ., so it cannot have this bug. It is the only other Dockerfile.

The local build that proved the failure is reproduced in the message above; the
verifying build of the fix stalled on registry metadata and was killed rather
than waited out, so CI running the new step is the proof this works.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
robertleifke added a commit that referenced this pull request Sep 20, 2026
…leton services (#73)

execution-service and market-maker-spot deployed with the ECS defaults, 100%
minimum healthy and 200% maximum, so every rolling deploy ran two tasks at once.
Both are singletons, and the matcher already sets 0/100 for exactly this reason.
This applies the same setting to the two that were missed.

execution-service: two tasks sign with the SAME KMS key, and nothing coordinates
the executor EOA's nonce across processes. serial-queue.ts serialises within one
process and states the hazard plainly -- "viem reads the next nonce from the RPC
at send time, so two sends in flight at once can take the same nonce and one
replaces or rejects the other". There is no load balancer to drain: Cloud Map
returns an A record for BOTH tasks (MULTIVALUE, TTL 10s), so a settlement lands
on whichever the matcher resolves. The deployment circuit breaker is disabled, so
nothing would have caught it either.

market-maker-spot: two market makers quote the same book. Already observed --
a draining task kept placing orders for seconds after its successor had finished
its startup reconciliation, leaving quotes resting under the previous
configuration that nothing re-examined.

markets-service is deliberately left rolling. It is an HTTP API with no singleton
constraint, so rolling avoids downtime and costs nothing.

The cost is a gap of roughly a minute or two per deploy on those two services.
For settlement that is lag, not loss: the matcher classifies an unrecognised
executor error as transient (internal/matching/revert.go, revertUnknown is the
zero value and documented as "treated like a transient failure") and retries on a
doubling backoff -- 2s to a 5m cap, then every 30m after twelve failures -- that
never cancels an order. A connection failure carries no revert selector, so it
cannot be classified permanent and cannot park a pair.

Applying this changes deploymentConfiguration only. It registers no task
definition and forces no new deployment, so nothing restarts.

Found while preparing the execution-service deploy of #71/#69: the rollout plan
was a rolling one, and the nonce sequence is shared.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant