Skip to content

perf(controller): reconcile tasks concurrently within one worker - #412

Open
anzal1 wants to merge 1 commit into
google:mainfrom
anzal1:perf/concurrent-worker
Open

anzal1 wants to merge 1 commit into
google:mainfrom
anzal1:perf/concurrent-worker

Conversation

@anzal1

@anzal1 anzal1 commented Sep 25, 2026

Copy link
Copy Markdown

Implements the proposal in #384. Opening it so the code is easy to review; happy to change the approach or the default.

Problem

Worker.Run handled events one at a time and ax-controller runs a single worker, so a controller process is bounded by the latency of one reconcile: a chain of sequential Substrate RPCs plus, for a cold workspace, up to WorkspaceReadyTimeout (15s) of /readyz polling. Throughput only scaled by adding pods.

Change

  • Worker.Concurrency, exposed as ax-controller --concurrency (default 16). Values below 1 mean 1, so library callers that don't set it keep sequential behaviour.
  • Events are sharded to slots by an FNV hash of atespace/name. Events for one task always go to the same slot, in order, so a task never reconciles concurrently with itself; unrelated tasks run in parallel.
  • Each slot has a bounded queue (64), so a task with several queued events doesn't stall the other slots. The reader blocks only when a slot's queue is full.
  • Events are still acknowledged only after processing. On shutdown the reader stops, and in-flight and queued events finish before Run returns.

Benchmark

BenchmarkWorkerThroughput uses a fake Substrate at 10ms per RPC and a 200ms readiness poll standing in for the 15s default, 64 tasks per run, rebased on current main:

concurrency warm tasks/s cold tasks/s
1 18.3 3.9
4 68.3 14.8
16 191.4 41.8
64 379.2 83.4
go test ./internal/controller -run '^$' -bench WorkerThroughput -benchtime 1x

Tests

TestWorkerConcurrencyKeepsTasksSerial queues three revisions of each of 16 tasks back to back and asserts that no task ever has two Substrate calls in flight, and that the batch finishes well under the serial time. I checked that it fails when sharding is replaced with round-robin (3 concurrent calls for one task) and when concurrency is 1 (too slow). make test and go test -race ./... pass.

Open questions

  • Is 16 a reasonable default, or would you prefer 1 (opt-in) until this has run against a real Substrate?
  • A natural follow-up is merging duplicate reconcile events for the same task while one is queued, since a reconcile always reads the latest spec.

Worker.Run reconciled one event at a time, so a controller process was
bounded by the latency of a single reconcile: several sequential
Substrate RPCs plus up to WorkspaceReadyTimeout (15s) of readiness
polling for a cold workspace. Throughput scaled only by adding pods.

Worker gains a Concurrency setting (ax-controller --concurrency,
default 16). Events are sharded to slots by a hash of atespace/name, so
events for one task stay ordered and never reconcile concurrently,
while unrelated tasks proceed in parallel. Each slot has a bounded
queue so a task with several queued events does not stall the others.
Events are still acknowledged only after processing, and shutdown
waits for in-flight reconciles.

BenchmarkWorkerThroughput (10ms per Substrate RPC, 200ms readiness
poll for cold workspaces, 64 tasks):

  concurrency   warm tasks/s   cold tasks/s
            1           15.2            3.8
            4           57.5           14.2
           16          160.0           40.2
           64          309.3           79.8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant