Conversation
Worker.Run reconciled one event at a time, so a controller process was
bounded by the latency of a single reconcile: several sequential
Substrate RPCs plus up to WorkspaceReadyTimeout (15s) of readiness
polling for a cold workspace. Throughput scaled only by adding pods.
Worker gains a Concurrency setting (ax-controller --concurrency,
default 16). Events are sharded to slots by a hash of atespace/name, so
events for one task stay ordered and never reconcile concurrently,
while unrelated tasks proceed in parallel. Each slot has a bounded
queue so a task with several queued events does not stall the others.
Events are still acknowledged only after processing, and shutdown
waits for in-flight reconciles.
BenchmarkWorkerThroughput (10ms per Substrate RPC, 200ms readiness
poll for cold workspaces, 64 tasks):
concurrency warm tasks/s cold tasks/s
1 15.2 3.8
4 57.5 14.2
16 160.0 40.2
64 309.3 79.8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the proposal in #384. Opening it so the code is easy to review; happy to change the approach or the default.
Problem
Worker.Runhandled events one at a time andax-controllerruns a single worker, so a controller process is bounded by the latency of one reconcile: a chain of sequential Substrate RPCs plus, for a cold workspace, up toWorkspaceReadyTimeout(15s) of/readyzpolling. Throughput only scaled by adding pods.Change
Worker.Concurrency, exposed asax-controller --concurrency(default 16). Values below 1 mean 1, so library callers that don't set it keep sequential behaviour.Runreturns.Benchmark
BenchmarkWorkerThroughputuses a fake Substrate at 10ms per RPC and a 200ms readiness poll standing in for the 15s default, 64 tasks per run, rebased on current main:Tests
TestWorkerConcurrencyKeepsTasksSerialqueues three revisions of each of 16 tasks back to back and asserts that no task ever has two Substrate calls in flight, and that the batch finishes well under the serial time. I checked that it fails when sharding is replaced with round-robin (3 concurrent calls for one task) and when concurrency is 1 (too slow).make testandgo test -race ./...pass.Open questions