Skip to content

fix(executor): scrape the execution client through a short network alias - #29

Open
kamilchodola wants to merge 1 commit into
mainfrom
fix/scrape-client-by-network-alias
Open

fix(executor): scrape the execution client through a short network alias#29
kamilchodola wants to merge 1 commit into
mainfrom
fix/scrape-client-by-network-alias

Conversation

@kamilchodola

Copy link
Copy Markdown
Contributor

Problem

ExecutorConfig.get_execution_metrics_address() handed Alloy <client-container-name>:6060. In multi-image runs the client is named expb-executor-nethermind-multi-<tag>-run<N>-nethermind, which is 47 characters plus the tag. With a tag of 17+ characters the name exceeds the 63-character DNS label limit and Docker's embedded DNS cannot resolve it (verified: a 63-character name resolves, a 64-character one does not). Alloy then never scrapes the client and nothing reports it.

The client that is not scraped skips the Prometheus exposition work, about 0.1 core for the whole container lifetime. On the amd64 reproducible-benchmarks box that is ~4% of the run's cgroup CPU with zero effect on block processing, i.e. exactly the shape of a plausible allocator or runtime improvement. It masqueraded as one for five benchmark batches until swapping tags between two bit-identical images moved the "saving" with the tag: the same image measured −0.4% under a 57-character name and −3.6% under a 69-character one. The whole difference sits in the post-benchmark idle minute (0.0 vs 0.4–0.5 CPU-s per 5 s), which is where scrape load shows.

Fix

  • The client container gets a short alias, execution-client, on the scenario network. Since containers.run(network=...) cannot set aliases, the client is created, re-attached to the network with the alias, then started. Without a network the previous run path is kept.
  • get_execution_metrics_address() returns execution-client:6060, so the Alloy config (rendered before the container exists) no longer depends on the container name.
  • A warning is logged when the container name exceeds 63 characters, for anything else that might address it by name. k6 and the payload server already use the container's IP.

Validated locally with docker-py 7.x against a real daemon: a 69-character container resolves through the alias while its own name fails. pytest and ruff pass.

Consequence for past results

Any multi-image CPU comparison on this rig in which one arm's tag had 17+ characters and another's did not is biased by ~4% in favour of the long-named arm. Latency numbers are unaffected.

Alloy's prometheus.scrape target was the client's container name, which the
scenario name can push past the 63-character DNS label limit. Docker's
embedded DNS then never resolves it, the client is silently never scraped,
and the run's cgroup CPU comes out ~4% lower than for a client with a shorter
name: the exposition work Alloy would have triggered is missing, while block
processing is untouched. Swapping tags between two bit-identical images moved
the "saving" with the tag.

Give the client an `execution-client` alias on the scenario network and point
Alloy at that; warn when the container name itself exceeds the limit.
@kamilchodola

Copy link
Copy Markdown
Contributor Author

Validated on the amd64 reproducible-benchmarks box with expb_branch=fix/scrape-client-by-network-alias (nethermind runs 34219599850 and 34219631846, fusaka 1000 blocks, n=6 per arm, mirrored orders). The 64-character arm jemalloc-complete, which measured −4.0% to −4.6% CPU against alloc5-private in every batch on main, now measures +0.27% (orders +0.10 / +0.44); the 63-character arm complete-build21 is at −0.27%. The expb warning about the over-long container name fires for the 64-character arm, and its post-benchmark idle phase now carries the same scrape load as the short-named arms. Not self-merging; ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant