fix(executor): scrape the execution client through a short network alias - #29
Open
kamilchodola wants to merge 1 commit into
Open
fix(executor): scrape the execution client through a short network alias#29kamilchodola wants to merge 1 commit into
kamilchodola wants to merge 1 commit into
Conversation
Alloy's prometheus.scrape target was the client's container name, which the scenario name can push past the 63-character DNS label limit. Docker's embedded DNS then never resolves it, the client is silently never scraped, and the run's cgroup CPU comes out ~4% lower than for a client with a shorter name: the exposition work Alloy would have triggered is missing, while block processing is untouched. Swapping tags between two bit-identical images moved the "saving" with the tag. Give the client an `execution-client` alias on the scenario network and point Alloy at that; warn when the container name itself exceeds the limit.
Contributor
Author
|
Validated on the amd64 reproducible-benchmarks box with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ExecutorConfig.get_execution_metrics_address()handed Alloy<client-container-name>:6060. In multi-image runs the client is namedexpb-executor-nethermind-multi-<tag>-run<N>-nethermind, which is 47 characters plus the tag. With a tag of 17+ characters the name exceeds the 63-character DNS label limit and Docker's embedded DNS cannot resolve it (verified: a 63-character name resolves, a 64-character one does not). Alloy then never scrapes the client and nothing reports it.The client that is not scraped skips the Prometheus exposition work, about 0.1 core for the whole container lifetime. On the amd64 reproducible-benchmarks box that is ~4% of the run's cgroup CPU with zero effect on block processing, i.e. exactly the shape of a plausible allocator or runtime improvement. It masqueraded as one for five benchmark batches until swapping tags between two bit-identical images moved the "saving" with the tag: the same image measured −0.4% under a 57-character name and −3.6% under a 69-character one. The whole difference sits in the post-benchmark idle minute (0.0 vs 0.4–0.5 CPU-s per 5 s), which is where scrape load shows.
Fix
execution-client, on the scenario network. Sincecontainers.run(network=...)cannot set aliases, the client is created, re-attached to the network with the alias, then started. Without a network the previousrunpath is kept.get_execution_metrics_address()returnsexecution-client:6060, so the Alloy config (rendered before the container exists) no longer depends on the container name.Validated locally with docker-py 7.x against a real daemon: a 69-character container resolves through the alias while its own name fails.
pytestandruffpass.Consequence for past results
Any multi-image CPU comparison on this rig in which one arm's tag had 17+ characters and another's did not is biased by ~4% in favour of the long-named arm. Latency numbers are unaffected.