Inject traceparent on egress so a trace survives the service hop - #95
Merged
Conversation
…fb312d) In 3 hours of production traffic: 34,961 single-service traces and 4 multi-service ones. Not because services don't call each other — because NOTHING injects W3C trace context on the way out. Every hop starts a new root, so a request crossing three services looks like three unrelated traces and "show me this request end to end" cannot be answered. The inbound half was already correct: api-prime EXTRACTs traceparent, and the observability SDK ships a reqwest middleware that injects. It had zero consumers. The missing piece is the client Rust services actually use for outbound calls — this one. Injects at `do_single_request`, so retries and redirects each carry a current traceparent rather than reusing a stale one. Behind an optional `otel` feature: default builds keep the dependency-free client, and callers that opt in get propagation with no code change. Context resolution is tracing-first, then `opentelemetry::Context::current()`. A subagent claimed the second read would be dead code under tracing-opentelemetry. I tested it rather than believing it — both are valid (tracing_span_context_valid=true otel_current_valid=true) — so the fallback stays for callers using the OTel API directly without a tracing subscriber. Two guards, both tested at the wire: - an invalid/absent span context injects NOTHING (no `00-000…-00` headers) - a caller-set traceparent is never overwritten 3 wire-level tests; 110 tests pass; clippy clean on both feature sets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 3565561 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…NET already does it
Same semantics in every language, because a trace that survives one hop and
breaks at the next is not much better than no trace:
- inject at the SINGLE-REQUEST site, inside the retry/timeout/breaker
wrappers, so each attempt carries a CURRENT traceparent instead of
re-sending a stale one
- never overwrite a caller-supplied traceparent
- emit NOTHING when there is no valid span context — never `00-000…-00`
Per-language notes worth keeping:
TypeScript (src/fetch.ts, in doGlobalFetch) — builds a COPY of the headers
rather than mutating `init`. Mutating would make attempt 2 see our own header as
"caller-supplied" and pin every retry to a stale traceparent. Optional peer dep
on @opentelemetry/api via a guarded dynamic import; verified the built bundles
keep it external.
Go (go/fetch/client.go, in executeHTTPRequest) — injected AFTER the pre-request
hook, because that hook may replace the whole *http.Request and would silently
discard a header injected before it. Pinned otel v1.38: v1.39+ declare go 1.24
and CI pins 1.23.
Python (_client.py, in _do_request) — copies request kwargs per attempt so
retries re-inject. Optional `[otel]` extra.
.NET — NO production change. HttpClient's DiagnosticsHandler already injects
from the current Activity, outside redirects and pooling, and this client keeps
that chain intact. Adding our own would double-inject. Tests pin the behaviour
so a future custom primary handler can't silently drop it; the negative control
(propagation disabled) fails 5 of 7, which is what makes the verdict worth
anything.
Every language is tested at the WIRE — a real local server, asserting the
headers it actually received — not against mocks. A mock that agrees with our
own bookkeeping is precisely the failure mode being fixed here. Each suite was
mutation-checked: remove a guard, watch the specific test fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The number that motivates this
In 3 hours of production traffic: 34,961 single-service traces and 4 multi-service ones.
Not because services don't call each other — because nothing injects W3C trace context on the way out. Every hop starts a new root, so a request crossing three services appears as three unrelated traces, and "show me this request end to end" cannot be answered.
Why here
The inbound half was already right: api-prime extracts
traceparent, and the observability SDK ships a reqwest middleware that injects — with zero consumers. The missing piece is the client Rust services actually use for outbound calls, which is this one.Injection happens at
do_single_request, so retries and redirects each carry a current traceparent instead of reusing a stale one.Opt-in
Behind an optional
otelfeature. Default builds keep the dependency-free client; callers that opt in get propagation with no code change.One thing I checked instead of believing
Context resolution is tracing-first, then
opentelemetry::Context::current(). A subagent told me that second read would be dead code undertracing-opentelemetry. I tested it — both are valid (tracing_span_context_valid=true otel_current_valid=true) — so the fallback stays for callers using the OTel API directly without a tracing subscriber.Guards, tested at the wire
00-000…-00headers)traceparentis never overwritten3 wire-level tests; 110 tests pass; clippy clean on both feature sets. Tracked in th-fb312d.