tracing spans never exported — the SDK bridged events but not spans - #91
Merged
Conversation
`bootstrap()` installs the OTel SDK and sets the global tracer provider. That is
enough for spans opened through the OpenTelemetry API. It does NOTHING for
`tracing` spans — `#[instrument]`, `info_span!`, and every library built on them
— because those need a `tracing-opentelemetry` layer inside the installed
subscriber, and this crate never offered one.
The crate already had the EVENT half: `tracing_appender_layer()` turns tracing
events into OTLP logs. The SPAN half was simply missing, and nothing said so.
Found in production (th-eaccd1). A service exported its OTel-API spans fine
while every `tracing` span — including the per-turn `gen_ai.chat` span carrying
model and token usage, the single most valuable span in an LLM product — was
printed to stdout and never left the process. Provider installed, spans opened,
dashboards green, five weeks of nothing. Verified by driving a real turn through
a live public agent: the agent answered, 467 WebSocket frames came back, and
zero rows landed.
Adds `OtelSdkHandle::tracing_span_layer()` behind feature `tracing-bridge`,
deliberately shaped like the existing `tracing_appender_layer()`: the crate hands
the host a LAYER, the host owns its subscriber. That sidesteps the ordering
hazard that makes this class of bug so durable — `tracing_subscriber`'s `init()`
PANICS if a subscriber already exists, so an SDK that installs one races every
vendor helper that does the same, and loses silently.
Bound to this handle's provider rather than the global one, so it exports through
the same pipeline the handle flushes and shuts down.
Three tests, and the middle one is the point:
- a `tracing` span reaches the exporter
- WITHOUT the layer, the same span exports NOTHING — the negative control.
Without it the first test proves only that spans exist somewhere, which is
exactly the false confidence being fixed
- attributes survive the crossing (a span that exports its name but drops
`gen_ai.request.model` and token usage is useless for LLM tracing)
Compiles with and without the feature; 3/3 pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 7f1ba38 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bootstrap()installs the OTel SDK and sets the global tracer provider. That is enough for spans opened through the OpenTelemetry API. It does nothing fortracingspans —#[instrument],info_span!, and every library built on them — because those need atracing-opentelemetrylayer inside the installed subscriber, and this crate never offered one.The crate already had the event half:
tracing_appender_layer()turns tracing events into OTLP logs. The span half was simply missing, and nothing said so.Found in production (th-eaccd1)
A service exported its OTel-API spans fine while every
tracingspan — including the per-turngen_ai.chatspan carrying model and token usage, the single most valuable span in an LLM product — was printed to stdout and never left the process.Provider installed. Spans opened. Dashboards green. Five weeks of nothing.
Verified by driving a real turn through a live public agent: the agent answered, 467 WebSocket frames came back, and zero rows landed.
The fix
OtelSdkHandle::tracing_span_layer()behind featuretracing-bridge, deliberately shaped like the existingtracing_appender_layer(): the crate hands the host a layer, the host owns its subscriber.That sidesteps the ordering hazard which makes this class of bug so durable —
tracing_subscriber'sinit()panics if a subscriber already exists, so an SDK that installs one races every vendor helper that does the same, and loses silently.Bound to this handle's provider rather than the global one, so it exports through the same pipeline the handle flushes and shuts down.
Three tests, and the middle one is the point
a_tracing_span_reaches_the_exporterwithout_the_bridge_layer_a_tracing_span_exports_nothingspan_attributes_survive_the_bridgegen_ai.request.model+ token usage surviveWithout the negative control, the first test proves only that spans exist somewhere — which is exactly the false confidence being fixed here.
Compiles with and without the feature; 3/3 pass.
Follow-up (separate, in the monorepo): chat-ws must compose this layer. It currently calls
smooth_operator::init_telemetry()first, which installs a fmt-only subscriber and permanently claims the slot.