Proposal: emit trace-correlated L3/L4 network flow spans for TCP, UDP, and ICMP

[ajimenez1503]

Summary

Hubble shows that L3/L4 flow telemetry can be aggregated into a useful service dependency graph. OBI already solves trace context propagation and can recover the trace ID across service boundaries.

This proposal is to extend OBI so that correlated network flows are emitted as spans in the current trace. The goal is not to replace cluster-wide topology tools like Hubble, but to bring transport-level visibility into per-request trace analysis.

Motivation

Today, OBI gives application-level spans and trace correlation, while network topology tools provide independent L3/L4 visibility. These two views are complementary but disconnected.

This creates a gap:

• application traces show logical operations, but not the actual transport path
• network tools show traffic edges, but not which flows belonged to a specific trace
• transport issues such as failed connects, resets, blackholes, asymmetric routing, or unexpected intermediaries are difficult to explain from a trace alone

Since OBI already has trace context propagation, it is in a good position to correlate network activity with the trace that caused it.

Proposal

Add an opt-in feature that emits L3/L4 network flow spans for TCP, UDP, and ICMP when the flow can be correlated to an existing trace.

This should work as follows:

• observe L3/L4 network flow activity through OBI’s existing network flow pipeline
• correlate a flow with an existing trace context when available
• aggregate packets/events into a bounded flow representation
• emit a network span into the same trace
• use protocol-aware aggregation semantics for TCP, UDP, and ICMP

Scope

• support TCP, UDP, and ICMP
• only emit spans for flows that can be correlated to a known trace
• aggregate multiple packets/events into one flow span rather than one span per packet
• keep the feature disabled by default
• bound span volume with aggregation and per-trace limits

Why this is different from Hubble

Hubble builds topology by aggregating observed traffic into workload/service edges. It is flow-first and cluster-wide.

This proposal is different:

• Hubble answers: “what is talking to what?”
• OBI should answer: “what transport-level flows were part of this trace?”

The value here is not another service map. The value is adding transport-level evidence directly into the distributed trace.

Configuration

We can add network to

  otel_traces_export:
    instrumentations: ["network"]

Acceptance criteria

• OBI can emit TCP, UDP, and ICMP network flow spans into an existing correlated trace
• spans are emitted only when correlation is available
• span count is bounded and aggregation-based
• traces show transport-level spans alongside application spans
• feature is disabled by default
• integration tests cover correlation and cardinality control

[ajimenez1503]

I imagine we will need to do some exploration and split that project in several tickets (with the knowledge of the OBI experts).
I want to mention that I would like to help on that project :)

[skl]

@mariomac mentioned that network flows can have multiple traces, so I wonder if an alternative (to emitting spans into all existing traces) would be to create a separate network-specific trace and use Span Links to link all traces in a flow to that network-specific trace?

So then you're injecting Span Links into multiple traces, and creating a single network-specific trace per flow, where you could potentially then correlate network traces to application traces?

[ajimenez1503]

@mariomac mentioned that network flows can have multiple traces, so I wonder if an alternative (to emitting spans into all existing traces) would be to create a separate network-specific trace and use Span Links to link all traces in a flow to that network-specific trace?

So then you're injecting Span Links into multiple traces, and creating a single network-specific trace per flow, where you could potentially then correlate network traces to application traces?

Hey @skl ,
Thank you for the comment. I can see the benefits also by reducing the amount of spans.

Create a separate network-specific trace

@skl, can we really build an individual network flow trace that correlate L3/L4 network interaction? Or that trace that you are suggesting would just have a span? Because if a network flow could be correlate with several application traces, how would we correlate the spans of an individual network flow trace?

What is your opinion @mariomac ?

@grcevski, in your Kubecon presentation you are adding DNS network flows into Application traces. Have you explore that? is that also happening in DNS that a DNS request could belong to several Application traces?

cc @MrAlias

[ajimenez1503]

Thank you @mariomac @skl @grcevski for the conversation yesterday in the OpenTelemetry eBPF Instrumentation SIG https://zoom.us/rec/play/o8oKgry5aR-kV04gTKdgqBUofKBwZ7SbtvZAm0t2MZ_nowG9hBGgTJFHa0tdc2EFW-hc-DDkFVdZEvpt.wLdZx_COg9DwoJm6, here is a short summary:

• Flows and traces are different abstraction layers, so a many-to-many relationship is expected: one network flow may map to multiple trace IDs, and one trace may involve multiple flows.
• Because of that, the idea of creating a separate network-only trace and linking application traces to it via Span Links is interesting, but it does not fully solve the modeling problem by itself. So, we will add the the network flow spans as part of the application layer.
• The goal remains similar in spirit to the DNS work: keep network visibility alongside the application trace so retries, reconnects, send/receive patterns, and other transport details help explain latency and failures that are otherwise invisible at the application layer.
• On the implementation side, the likely direction is a new opt-in generic network event pipeline that emits correlated TCP/UDP activity when a trace ID is known, and then builds spans from that data.
• This should be and explicitly enabled, otherwise the amount of noise will be too high.
• The plan would be to start with TCP, then UDP, and iterate from there before considering broader protocol coverage.

Proposed next step: split this into smaller follow-up issues and start with a TCP-focused POC to validate the event model, correlation behavior, retries/reconnects visibility, and span volume/cardinality controls.

Starting issue: #1781