[Enhancement]: Account for LLM calls made inside native routing plugins

[afourniernv]

Affected area

• Native plugins
• Observability and ATIF

Problem

A native routing plugin can make judge or classifier calls before returning one answer. Relay
prices the answer, but those internal calls are not part of the parent call's cost summary.

Relay can already price the internal calls if the plugin emits each one as an LLM scope. I tested
that on current main: ATIF correctly summed a $0.11 routing call and a $0.00045 answer to
$0.11045. The problem is that ATIF also turned both calls into normal source="agent" steps.
The routing call was never visible to the agent, so it should not appear as another agent message.

The other existing paths do not close the gap:

• A metric mark avoids the extra ATIF step, but Relay does not price it or include it in ATIF totals.
• Calling LlmNext::call twice reaches the provider twice, but Relay still emits one LLM start and
  end and prices only the returned response.
• Optimization contributions from request intercepts arrive before the routing calls have run.

Desired behavior

Provide a supported way for a native execution plugin to report internal LLM work so Relay can:

• price it with the existing pricing catalog;
• keep answer cost and internal-call cost separate;
• include both in an observed total when usage and pricing are available; and
• keep internal calls out of the ATIF conversation while retaining their cost in aggregate metrics.

If usage or pricing is missing, the total should remain partial rather than treating the call as
free. Calculating a savings baseline is separate from this issue.

Switchyard still owns emitting the model, provider, usage, and timing for its calls. This issue is
only about the Relay surface and export behavior needed to account for them.

Related: Relay #389,
Relay #814,
Switchyard #528, and
Switchyard #629.

[willkill07]

First, the overall issue seems inflated -- Did you do Child LLM scopes or Call-scoped optimization evidence? This if the issue evaluated only one, then please restrict this issue's scope. There's a hell of a lot going on here.

Two important distinctions also have to be made here:

1. What is an LLM Scope in the context of NeMo Relay?
   a. Option 1: an abstraction over what the agent interacts with
   b. Option 2: an exact provider request+response regardless of ownership/visibility semantics
2. What does NeMo Relay model with observability?
   a. Option 1: only what the agent sees.
   b. Option 2: every fine-grained detail of the total runtime.

In general, Option 2 would inflate the number of events and send each new event to every registered subscriber. Does this make sense? For the scope of this issue and for observability overall, I believe Option 1 is the most truthful. In my opinion, child LLM scopes with ATIF should simply be ignored. But for accurate cost reconciliation, we must have child costs get aggregated to the parent. This is a limitation of ATIF. But if you think about it, there is no clear correct answer here.

Explicit timestamp helper prototype
Because Switchyard receives completed routing-call observations, retrospective child scopes need explicit start and end timestamps to show honest durations. The native ABI already accepts these
timestamps, but the safe PluginRuntime helpers do not expose them.

This is confusing to me. Switchyard can create the scopes exactly when it needs to. We should not allow plugins to alter the source of truth of events. If they not have any sort of event recorder or callback, then they should consider adding one rather than emit static events. Realtime events and the ability to react instantly are probably things they will need eventually.

Runtime contract and binding impact

The initial producer is the Rust native Switchyard plugin.

If this is true, then this issue should live under Switchyard? It is unclear as to what changes there are to the runtime contract and binding impact itself here. I was hoping this section would be more definitive.

Details that need to be settled

Most of these are not Relay issues -- they are Switchyard issues. And again, in the scope of ATIF, Router LLM calls should not be present in ATIF due to opinionated perspective of Option 1 I declared above.

r.e. Cached input. Switchyard's normalized input_tokens excludes cached tokens. -- this makes sense simply because by default I would imagine a different model is in use and that there would be no caching. But really, this is dependent on whatever Switchyard does. They need to account for appropriate token mix when determining their costs. Relay will report whatever is given.

[afourniernv]

Thanks, this helped clarify the boundary. I was trying to make ATIF carry routing detail that fits better in OpenTelemetry.

The remaining Switchyard work is tracked in NVIDIA-NeMo/Switchyard#299.