From 5dfbc2c41d644d6f38dd9ce0391732735c2474a5 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Fri, 4 Sep 2026 10:56:58 -0700 Subject: [PATCH 1/9] docs(relay): document the native integration Signed-off-by: Alex Fournier --- docs/index.md | 4 +- docs/integrations/nemo_relay.md | 315 ++++++++++++++++++++++++++++++++ mkdocs.yml | 2 + 3 files changed, 319 insertions(+), 2 deletions(-) create mode 100644 docs/integrations/nemo_relay.md diff --git a/docs/index.md b/docs/index.md index 9b8fb17f9..a3c480ece 100644 --- a/docs/index.md +++ b/docs/index.md @@ -9,7 +9,7 @@ It supports OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. |---|---|---| | Run Switchyard as a standalone proxy for API clients | Server Path | [Build and run the Rust server](getting_started.md#server-path) | | Add Switchyard routing to a Rust application | Library Path | [`switchyard-libsy`](../crates/libsy/README.md) | -| Add Switchyard routing to NeMo Relay | Native Plugin Path | [`switchyard-nemo-relay-plugin`](../crates/switchyard-nemo-relay-plugin/README.md) | +| Add Switchyard routing to NeMo Relay | Native Plugin Path | [Use Switchyard with NeMo Relay](integrations/nemo_relay.md) | The Server Path builds and runs the standalone `switchyard-server` binary. @@ -28,4 +28,4 @@ The Server Path builds and runs the standalone `switchyard-server` binary. - [`switchyard-libsy`](reference/rust_api.md#switchyard-libsy): embeddable routing algorithms - [`switchyard-protocol`](reference/rust_api.md#switchyard-protocol): provider-neutral API types - [`switchyard-translation`](../crates/switchyard-translation/README.md): protocol translation -- [`switchyard-nemo-relay-plugin`](../crates/switchyard-nemo-relay-plugin/README.md): native NeMo Relay integration +- [`switchyard-nemo-relay-plugin`](../crates/switchyard-nemo-relay-plugin/README.md): native plugin build and configuration diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md new file mode 100644 index 000000000..175061f26 --- /dev/null +++ b/docs/integrations/nemo_relay.md @@ -0,0 +1,315 @@ +# Use Switchyard with NeMo Relay + +Use the +[Switchyard native plugin](../../crates/switchyard-nemo-relay-plugin/README.md) +to add model routing to an existing +[NeMo Relay](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/overview#integrating-with-relay) +deployment without running a second service. It runs through Relay's +[native plugin system](https://docs.nvidia.com/nemo/relay/v0.8.3/build-plugins/native/about). +Relay receives the application's model request and keeps its normal +[middleware](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#what-middleware-is) +and +[observability](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/about#shortest-path). +Switchyard chooses and calls the configured model target. + +Use this integration when Relay already handles your agent's model calls and +you want to: + +- present one route name while Switchyard chooses the model; +- reuse the same Switchyard TOML deployment as `switchyard-server`; +- inspect routing decisions and model attempts through Relay's existing + telemetry; and +- keep Relay's existing handling for models that Switchyard does not manage. + +If Relay is not part of the application, run the [standalone server](../getting_started.md#server-path) +or embed [`switchyard-libsy`](../../crates/libsy/README.md) directly. + +## How Requests Flow + +Relay loads the plugin into its own process. For each supported model request, +the plugin checks whether the requested model matches a configured Switchyard +route. Matching requests go through Switchyard. Other requests are left +unchanged by Switchyard and passed to the next Relay handler. The plugin uses +Relay's +[execution intercepts](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#execution-intercepts) +for non-streaming requests and +[stream execution intercepts](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#stream-execution-intercepts) +for streaming requests. + +```mermaid +flowchart TB + caller["Agent or SDK"] + + subgraph relay_process["NeMo Relay process"] + relay["Relay LLM execution"] + relay --> match{"Model matches a
Switchyard route ID?"} + match -->|"No"| next["Next Relay handler"] + match -->|"Yes"| plugin["Native Switchyard plugin
decode, route, translate, and call"] + next --> relay_out + plugin -.->|"Routing observations"| telemetry["Relay marks and metrics"] + plugin -->|"Translated output"| relay_out["Relay returns response or stream"] + end + + target["Configured model provider"] + caller --> relay + plugin <-->|"Target call"| target +``` + +For a matching request, Switchyard performs model selection, provider calls, +retries, and fallback itself. Model calls used to make a routing decision, as +well as calls to the selected or fallback answer model, do not run through +Relay's LLM middleware again. This avoids treating a router's judge call or +fallback attempt as another application request. + +Relay still records the caller-facing +[LLM call](https://docs.nvidia.com/nemo/relay/v0.8.3/instrument-applications/instrument-llm-call#integration-pattern), +while Switchyard adds +[marks](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#mark) +and metrics for the routing work inside it. + +| Owner | Responsibilities | +| --- | --- | +| Relay | Receives the caller's request, runs Relay middleware, returns the response or stream, and exports telemetry. | +| Switchyard | Chooses a target, translates formats, calls the provider, and handles configured retries and fallback. | +| Model provider | Runs the model and returns its response, stream, and available usage. | + +For more detail, see Relay's +[managed execution pipeline](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/architecture#managed-execution-pipeline) +and +[plugin delivery models](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/plugins#plugin-delivery-models). + +## Set Up the Plugin + +Follow the [plugin README](../../crates/switchyard-nemo-relay-plugin/README.md) +to build and package the native library, register and enable it in Relay, and +configure its deployment. Relay documents how to +[add and enable a discoverable plugin](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/discoverable-plugins#add-and-enable-a-plugin) +and how it +[validates the package before loading code](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/discoverable-plugins#validate-before-loading-code). + +!!! note "Relay compatibility" + + You do not need Relay 0.8.3 specifically. The packaged + [`relay-plugin.toml`](../../crates/switchyard-nemo-relay-plugin/relay-plugin.toml) + is the source of truth: it currently accepts Relay `>=0.8.1,<0.9.0` and + native plugin API `1`. Relay checks both before loading the library. Links + on this page point to version 0.8.3 so the documentation does not drift to + an unsupported Relay release. + +The plugin accepts exactly one Switchyard deployment source: a +`switchyard_config_path` shared with `switchyard-server`, or the same version-1 +deployment nested under `switchyard_config`. Both use the +[Switchyard TOML schema](../reference/toml_schema.md). + +The plugin reuses the deployment's routes, targets, and LLM clients. It does not +use Switchyard's `fallback_client` for an unmatched model. Relay's next handler +decides what happens to that request. + +The `id` of each configured Switchyard route becomes a model name that callers +can send through Relay. No additional Relay route table is required for those +model names. + +## Request Handling + +The plugin handles these Relay LLM calls: + +- OpenAI Chat Completions (`openai.chat_completions`) +- OpenAI Responses (`openai.responses`) +- Anthropic Messages (`anthropic.messages`) + +Only requests whose `model` is a string matching a Switchyard route ID are +routed. Other call types, missing or non-string model values, and unconfigured +model names are left unchanged by Switchyard and passed to Relay's next handler. + +The caller and selected target may use different supported API formats. +Switchyard normalizes the request, routes it, and returns the response in the +caller's original format. If Switchyard forwards the caller's credential, both +formats must use the same credential family: OpenAI-compatible or Anthropic. + +Not every provider-specific field has a lossless equivalent. Switchyard rejects +a conversion it cannot perform safely instead of silently dropping data. + +### Streaming + +For streaming requests, Switchyard hands Relay a lazy translated stream. Relay +drives delivery and cancellation and records when the caller-facing stream +starts and ends. As Relay consumes the stream, Switchyard continues to translate +chunks and record late usage or errors. + +- Initial routing marks are available when the stream opens. +- An answer-call result of `ok` means the provider opened the stream. It does + not guarantee that the full stream completed. +- For an upstream response that remains streamed, answer token metrics appear + only if the provider reports usage and the stream reaches its final event. A + canceled or dropped stream may have no answer-token metrics. +- Later provider failures can emit `switchyard.routing.error`. Some failures + before routing or while encoding Relay output have no Switchyard mark, so the + marks are not a complete request-failure log. +- If Relay rejects a telemetry event, the plugin writes the error to standard + error and still returns the model response. + +## State and Identity + +The plugin keeps request and response data in memory only while handling the +call. For a stream, that data remains until the stream finishes or the caller +drops it. The plugin does not store these payloads on disk. + +The plugin creates one Switchyard runner when Relay activates it and shares the +runner across requests until the plugin is deactivated. Some routing algorithms +keep in-memory state there, such as session affinity or an escalation decision. +Each algorithm controls when that state expires. The state is not shared between +Relay processes and is lost when a process restarts. See Relay's documentation +on +[plugin ownership](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/plugins#ownership-and-scope) +and +[runtime state](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/architecture#where-runtime-state-lives). + +To associate routing telemetry with the rest of an agent run, the plugin adds +these fields to every routing mark, including marks that carry metric +measurements. Missing values are `null`: + +- `session_id` +- `agent_id` +- `parent_agent_id` +- `task_id` +- `turn_id` +- `correlation_id` + +These fields are event metadata used to correlate routing records with a run. +Subscribers and log or trace exporters can read them, but Relay does not copy +them into exported metric attributes. + +These values come from request headers rather than Relay's active scope. Relay's +[session and subagent headers](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#runtime-mapping) +can populate them for correlation. They do not by themselves mark a request as +delegated work for Switchyard's +[`subagents` router](../routing_algorithms/subagent_routing.md). For algorithms +that keep per-session state, reuse a stable session ID across turns. Relay's +`x-nemo-relay-session-id` header is accepted; `x-switchyard-session-id` +provides an explicit override. If Relay's gateway has no stable session ID, the +plugin leaves the Switchyard session ID unset. Send `x-switchyard-session-id` +when an algorithm must keep the same per-session state across turns. + +## Routing Telemetry + +Switchyard sends its routing records into the same Relay telemetry stream as +the caller-facing LLM call. Existing Relay +[subscribers](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/subscribers#how-subscribers-relate-to-events) +can export both, so a separate Switchyard telemetry pipeline is not required. +How those records appear in a backend depends on Relay's +[OpenTelemetry trace projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections) +and +[OpenInference projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/openinference#plugin-configuration). + +### How Routing Appears in Traces + +When the caller-facing LLM call uses Relay's active agent scope as its parent, +the Switchyard marks use that same scope and appear alongside the call. +Switchyard does not create another nested scope. For Relay's +[full and OpenInference trace projections](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections), +[`mark_projection`](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-endpoint-fields) +controls whether a backend displays each eligible mark as an event on the +parent or as a visible zero-duration child span. The +[`gen_ai` projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#genai-projection) +omits marks. With `mark_projection = "tool"`, the trace has this shape: + +```mermaid +flowchart LR + agent["Relay agent scope"] + llm["LLM call"] + requested["switchyard.routing.requested"] + calls["switchyard.routing.llm_call
routing and answer attempts"] + overhead["switchyard.routing.overhead"] + decision["switchyard.routing.decision"] + error["switchyard.routing.error
when execution fails"] + agent --> llm + agent --> requested + agent --> calls + agent --> overhead + agent --> decision + agent -.->|"Failure"| error +``` + +If the application did not create an agent scope, an observability backend can +display the LLM span and marks as separate roots. The metadata field +`parent_agent_id` is a correlation value; it does not set Relay trace +parentage. See Relay's +[scope hierarchy](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/scopes#scope-hierarchy-and-ownership) +for the parentage rules. + +### Mark Contract + +Dashboards and subscribers can use `data_schema` to identify the payload +contract. Each non-metric mark uses the mark name as its schema name and version +`1`. Consumers should accept additional fields and values within a version. +Removing or renaming a field, changing its type, or changing its meaning +requires a new version. Relay's +[event envelope](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#fields-common-to-every-event) +describes the surrounding event envelope. + +| Mark | Severity | Data | +| --- | --- | --- | +| `switchyard.routing.requested` | Info | Routing `algorithm` for a managed request. | +| `switchyard.routing.llm_call` | Debug | `call_index`, model in `selected_model`, `call_role` (`routing` or `answer`), `outcome`, and `latency_ms` for each observed model call. | +| `switchyard.routing.overhead` | Info | `latency_ms` spent producing the routing outcome, including routing-model calls. This is not the end-to-end request duration. | +| `switchyard.routing.decision` | Info | `algorithm`, initial `selected_model`, nullable final `served_model`, and nullable `fallback_used`. | +| `switchyard.routing.error` | Error | Generic failures contain `failure_kind`. Route-execution failures also contain `category` and `phase`, plus nullable `upstream_status` and `target`. | + +Call marks describe Switchyard observations, not every HTTP retry made inside a +client. `call_role` records whether Switchyard classified the call as routing +or answer work. + +`switchyard.routing.llm_call` uses Debug severity. It still appears in the +supported trace projections, but Relay's OTLP logs default to Info. Set +`minimum_severity` to `debug` to include these call records in +[log export](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#log-export). + +Model fields use the target's upstream model ID, not its local TOML key. For +example, if `[targets.fast].id = "provider/model-a"`, the mark records +`provider/model-a`, not `fast`. Relay records the model value after its request +middleware runs, normally the Switchyard route ID. On the response, Relay can +report the model that actually answered. + +`fallback_used` is `true` when the final served model differs from the initial +selection and `false` when they match. It and `served_model` are `null` when the +response does not provide serving metadata. If route execution fails before a +response is available, the error mark describes the terminal failure instead. + +### Metrics + +| Metric | Kind and unit | Meaning and attributes | +| --- | --- | --- | +| `switchyard.routing.requests` | Counter, events | Managed requests, labeled by `algorithm`. | +| `switchyard.routing.llm_calls` | Counter, events | Routing-model calls, labeled by `outcome`. | +| `switchyard.routing.llm_call.duration` | Histogram, milliseconds | Routing-model call duration, labeled by `outcome`. | +| `switchyard.routing.overhead` | Histogram, milliseconds | Time spent producing the routing outcome. | +| `switchyard.routing.llm_tokens` | Counter, tokens | Provider-reported token values, labeled by `call_role`, `target_model`, and `token_type`. | +| `switchyard.routing.failures` | Counter, events | Terminal failures, labeled by safe failure kind and available classification fields. | + +`switchyard.routing.llm_calls` and `switchyard.routing.llm_call.duration` cover +routing-model calls only. Answer calls appear in the per-call marks and token +metrics. + +Token metrics cover routing and answer calls when the provider reports usage. +The plugin does not synthesize zeroes for missing values. The supported token +types are `input`, `cached_input`, `cache_creation_input`, `output`, +`reasoning`, and `total`. + +Configure delivery through Relay's +[OpenTelemetry metric export](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#metric-export). + +## Data Handling + +Switchyard routing telemetry contains a small, defined set of routing fields, +not request or response content. Its marks do not contain prompts, request or +response bodies, headers, credentials, raw provider response bodies, or +free-form provider error messages. Relay's caller-facing LLM events can +capture request and response data according to Relay's +[input and output event semantics](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#input-and-output-payloads), +independently of these Switchyard marks. + +Header forwarding is part of request execution, not telemetry. Caller headers +are forwarded upstream except credentials and headers owned by the HTTP client, +such as connection and content headers. Authentication and configured extra +headers follow the selected client's settings in the +[TOML schema](../reference/toml_schema.md). diff --git a/mkdocs.yml b/mkdocs.yml index 27bf9d444..5d99499d9 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -23,6 +23,8 @@ nav: - Concepts: - Core Concepts: core_concepts.md - Architecture: architecture.md + - Integrations: + - NeMo Relay: integrations/nemo_relay.md - Routing: - Overview: routing_algorithms/overview.md - Sub-Agent-Aware Routing: routing_algorithms/subagent_routing.md From 0bb33d414be66eb06c4c1a0eba547b1e3b2234e2 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Fri, 4 Sep 2026 19:33:38 -0700 Subject: [PATCH 2/9] docs(relay): explain routing value and cost measurement Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 38 +++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 175061f26..3af3cf256 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -12,15 +12,39 @@ and [observability](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/about#shortest-path). Switchyard chooses and calls the configured model target. -Use this integration when Relay already handles your agent's model calls and -you want to: - -- present one route name while Switchyard chooses the model; -- reuse the same Switchyard TOML deployment as `switchyard-server`; -- inspect routing decisions and model attempts through Relay's existing - telemetry; and +## Why Use Switchyard with NeMo Relay? + +Switchyard decides which model should handle each request. Relay keeps the +caller-facing request in its normal lifecycle and carries Switchyard's routing +records with the rest of the agent run. Together, they add routing without +changing the agent or running a separate Switchyard service. + +This lets you: + +- with an adaptive route, send routine work to a lower-cost model while keeping + a stronger model available for harder turns; +- compare a routed deployment with a fixed-model baseline on the same workload; +- see the initial selection, final model when reported, fallbacks, routing time, + and failures; +- separate tokens spent choosing a model from tokens spent generating the + answer; and - keep Relay's existing handling for models that Switchyard does not manage. +### Measure Routing Cost on Your Workload + +The plugin reports provider-supplied token usage separately for routing and +answer calls, labeled by target model and token type. It also reports routing +time, the initial selection, the final model when reported, fallbacks, and +failures. Relay can export those records through its existing telemetry +pipeline. + +Relay can [estimate cost for the caller-facing response](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) +when model pricing is configured. Internal routing calls do not run as separate +Relay LLM calls, so include the plugin's routing-token metrics when calculating +the total cost. Compare the same workload against a fixed-model route to see +whether the savings outweigh the routing work. Missing provider usage is +unknown, not zero. + If Relay is not part of the application, run the [standalone server](../getting_started.md#server-path) or embed [`switchyard-libsy`](../../crates/libsy/README.md) directly. From 0af47236141a8d739937e0d8929dbf7f4ee0e1b7 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Sun, 6 Sep 2026 08:28:40 -0700 Subject: [PATCH 3/9] docs(relay): simplify integration overview Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 73 +++++++++++++++------------------ 1 file changed, 34 insertions(+), 39 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 3af3cf256..86ef588d8 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -1,49 +1,44 @@ # Use Switchyard with NeMo Relay -Use the -[Switchyard native plugin](../../crates/switchyard-nemo-relay-plugin/README.md) -to add model routing to an existing +The [Switchyard native plugin](../../crates/switchyard-nemo-relay-plugin/README.md) +loads Switchyard into an existing [NeMo Relay](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/overview#integrating-with-relay) -deployment without running a second service. It runs through Relay's +deployment through Relay's [native plugin system](https://docs.nvidia.com/nemo/relay/v0.8.3/build-plugins/native/about). -Relay receives the application's model request and keeps its normal -[middleware](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#what-middleware-is) -and -[observability](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/about#shortest-path). -Switchyard chooses and calls the configured model target. ## Why Use Switchyard with NeMo Relay? -Switchyard decides which model should handle each request. Relay keeps the -caller-facing request in its normal lifecycle and carries Switchyard's routing -records with the rest of the agent run. Together, they add routing without -changing the agent or running a separate Switchyard service. - -This lets you: - -- with an adaptive route, send routine work to a lower-cost model while keeping - a stronger model available for harder turns; -- compare a routed deployment with a fixed-model baseline on the same workload; -- see the initial selection, final model when reported, fallbacks, routing time, - and failures; -- separate tokens spent choosing a model from tokens spent generating the - answer; and -- keep Relay's existing handling for models that Switchyard does not manage. - -### Measure Routing Cost on Your Workload - -The plugin reports provider-supplied token usage separately for routing and -answer calls, labeled by target model and token type. It also reports routing -time, the initial selection, the final model when reported, fallbacks, and -failures. Relay can export those records through its existing telemetry -pipeline. - -Relay can [estimate cost for the caller-facing response](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) -when model pricing is configured. Internal routing calls do not run as separate -Relay LLM calls, so include the plugin's routing-token metrics when calculating -the total cost. Compare the same workload against a fixed-model route to see -whether the savings outweigh the routing work. Missing provider usage is -unknown, not zero. +Relay receives your agent's model request. Switchyard picks and calls the model. +Relay continues to track the rest of the agent run and can export the routing +details from Switchyard. Because the plugin runs inside Relay, you do not need +to change the agent or run a separate Switchyard service. + +Together, they let you: + +- send simpler work to a less expensive model and keep a stronger model for + harder work; +- see which model was chosen and, when available, which model answered after a + fallback; +- track routing time, failures, and token use alongside the rest of the agent + run; and +- compare routing with using one model for every request. + +Requests for models that Switchyard does not manage continue through Relay as +usual. + +### Check Whether Routing Saves Money + +Routing is not free. Some routes call another model to help choose the model +that will answer. The plugin records the tokens used to make that choice +separately from the tokens used for the answer. It also records the routing +time, model choice, fallback, and failures. + +Relay can [estimate the cost of the response returned to the agent](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) +when model pricing is configured. It does not automatically price Switchyard's +internal routing calls, so include the token counts from those calls when +calculating the total. Run the same work once with a fixed model and once with +routing to see whether routing actually saved money. If a provider does not +report usage, the cost is unknown rather than zero. If Relay is not part of the application, run the [standalone server](../getting_started.md#server-path) or embed [`switchyard-libsy`](../../crates/libsy/README.md) directly. From 0803b0495ca43058645072391de539bb5a227201 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Sun, 6 Sep 2026 08:49:46 -0700 Subject: [PATCH 4/9] docs(relay): clarify routing cost accounting Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 82 +++++++++++++++++++++++---------- 1 file changed, 58 insertions(+), 24 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 86ef588d8..937507a78 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -8,37 +8,71 @@ deployment through Relay's ## Why Use Switchyard with NeMo Relay? -Relay receives your agent's model request. Switchyard picks and calls the model. -Relay continues to track the rest of the agent run and can export the routing -details from Switchyard. Because the plugin runs inside Relay, you do not need -to change the agent or run a separate Switchyard service. +Using the strongest model for every request is simple, but routine work may not +need it. Switchyard can send simpler requests to a lower-cost model and reserve +a stronger model for harder work. -Together, they let you: +Relay keeps the model choice, fallback, latency, and token use alongside the +rest of the agent run. This lets teams answer practical questions: -- send simpler work to a less expensive model and keep a stronger model for - harder work; -- see which model was chosen and, when available, which model answered after a - fallback; -- track routing time, failures, and token use alongside the rest of the agent - run; and -- compare routing with using one model for every request. +- How often did the lower-cost model handle the work? +- How often did Switchyard fall back to another model? +- How much time and token use did routing add? +- Did the routed workload cost less than using one model for every request? + +The plugin runs inside Relay, so the agent does not need to change and +Switchyard does not need to run as a separate service. Requests for models that Switchyard does not manage continue through Relay as usual. -### Check Whether Routing Saves Money +### Measure the Cost of Routing + +A routed request can spend tokens in two places. Some routes call a judge or +classifier to choose a model. The selected model, or a fallback, then produces +the answer. Relay and Switchyard record these parts separately. + +Relay records the caller-facing request and response. In OpenInference traces, +answer tokens appear as `llm.token_count.prompt`, `llm.token_count.completion`, +and `llm.token_count.total`. When available, cache use appears as +`llm.token_count.prompt_details.cache_read` and +`llm.token_count.prompt_details.cache_write`. Relay also records +`llm.cost.total` in USD when the provider reports a cost, or when the response +contains enough model and usage data and a configured Relay pricing catalog +has all required rates. + +Switchyard records token use for its model calls in the +`switchyard.routing.llm_tokens` metric when the provider reports usage. Each +measurement identifies: + +- `call_role`: `routing` or `answer` +- `target_model`: the upstream model ID +- `token_type`: `input`, `cached_input`, `cache_creation_input`, `output`, + `reasoning`, or `total` + +Switchyard also records the selected and served models, fallback use, +routing-call latency, routing overhead, and failures. + +No single total-cost or savings metric is emitted today. Relay prices the +caller-facing answer, but it does not automatically price Switchyard's internal +routing calls. For the common case, calculate the observed routed cost as: + +```text +Relay answer cost + cost of Switchyard tokens where call_role = "routing" +``` -Routing is not free. Some routes call another model to help choose the model -that will answer. The plugin records the tokens used to make that choice -separately from the tokens used for the answer. It also records the routing -time, model choice, fallback, and failures. +Apply the price of each `target_model` to those routing tokens. Do not add +`call_role = "answer"` again because it describes the same successful answer +that Relay recorded. Do not sum every `token_type`; `total` is a rollup, and +cache or reasoning values may be a more detailed view of another count. -Relay can [estimate the cost of the response returned to the agent](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) -when model pricing is configured. It does not automatically price Switchyard's -internal routing calls, so include the token counts from those calls when -calculating the total. Run the same work once with a fixed model and once with -routing to see whether routing actually saved money. If a provider does not -report usage, the cost is unknown rather than zero. +Relay does not ship a canonical price catalog. Follow its +[model-pricing guide](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) +to supply and validate model rates. Run the same representative workload once +with a fixed model and once with routing, then compare observed cost, latency, +fallbacks, and your task-success measure. Missing usage, failed attempts, +internal HTTP retries, and incomplete streams can leave some cost unknown. An +absent value means unknown, not zero. If Relay is not part of the application, run the [standalone server](../getting_started.md#server-path) or embed [`switchyard-libsy`](../../crates/libsy/README.md) directly. @@ -302,7 +336,7 @@ response is available, the error mark describes the terminal failure instead. | `switchyard.routing.llm_calls` | Counter, events | Routing-model calls, labeled by `outcome`. | | `switchyard.routing.llm_call.duration` | Histogram, milliseconds | Routing-model call duration, labeled by `outcome`. | | `switchyard.routing.overhead` | Histogram, milliseconds | Time spent producing the routing outcome. | -| `switchyard.routing.llm_tokens` | Counter, tokens | Provider-reported token values, labeled by `call_role`, `target_model`, and `token_type`. | +| `switchyard.routing.llm_tokens` | Counter, tokens | Normalized token values derived from provider usage, labeled by `call_role`, `target_model`, and `token_type`. | | `switchyard.routing.failures` | Counter, events | Terminal failures, labeled by safe failure kind and available classification fields. | `switchyard.routing.llm_calls` and `switchyard.routing.llm_call.duration` cover From 83f024d205c231342083e148587d677e3e238d98 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Mon, 7 Sep 2026 17:32:05 -0700 Subject: [PATCH 5/9] docs(relay): clarify combined cost wording Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 937507a78..8ab0f9d8e 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -53,9 +53,9 @@ measurement identifies: Switchyard also records the selected and served models, fallback use, routing-call latency, routing overhead, and failures. -No single total-cost or savings metric is emitted today. Relay prices the -caller-facing answer, but it does not automatically price Switchyard's internal -routing calls. For the common case, calculate the observed routed cost as: +The current integration does not emit a single total-cost or savings metric. +Relay prices the caller-facing answer, while Switchyard reports internal routing +usage separately. To estimate the full routed cost, use: ```text Relay answer cost + cost of Switchyard tokens where call_role = "routing" From 14a54855da90d8bd2332286d77097f5eaab804ec Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 8 Sep 2026 09:39:14 -0700 Subject: [PATCH 6/9] docs(relay): distinguish ATIF and OpenTelemetry Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 41 ++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 8ab0f9d8e..0fe89ffcb 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -36,10 +36,10 @@ Relay records the caller-facing request and response. In OpenInference traces, answer tokens appear as `llm.token_count.prompt`, `llm.token_count.completion`, and `llm.token_count.total`. When available, cache use appears as `llm.token_count.prompt_details.cache_read` and -`llm.token_count.prompt_details.cache_write`. Relay also records -`llm.cost.total` in USD when the provider reports a cost, or when the response -contains enough model and usage data and a configured Relay pricing catalog -has all required rates. +`llm.token_count.prompt_details.cache_write`. Relay records `llm.cost.total` +when it has a provider-reported USD cost, or when the response contains enough +model and usage data for Relay to estimate one from a configured pricing +catalog. Switchyard records token use for its model calls in the `switchyard.routing.llm_tokens` metric when the provider reports usage. Each @@ -66,6 +66,13 @@ Apply the price of each `target_model` to those routing tokens. Do not add that Relay recorded. Do not sum every `token_type`; `total` is a rollup, and cache or reasoning values may be a more detailed view of another count. +ATIF includes the caller-facing LLM call in its standard steps and, when +available, cost totals. It does not add Switchyard's routing work to those +totals. Plugin-managed ATIF files may retain the raw routing records under +`extra.observed_events`, but ATIF does not interpret or aggregate them. Use +Relay's OpenTelemetry metrics to estimate routing cost from the usage that +providers report. + Relay does not ship a canonical price catalog. Follow its [model-pricing guide](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) to supply and validate model rates. Run the same representative workload once @@ -245,11 +252,29 @@ when an algorithm must keep the same per-session state across turns. ## Routing Telemetry -Switchyard sends its routing records into the same Relay telemetry stream as -the caller-facing LLM call. Existing Relay +Switchyard emits routing marks and metrics through Relay. This does not turn +each model call inside Switchyard into another Relay LLM lifecycle. Existing +Relay [subscribers](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/subscribers#how-subscribers-relate-to-events) -can export both, so a separate Switchyard telemetry pipeline is not required. -How those records appear in a backend depends on Relay's +can receive the records, but each output format presents a different view. + +### ATIF and OpenTelemetry Show Different Views + +| Output | What the current integration shows | +| --- | --- | +| ATIF | The caller-facing LLM request and response. Internal judge, classifier, fallback, and retry work is not added as separate steps or included in `final_metrics`. Plugin-managed files may retain the raw routing records under `extra.observed_events`. | +| OpenTelemetry traces, including the OpenInference projection | Relay's caller-facing LLM span and, when the selected trace projection includes them, Switchyard routing marks. These marks are point-in-time records rather than full model-call spans. | +| OpenTelemetry metrics | Switchyard request, routing-call, latency, token, and failure measurements. Combine the reported routing usage with Relay's answer cost to estimate the observed routed cost. | +| OTLP logs | Non-metric Switchyard marks that meet the configured severity threshold. | + +The plugin does not currently export Switchyard's internal `libsy.client_call` +or `libsy.upstream_attempt` tracing spans through Relay. A Relay trace therefore +does not contain nested duration spans for internal model calls or individual +HTTP retries, and a dropped internal response stream does not appear as its own +Relay cancellation lifecycle. Switchyard still performs that work; this is a +limit of the telemetry currently passed from the plugin to Relay. + +How marks appear in a trace depends on Relay's [OpenTelemetry trace projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections) and [OpenInference projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/openinference#plugin-configuration). From f9d9c95d44bfab7a37f9652d340cd53d941d431e Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Tue, 8 Sep 2026 12:34:38 -0700 Subject: [PATCH 7/9] docs(relay): clarify telemetry boundaries Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 223 ++++++++++++++------------------ 1 file changed, 98 insertions(+), 125 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index 0fe89ffcb..a9d72424a 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -12,8 +12,8 @@ Using the strongest model for every request is simple, but routine work may not need it. Switchyard can send simpler requests to a lower-cost model and reserve a stronger model for harder work. -Relay keeps the model choice, fallback, latency, and token use alongside the -rest of the agent run. This lets teams answer practical questions: +The plugin reports the selected and served models, fallback use, routing +latency, and token use through Relay telemetry. Use it to answer: - How often did the lower-cost model handle the work? - How often did Switchyard fall back to another model? @@ -26,63 +26,44 @@ Switchyard does not need to run as a separate service. Requests for models that Switchyard does not manage continue through Relay as usual. -### Measure the Cost of Routing - -A routed request can spend tokens in two places. Some routes call a judge or -classifier to choose a model. The selected model, or a fallback, then produces -the answer. Relay and Switchyard record these parts separately. - -Relay records the caller-facing request and response. In OpenInference traces, -answer tokens appear as `llm.token_count.prompt`, `llm.token_count.completion`, -and `llm.token_count.total`. When available, cache use appears as -`llm.token_count.prompt_details.cache_read` and -`llm.token_count.prompt_details.cache_write`. Relay records `llm.cost.total` -when it has a provider-reported USD cost, or when the response contains enough -model and usage data for Relay to estimate one from a configured pricing -catalog. +Routing does not require Relay. You can instead run the +[standalone server](../getting_started.md#server-path) or embed +[`switchyard-libsy`](../../crates/libsy/README.md) directly. -Switchyard records token use for its model calls in the -`switchyard.routing.llm_tokens` metric when the provider reports usage. Each -measurement identifies: +### Measure the Cost of Routing -- `call_role`: `routing` or `answer` -- `target_model`: the upstream model ID -- `token_type`: `input`, `cached_input`, `cache_creation_input`, `output`, - `reasoning`, or `total` +A routed request can spend tokens choosing a model and generating the answer. +Some routes call a judge or classifier first. The selected model, or a fallback, +then answers the request. -Switchyard also records the selected and served models, fallback use, -routing-call latency, routing overhead, and failures. +Relay records the request and answer seen by the agent. It can record a +provider-reported cost or estimate one when model pricing is configured. +Switchyard separately records model calls, available token usage and timing, +fallbacks, and failures. -The current integration does not emit a single total-cost or savings metric. -Relay prices the caller-facing answer, while Switchyard reports internal routing -usage separately. To estimate the full routed cost, use: +The integration does not calculate a combined cost or savings figure. Relay can +price the answer returned to the agent, while Switchyard reports internal +routing usage separately. To estimate the observed cost for a workload, use: ```text Relay answer cost + cost of Switchyard tokens where call_role = "routing" ``` Apply the price of each `target_model` to those routing tokens. Do not add -`call_role = "answer"` again because it describes the same successful answer -that Relay recorded. Do not sum every `token_type`; `total` is a rollup, and -cache or reasoning values may be a more detailed view of another count. - -ATIF includes the caller-facing LLM call in its standard steps and, when -available, cost totals. It does not add Switchyard's routing work to those -totals. Plugin-managed ATIF files may retain the raw routing records under -`extra.observed_events`, but ATIF does not interpret or aggregate them. Use -Relay's OpenTelemetry metrics to estimate routing cost from the usage that -providers report. - -Relay does not ship a canonical price catalog. Follow its +`call_role = "answer"` again because Relay already counted that answer. Do not +sum every `token_type`; `total` is a rollup, and cache or reasoning values may +already be included in broader counts. + +Relay does not copy request identity into exported metric attributes. Use this +calculation for a workload or time window, not to reconcile one trace. + +Relay does not include a model-price catalog. Follow its [model-pricing guide](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) to supply and validate model rates. Run the same representative workload once -with a fixed model and once with routing, then compare observed cost, latency, -fallbacks, and your task-success measure. Missing usage, failed attempts, -internal HTTP retries, and incomplete streams can leave some cost unknown. An -absent value means unknown, not zero. - -If Relay is not part of the application, run the [standalone server](../getting_started.md#server-path) -or embed [`switchyard-libsy`](../../crates/libsy/README.md) directly. +with a fixed model and once with routing. Compare cost, latency, fallback rate, +and task success. Missing usage, failed attempts, internal HTTP retries, and +incomplete streams can leave some cost unknown. An absent value means unknown, +not zero. ## How Requests Flow @@ -121,11 +102,12 @@ well as calls to the selected or fallback answer model, do not run through Relay's LLM middleware again. This avoids treating a router's judge call or fallback attempt as another application request. -Relay still records the caller-facing -[LLM call](https://docs.nvidia.com/nemo/relay/v0.8.3/instrument-applications/instrument-llm-call#integration-pattern), -while Switchyard adds +Relay records the +[LLM call](https://docs.nvidia.com/nemo/relay/v0.8.3/instrument-applications/instrument-llm-call#integration-pattern) +made by the application. Switchyard reports its internal routing work through +Relay [marks](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#mark) -and metrics for the routing work inside it. +and metrics. | Owner | Responsibilities | | --- | --- | @@ -149,16 +131,15 @@ and how it !!! note "Relay compatibility" - You do not need Relay 0.8.3 specifically. The packaged + The plugin supports Relay `>=0.8.1,<0.9.0` and native plugin API `1`. The + packaged [`relay-plugin.toml`](../../crates/switchyard-nemo-relay-plugin/relay-plugin.toml) - is the source of truth: it currently accepts Relay `>=0.8.1,<0.9.0` and - native plugin API `1`. Relay checks both before loading the library. Links - on this page point to version 0.8.3 so the documentation does not drift to - an unsupported Relay release. - -The plugin accepts exactly one Switchyard deployment source: a -`switchyard_config_path` shared with `switchyard-server`, or the same version-1 -deployment nested under `switchyard_config`. Both use the + is the source of truth. Links on this page point to Relay 0.8.3 so they stay + within that supported range. + +Configure exactly one Switchyard deployment source: either +`switchyard_config_path`, which points to the TOML used by `switchyard-server`, +or an inline version-1 deployment under `switchyard_config`. Both use the [Switchyard TOML schema](../reference/toml_schema.md). The plugin reuses the deployment's routes, targets, and LLM clients. It does not @@ -186,20 +167,28 @@ Switchyard normalizes the request, routes it, and returns the response in the caller's original format. If Switchyard forwards the caller's credential, both formats must use the same credential family: OpenAI-compatible or Anthropic. -Not every provider-specific field has a lossless equivalent. Switchyard rejects -a conversion it cannot perform safely instead of silently dropping data. +Support for provider-specific fields depends on the source and target formats. +Test any fields that your application relies on before deploying a translated +route. + +### Header Forwarding + +Caller headers are forwarded upstream except credentials and headers owned by +the HTTP client, such as connection and content headers. Authentication and +configured extra headers follow the selected client's settings in the +[TOML schema](../reference/toml_schema.md). ### Streaming -For streaming requests, Switchyard hands Relay a lazy translated stream. Relay -drives delivery and cancellation and records when the caller-facing stream -starts and ends. As Relay consumes the stream, Switchyard continues to translate -chunks and record late usage or errors. +For streaming requests, Switchyard returns a translated stream that Relay +consumes lazily. Relay drives delivery and cancellation and records when the +stream starts and ends. Switchyard continues to translate chunks and record +late usage or errors as Relay consumes them. - Initial routing marks are available when the stream opens. - An answer-call result of `ok` means the provider opened the stream. It does not guarantee that the full stream completed. -- For an upstream response that remains streamed, answer token metrics appear +- For a streamed upstream response, answer token metrics appear only if the provider reports usage and the stream reaches its final event. A canceled or dropped stream may have no answer-token metrics. - Later provider failures can emit `switchyard.routing.error`. Some failures @@ -217,16 +206,15 @@ drops it. The plugin does not store these payloads on disk. The plugin creates one Switchyard runner when Relay activates it and shares the runner across requests until the plugin is deactivated. Some routing algorithms keep in-memory state there, such as session affinity or an escalation decision. -Each algorithm controls when that state expires. The state is not shared between -Relay processes and is lost when a process restarts. See Relay's documentation -on +State behavior and expiry depend on the algorithm. The state is not shared +between Relay processes and is lost when a process restarts. See Relay's +documentation on [plugin ownership](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/plugins#ownership-and-scope) and [runtime state](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/architecture#where-runtime-state-lives). -To associate routing telemetry with the rest of an agent run, the plugin adds -these fields to every routing mark, including marks that carry metric -measurements. Missing values are `null`: +The plugin adds these fields to each routing mark, including metric marks. +Missing values are `null`: - `session_id` - `agent_id` @@ -235,7 +223,6 @@ measurements. Missing values are `null`: - `turn_id` - `correlation_id` -These fields are event metadata used to correlate routing records with a run. Subscribers and log or trace exporters can read them, but Relay does not copy them into exported metric attributes. @@ -244,50 +231,44 @@ These values come from request headers rather than Relay's active scope. Relay's can populate them for correlation. They do not by themselves mark a request as delegated work for Switchyard's [`subagents` router](../routing_algorithms/subagent_routing.md). For algorithms -that keep per-session state, reuse a stable session ID across turns. Relay's -`x-nemo-relay-session-id` header is accepted; `x-switchyard-session-id` -provides an explicit override. If Relay's gateway has no stable session ID, the -plugin leaves the Switchyard session ID unset. Send `x-switchyard-session-id` -when an algorithm must keep the same per-session state across turns. +that keep per-session state, reuse a stable ID across turns. The plugin accepts +`x-nemo-relay-session-id`; `x-switchyard-session-id` overrides it. If neither +provides a value, the Switchyard session ID remains unset. ## Routing Telemetry -Switchyard emits routing marks and metrics through Relay. This does not turn -each model call inside Switchyard into another Relay LLM lifecycle. Existing -Relay +Relay sees one LLM lifecycle for the request made by the agent. Switchyard +reports the routing work through Relay marks and metrics. Existing Relay [subscribers](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/subscribers#how-subscribers-relate-to-events) -can receive the records, but each output format presents a different view. +can receive these records, but each output format presents them differently. ### ATIF and OpenTelemetry Show Different Views | Output | What the current integration shows | | --- | --- | -| ATIF | The caller-facing LLM request and response. Internal judge, classifier, fallback, and retry work is not added as separate steps or included in `final_metrics`. Plugin-managed files may retain the raw routing records under `extra.observed_events`. | -| OpenTelemetry traces, including the OpenInference projection | Relay's caller-facing LLM span and, when the selected trace projection includes them, Switchyard routing marks. These marks are point-in-time records rather than full model-call spans. | +| ATIF | The request and response seen by the agent. Internal routing is not added as separate steps or included in `final_metrics`. Relay-managed ATIF files may retain the raw records under `extra.observed_events`. | +| OpenTelemetry traces, including OpenInference | Relay's LLM span plus eligible routing marks, depending on the projection. Marks are point-in-time records, not duration spans. | | OpenTelemetry metrics | Switchyard request, routing-call, latency, token, and failure measurements. Combine the reported routing usage with Relay's answer cost to estimate the observed routed cost. | | OTLP logs | Non-metric Switchyard marks that meet the configured severity threshold. | -The plugin does not currently export Switchyard's internal `libsy.client_call` -or `libsy.upstream_attempt` tracing spans through Relay. A Relay trace therefore -does not contain nested duration spans for internal model calls or individual -HTTP retries, and a dropped internal response stream does not appear as its own -Relay cancellation lifecycle. Switchyard still performs that work; this is a -limit of the telemetry currently passed from the plugin to Relay. - -How marks appear in a trace depends on Relay's -[OpenTelemetry trace projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections) -and -[OpenInference projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/openinference#plugin-configuration). +Switchyard creates `libsy.run`, `libsy.llm_call`, `libsy.client_call`, and +`libsy.upstream_attempt` spans internally. Together they cover the algorithm +run, model calls requested by the algorithm, candidate attempts including +fallbacks, and individual HTTP attempts. They are not one connected hierarchy +today, and the plugin does not send them to Relay. As a result, Relay traces do +not show those internal operations as nested duration spans or give an internal +stream its own cancellation lifecycle. ### How Routing Appears in Traces -When the caller-facing LLM call uses Relay's active agent scope as its parent, -the Switchyard marks use that same scope and appear alongside the call. -Switchyard does not create another nested scope. For Relay's +In Relay's [full and OpenInference trace projections](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections), [`mark_projection`](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-endpoint-fields) -controls whether a backend displays each eligible mark as an event on the -parent or as a visible zero-duration child span. The +controls how marks appear. With `inherit` or `event`, a routing mark is an event +on its parent span while that span is open. Otherwise, Relay emits it as a +zero-duration span and retains its parent when possible. With `tool`, routing +marks are always visible zero-duration spans. The plugin does not nest these +marks under the LLM span. The [`gen_ai` projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#genai-projection) omits marks. With `mark_projection = "tool"`, the trace has this shape: @@ -295,11 +276,11 @@ omits marks. With `mark_projection = "tool"`, the trace has this shape: flowchart LR agent["Relay agent scope"] llm["LLM call"] - requested["switchyard.routing.requested"] - calls["switchyard.routing.llm_call
routing and answer attempts"] - overhead["switchyard.routing.overhead"] - decision["switchyard.routing.decision"] - error["switchyard.routing.error
when execution fails"] + requested["mark:switchyard.routing.requested
zero duration"] + calls["mark:switchyard.routing.llm_call
zero duration"] + overhead["mark:switchyard.routing.overhead
zero duration"] + decision["mark:switchyard.routing.decision
zero duration"] + error["mark:switchyard.routing.error
zero duration"] agent --> llm agent --> requested agent --> calls @@ -308,8 +289,9 @@ flowchart LR agent -.->|"Failure"| error ``` -If the application did not create an agent scope, an observability backend can -display the LLM span and marks as separate roots. The metadata field +When the LLM call has an agent scope as its parent, the marks use that same scope +and appear beside the call. Without an agent scope, a backend can display the +LLM span and marks as separate roots. The metadata field `parent_agent_id` is a correlation value; it does not set Relay trace parentage. See Relay's [scope hierarchy](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/scopes#scope-hierarchy-and-ownership) @@ -328,7 +310,7 @@ describes the surrounding event envelope. | Mark | Severity | Data | | --- | --- | --- | | `switchyard.routing.requested` | Info | Routing `algorithm` for a managed request. | -| `switchyard.routing.llm_call` | Debug | `call_index`, model in `selected_model`, `call_role` (`routing` or `answer`), `outcome`, and `latency_ms` for each observed model call. | +| `switchyard.routing.llm_call` | Debug | `call_index`, `selected_model`, `call_role` (`routing` or `answer`), `outcome`, and `latency_ms` for each observed model call. | | `switchyard.routing.overhead` | Info | `latency_ms` spent producing the routing outcome, including routing-model calls. This is not the end-to-end request duration. | | `switchyard.routing.decision` | Info | `algorithm`, initial `selected_model`, nullable final `served_model`, and nullable `fallback_used`. | | `switchyard.routing.error` | Error | Generic failures contain `failure_kind`. Route-execution failures also contain `category` and `phase`, plus nullable `upstream_status` and `target`. | @@ -342,11 +324,9 @@ supported trace projections, but Relay's OTLP logs default to Info. Set `minimum_severity` to `debug` to include these call records in [log export](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#log-export). -Model fields use the target's upstream model ID, not its local TOML key. For -example, if `[targets.fast].id = "provider/model-a"`, the mark records -`provider/model-a`, not `fast`. Relay records the model value after its request -middleware runs, normally the Switchyard route ID. On the response, Relay can -report the model that actually answered. +Switchyard marks use the target's upstream model ID, not its local TOML key. +Relay request telemetry normally uses the Switchyard route ID, while response +telemetry can report the model that answered. `fallback_used` is `true` when the final served model differs from the initial selection and `false` when they match. It and `served_model` are `null` when the @@ -378,16 +358,9 @@ Configure delivery through Relay's ## Data Handling -Switchyard routing telemetry contains a small, defined set of routing fields, -not request or response content. Its marks do not contain prompts, request or -response bodies, headers, credentials, raw provider response bodies, or -free-form provider error messages. Relay's caller-facing LLM events can -capture request and response data according to Relay's +Switchyard routing telemetry excludes request and response content. Its marks +do not contain prompts, bodies, headers, credentials, raw provider responses, +or free-form provider errors. Relay's LLM events can capture request and +response data according to Relay's [input and output event semantics](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#input-and-output-payloads), independently of these Switchyard marks. - -Header forwarding is part of request execution, not telemetry. Caller headers -are forwarded upstream except credentials and headers owned by the HTTP client, -such as connection and content headers. Authentication and configured extra -headers follow the selected client's settings in the -[TOML schema](../reference/toml_schema.md). From ed1279f386a49377d321508a30b2431b8d411e25 Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Wed, 9 Sep 2026 19:46:19 -0700 Subject: [PATCH 8/9] docs(relay): address integration review Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 61 +++++++++++++++++---------------- 1 file changed, 31 insertions(+), 30 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index a9d72424a..e8204dbf7 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -2,15 +2,17 @@ The [Switchyard native plugin](../../crates/switchyard-nemo-relay-plugin/README.md) loads Switchyard into an existing -[NeMo Relay](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/overview#integrating-with-relay) +[NeMo Relay runtime](https://docs.nvidia.com/nemo/relay/about-nemo-relay/overview) deployment through Relay's -[native plugin system](https://docs.nvidia.com/nemo/relay/v0.8.3/build-plugins/native/about). +[native dynamic plugin system](https://docs.nvidia.com/nemo/relay/build-plugins/native/about). ## Why Use Switchyard with NeMo Relay? -Using the strongest model for every request is simple, but routine work may not -need it. Switchyard can send simpler requests to a lower-cost model and reserve -a stronger model for harder work. +Switchyard's routing algorithms select a model for each LLM request or step in +an agent trajectory, balancing cost and performance. The NeMo Relay integration +makes those algorithms available to coding agent harnesses supported by Relay. +Any Relay harness integration that can load manifest-driven plugins can +configure and use Switchyard's core routers. The plugin reports the selected and served models, fallback use, routing latency, and token use through Relay telemetry. Use it to answer: @@ -58,7 +60,7 @@ Relay does not copy request identity into exported metric attributes. Use this calculation for a workload or time window, not to reconcile one trace. Relay does not include a model-price catalog. Follow its -[model-pricing guide](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#add-model-pricing-for-cost-estimates) +[model-pricing guide](https://docs.nvidia.com/nemo/relay/configure-plugins/model-pricing) to supply and validate model rates. Run the same representative workload once with a fixed model and once with routing. Compare cost, latency, fallback rate, and task success. Missing usage, failed attempts, internal HTTP retries, and @@ -72,9 +74,9 @@ the plugin checks whether the requested model matches a configured Switchyard route. Matching requests go through Switchyard. Other requests are left unchanged by Switchyard and passed to the next Relay handler. The plugin uses Relay's -[execution intercepts](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#execution-intercepts) +[execution intercepts](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/middleware#execution-intercepts) for non-streaming requests and -[stream execution intercepts](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/middleware#stream-execution-intercepts) +[stream execution intercepts](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/middleware#stream-execution-intercepts) for streaming requests. ```mermaid @@ -103,10 +105,10 @@ Relay's LLM middleware again. This avoids treating a router's judge call or fallback attempt as another application request. Relay records the -[LLM call](https://docs.nvidia.com/nemo/relay/v0.8.3/instrument-applications/instrument-llm-call#integration-pattern) +[LLM call](https://docs.nvidia.com/nemo/relay/instrument-applications/instrument-llm-call#integration-pattern) made by the application. Switchyard reports its internal routing work through Relay -[marks](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#mark) +[marks](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/events#mark) and metrics. | Owner | Responsibilities | @@ -116,26 +118,25 @@ and metrics. | Model provider | Runs the model and returns its response, stream, and available usage. | For more detail, see Relay's -[managed execution pipeline](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/architecture#managed-execution-pipeline) +[managed execution pipeline](https://docs.nvidia.com/nemo/relay/about-nemo-relay/architecture#managed-execution-pipeline) and -[plugin delivery models](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/plugins#plugin-delivery-models). +[plugin delivery models](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/plugins#plugin-delivery-models). ## Set Up the Plugin Follow the [plugin README](../../crates/switchyard-nemo-relay-plugin/README.md) to build and package the native library, register and enable it in Relay, and configure its deployment. Relay documents how to -[add and enable a discoverable plugin](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/discoverable-plugins#add-and-enable-a-plugin) +[add and enable a discoverable plugin](https://docs.nvidia.com/nemo/relay/configure-plugins/discoverable-plugins#add-and-enable-a-plugin) and how it -[validates the package before loading code](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/discoverable-plugins#validate-before-loading-code). +[validates the package before loading code](https://docs.nvidia.com/nemo/relay/configure-plugins/discoverable-plugins#validate-before-loading-code). !!! note "Relay compatibility" - The plugin supports Relay `>=0.8.1,<0.9.0` and native plugin API `1`. The - packaged + The plugin requires `relay = ">=0.8.1,<0.9.0"` and native plugin API `1`. + The packaged [`relay-plugin.toml`](../../crates/switchyard-nemo-relay-plugin/relay-plugin.toml) - is the source of truth. Links on this page point to Relay 0.8.3 so they stay - within that supported range. + is the source of truth. Configure exactly one Switchyard deployment source: either `switchyard_config_path`, which points to the TOML used by `switchyard-server`, @@ -209,9 +210,9 @@ keep in-memory state there, such as session affinity or an escalation decision. State behavior and expiry depend on the algorithm. The state is not shared between Relay processes and is lost when a process restarts. See Relay's documentation on -[plugin ownership](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/plugins#ownership-and-scope) +[plugin ownership](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/plugins#ownership-and-scope) and -[runtime state](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/architecture#where-runtime-state-lives). +[runtime state](https://docs.nvidia.com/nemo/relay/about-nemo-relay/architecture#where-runtime-state-lives). The plugin adds these fields to each routing mark, including metric marks. Missing values are `null`: @@ -227,7 +228,7 @@ Subscribers and log or trace exporters can read them, but Relay does not copy them into exported metric attributes. These values come from request headers rather than Relay's active scope. Relay's -[session and subagent headers](https://docs.nvidia.com/nemo/relay/v0.8.3/nemo-relay-cli/basic-usage#runtime-mapping) +[session and subagent headers](https://docs.nvidia.com/nemo/relay/nemo-relay-cli/basic-usage#runtime-mapping) can populate them for correlation. They do not by themselves mark a request as delegated work for Switchyard's [`subagents` router](../routing_algorithms/subagent_routing.md). For algorithms @@ -239,7 +240,7 @@ provides a value, the Switchyard session ID remains unset. Relay sees one LLM lifecycle for the request made by the agent. Switchyard reports the routing work through Relay marks and metrics. Existing Relay -[subscribers](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/subscribers#how-subscribers-relate-to-events) +[subscribers](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/subscribers#how-subscribers-relate-to-events) can receive these records, but each output format presents them differently. ### ATIF and OpenTelemetry Show Different Views @@ -262,14 +263,14 @@ stream its own cancellation lifecycle. ### How Routing Appears in Traces In Relay's -[full and OpenInference trace projections](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-projections), -[`mark_projection`](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#trace-endpoint-fields) +[full and OpenInference trace projections](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/opentelemetry#trace-projections), +[`mark_projection`](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/opentelemetry#trace-endpoint-fields) controls how marks appear. With `inherit` or `event`, a routing mark is an event on its parent span while that span is open. Otherwise, Relay emits it as a zero-duration span and retains its parent when possible. With `tool`, routing marks are always visible zero-duration spans. The plugin does not nest these marks under the LLM span. The -[`gen_ai` projection](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#genai-projection) +[`gen_ai` projection](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/opentelemetry#genai-projection) omits marks. With `mark_projection = "tool"`, the trace has this shape: ```mermaid @@ -294,7 +295,7 @@ and appear beside the call. Without an agent scope, a backend can display the LLM span and marks as separate roots. The metadata field `parent_agent_id` is a correlation value; it does not set Relay trace parentage. See Relay's -[scope hierarchy](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/scopes#scope-hierarchy-and-ownership) +[scope hierarchy](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/scopes#scope-hierarchy-and-ownership) for the parentage rules. ### Mark Contract @@ -304,7 +305,7 @@ contract. Each non-metric mark uses the mark name as its schema name and version `1`. Consumers should accept additional fields and values within a version. Removing or renaming a field, changing its type, or changing its meaning requires a new version. Relay's -[event envelope](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#fields-common-to-every-event) +[event envelope](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/events#fields-common-to-every-event) describes the surrounding event envelope. | Mark | Severity | Data | @@ -322,7 +323,7 @@ or answer work. `switchyard.routing.llm_call` uses Debug severity. It still appears in the supported trace projections, but Relay's OTLP logs default to Info. Set `minimum_severity` to `debug` to include these call records in -[log export](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#log-export). +[log export](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/opentelemetry#log-export). Switchyard marks use the target's upstream model ID, not its local TOML key. Relay request telemetry normally uses the Switchyard route ID, while response @@ -354,7 +355,7 @@ types are `input`, `cached_input`, `cache_creation_input`, `output`, `reasoning`, and `total`. Configure delivery through Relay's -[OpenTelemetry metric export](https://docs.nvidia.com/nemo/relay/v0.8.3/configure-plugins/observability/opentelemetry#metric-export). +[OpenTelemetry metric export](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/opentelemetry#metric-export). ## Data Handling @@ -362,5 +363,5 @@ Switchyard routing telemetry excludes request and response content. Its marks do not contain prompts, bodies, headers, credentials, raw provider responses, or free-form provider errors. Relay's LLM events can capture request and response data according to Relay's -[input and output event semantics](https://docs.nvidia.com/nemo/relay/v0.8.3/about-nemo-relay/concepts/events#input-and-output-payloads), +[input and output event semantics](https://docs.nvidia.com/nemo/relay/about-nemo-relay/concepts/events#input-and-output-payloads), independently of these Switchyard marks. From f8ceb0b1b93eef3ba8616a991542973a1319a96e Mon Sep 17 00:00:00 2001 From: Alex Fournier Date: Thu, 10 Sep 2026 09:03:49 -0700 Subject: [PATCH 9/9] docs(relay): list supported routing algorithms Signed-off-by: Alex Fournier --- docs/integrations/nemo_relay.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md index e8204dbf7..8cf072c36 100644 --- a/docs/integrations/nemo_relay.md +++ b/docs/integrations/nemo_relay.md @@ -11,8 +11,9 @@ deployment through Relay's Switchyard's routing algorithms select a model for each LLM request or step in an agent trajectory, balancing cost and performance. The NeMo Relay integration makes those algorithms available to coding agent harnesses supported by Relay. -Any Relay harness integration that can load manifest-driven plugins can -configure and use Switchyard's core routers. +The integration is not tied to one routing algorithm. It supports passthrough, +random, Stage, composite, advisor, and LLM classifier routes, including +capability, escalation, and custom modes. The plugin reports the selected and served models, fallback use, routing latency, and token use through Relay telemetry. Use it to answer: