[feature] Skip repeatedly unavailable routing candidates with circuit breakers

[aknvda]

Problem

Switchyard retries transient upstream failures and then falls back through the ordered candidates in a RoutingOutcome. Both mechanisms are currently request-local.

TranslatingLlmClient::send_encoded exhausts the configured retry budget for a candidate, and call_first_available then tries the next candidate. A new request starts from the first candidate again, even when that candidate has just failed repeatedly.

For example, if a route has weak followed by strong, max_retries = 2, and weak consistently returns HTTP 503, every request makes three unsuccessful calls to weak before trying strong. Under sustained traffic, this increases request latency and amplifies load against an unhealthy backend.

Proposed solution

Add an optional circuit breaker to each configured LLM client:

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://example.test/v1"
max_retries = 2

[llm_clients.openrouter.circuit_breaker]
failure_threshold = 3
cooldown_ms = 30_000

When the table is absent, behavior remains unchanged. When present, both values must be greater than zero.

The configuration applies to completion calls made by the client, with independent state for each (ModelId, WireFormat). The TranslatingLlmClient instance already scopes that key to a configured LLM client.

State transitions

• Closed: Calls use the existing retry behavior.
• After failure_threshold logical transient failures, the circuit becomes Open.
• Open: Calls fail immediately without making an upstream HTTP request.
• After cooldown_ms, the circuit becomes HalfOpen.
• HalfOpen: Exactly one concurrent call is admitted as a probe. Other calls continue to fail fast.
• A successful probe closes the circuit and resets its failure count.
• A transiently failed probe reopens the circuit for another cooldown.

A failure is recorded once after the candidate exhausts its retry budget, not once per HTTP attempt.

Circuit state must also be safe under concurrency:

• Cancellation of a half-open probe must not leave the circuit permanently stuck.
• Results from older in-flight calls must not overwrite a newer circuit transition.
• Opening a circuit does not need to cancel requests that are already in flight.

Failure classification

The circuit breaker should reuse the existing backend retry classification:

• LlmClientError::Transport
• LlmClientError::Timeout
• HTTP 408
• HTTP 429
• HTTP 5xx

Only the final result after retries are exhausted increments the failure count.

A successful upstream response resets the consecutive transient-failure count. Request-specific or local failures do not open the circuit, including:

• context-window overflow;
• invalid requests;
• authentication and other deterministic 4xx failures;
• configuration errors;
• request or response translation errors.

Routing integration

Add a typed error such as:

LlmClientError::CircuitOpen {
    model: ModelId,
}

libsy-llm-client::run::fallback_reason should classify CircuitOpen as the existing RoutingFallbackReason::Unavailable. Ordered fallback can then skip an open candidate without introducing a new fallback reason.

If every candidate is unavailable, the HTTP server should map the final CircuitOpen error to HTTP 503. Runner failure telemetry should retain a safe, typed circuit-open classification so the behavior is shared by the standalone server and NeMo Relay integration.

Observability

Emit a structured log when a circuit opens, enters half-open state, or closes. Record counters for:

• circuit openings;
• calls skipped because a circuit was open.

Suggested metric attributes are model and wire_format. Logs and errors must not expose upstream URLs, credentials, or response bodies.

Acceptance criteria

• Omitting circuit_breaker preserves existing behavior.
• Invalid zero-valued thresholds or cooldowns fail configuration validation.
• A transient failure is counted once after backend retries are exhausted.
• The circuit opens after the configured number of failures.
• An open circuit produces no outbound HTTP request.
• Ordered routing immediately tries the next candidate when a circuit is open.
• Only one half-open probe is admitted under concurrent load.
• Cancellation cannot leave the circuit stuck in half-open state.
• Stale in-flight results cannot overwrite a newer state transition.
• A successful probe closes the circuit and resets its failure count.
• A failed probe reopens the circuit.
• Request-specific and non-transient failures do not open the circuit.
• State is independent per model, wire format, and client instance.
• When all candidates are unavailable, the server returns HTTP 503.
• State transitions and skipped calls are observable through logs and metrics.
• Tests use deterministic or paused time rather than wall-clock sleeps.
• Existing retry, fallback, server, and Relay tests remain unchanged when the feature is disabled.

Alternatives considered

Continue relying on retries and request-local fallback

This preserves the current behavior but repeats the complete retry cost on every request during a sustained outage.

Reuse session eviction

Session eviction is appropriate for request-inherent conditions such as context-window or capability rejection. Backend health is shared across independent requests and should recover after a cooldown, so it should not be tied to one session.

Add active background health checks

Active checks introduce extra traffic and require provider-specific decisions about endpoints and credentials. A passive circuit breaker can use failures Switchyard already observes and keeps the initial implementation provider-neutral.

Put the breaker in the routing algorithm

The upstream client owns retry classification and is shared across routes. Keeping the breaker there allows failures to be counted after retries and avoids separate health state for every algorithm or route.

Scope notes

The proposed change is owned primarily by the upstream client, with small integrations in:

• crates/libsy-llm-client for the state machine, retry-boundary integration, fallback classification, and metrics;
• crates/protocol for the typed CircuitOpen error;
• crates/switchyard-runner for TOML configuration and failure telemetry;
• crates/switchyard-server for HTTP 503 mapping.

This changes the public Rust error type and deployment TOML schema, but it is opt-in and backward-compatible.

Initial non-goals:

• active health polling;
• distributed state shared across Switchyard processes;
• state persistence across restarts;
• auxiliary operations such as token counting or compaction;
• replay or fallback after streamed output reaches the caller;
• changing candidate ordering or routing algorithms;
• changing the decision-only API based on runtime health;
• introducing a first-class target identifier.

Additional context

Related issues:

• #345 addresses request-inherent capability rejection and session-sticky eviction. Circuit breaking instead tracks transient backend health across independent requests.
• #277 proposes request and judge-call deadlines. Timeout failures can feed a circuit breaker, but deadlines and cross-request health memory are separate mechanisms.
• #354 discusses target identity. This proposal deliberately uses the client’s existing (ModelId, WireFormat) runtime identity rather than expanding that design.

The main design question is whether an opt-in breaker owned by TranslatingLlmClient, scoped to completion calls and keyed by the current runtime identity, matches the intended direction.

[aknvda]

@grahamking @ayushag-nv @ryan-lempka - What do you think?