diff --git a/CHANGELOG.md b/CHANGELOG.md index 76884e283..25528e92d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,14 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- **`subagents` on an `llm_classifier` route** — the nested sub-agent policy + already available on `passthrough`, `stage_router` and `composite` now also + parses and builds under `llm_classifier`, in all three of its modes. Wrapping + diverts delegated work before the parent judge runs, so the parent's session + affinity no longer sees it — the same trade the other three variants make. + `AlgorithmSpec::LlmClassifier` gains a field, which source-breaks a downstream + struct literal that does not end in `..`. + - **NeMo Relay native plugin** — a dynamically loaded integration that loads Switchyard's standard TOML deployment and executes its `switchyard-runner`- supported configured routes in process. Managed calls require NeMo Relay diff --git a/README.md b/README.md index efd341ee7..71f17ea0e 100644 --- a/README.md +++ b/README.md @@ -297,7 +297,7 @@ Most use an LLM as a judge. All of them pick between an **efficient** model and | **[Capability + Stage](docs/routing_algorithms/composite_routing.md)** | Combines the two above. | `composite` | not yet benchmarked | | **[Escalation](docs/routing_algorithms/escalation_router_routing.md)** | Starts efficient. Responses are judged by an LLM for issues, then escalated. | `llm_classifier` + `mode = "escalation"` | 75.7% at $85.00 | | **[Advisor Gate](docs/routing_algorithms/advisor_gate_routing.md)** | One model serves every turn; a stronger advisor approves its plans and "done" claims, or sends it back. | `advisor` | lifts a weak executor 43.8% → 54.7% | -| **[Sub-Agent-Aware](docs/routing_algorithms/subagent_routing.md)** | Delegated sub-agent traffic routes separately from the parent agent. | `subagents` on `passthrough` or `stage_router` | not yet benchmarked | +| **[Sub-Agent-Aware](docs/routing_algorithms/subagent_routing.md)** | Delegated sub-agent traffic routes separately from the parent agent. | `subagents` on `passthrough`, `llm_classifier`, `stage_router` or `composite` | not yet benchmarked | | **[Custom](docs/routing_algorithms/llm_classifier_routing.md#custom-multi-target-routing)** | The first request is judged by an LLM against criteria you define, routing among 2+ of your own models. | `llm_classifier` + `target_selector` policy | not yet benchmarked | | **[Random](docs/routing_algorithms/random_routing.md)** | Each request is routed at random, uniform or weighted. | `random` | baseline mechanism | diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rs index d3e5f78ac..952a1b916 100644 --- a/crates/switchyard-runner/src/algorithm.rs +++ b/crates/switchyard-runner/src/algorithm.rs @@ -183,7 +183,7 @@ pub struct LlmClassifierRouteConfig { } /// Routing policy applied only to delegated sub-agent work, nested inside a -/// `passthrough` or `stage_router` route. +/// `passthrough`, `llm_classifier`, `stage_router` or `composite` route. #[derive(Clone, Debug, Deserialize)] #[serde(tag = "type", rename_all = "snake_case", deny_unknown_fields)] pub enum SubagentRouteConfig { @@ -245,6 +245,9 @@ pub enum AlgorithmSpec { /// Judge and tier settings, written directly in the route table. #[serde(flatten)] config: LlmClassifierRouteConfig, + /// Separate policy for delegated sub-agent work. + #[serde(default)] + subagents: Option, }, /// Picks a tier per turn by scoring signals from recent tool results. StageRouter { @@ -433,31 +436,38 @@ impl AlgorithmSpec { } names } - Self::LlmClassifier { config, .. } => { - match config.mode.unwrap_or(if config.escalation.is_some() { - ClassifierMode::Escalation - } else { - ClassifierMode::Capability - }) { - ClassifierMode::Capability => config - .weak_target - .iter() - .chain(&config.strong_target) - .map(String::as_str) - .collect(), - ClassifierMode::Escalation => config - .strong_target - .iter() - .chain(&config.weak_target) - .map(String::as_str) - .collect(), - ClassifierMode::Custom => config - .targets - .iter() - .flatten() - .map(String::as_str) - .collect(), + Self::LlmClassifier { + config, subagents, .. + } => { + let mut names: Vec<&str> = + match config.mode.unwrap_or(if config.escalation.is_some() { + ClassifierMode::Escalation + } else { + ClassifierMode::Capability + }) { + ClassifierMode::Capability => config + .weak_target + .iter() + .chain(&config.strong_target) + .map(String::as_str) + .collect(), + ClassifierMode::Escalation => config + .strong_target + .iter() + .chain(&config.weak_target) + .map(String::as_str) + .collect(), + ClassifierMode::Custom => config + .targets + .iter() + .flatten() + .map(String::as_str) + .collect(), + }; + if let Some(subagents) = subagents { + names.extend(subagents.routing_target_names()); } + names } Self::StageRouter { tiers, subagents, .. @@ -499,7 +509,14 @@ impl AlgorithmSpec { pub fn callable_target_names(&self) -> Vec<&str> { let mut names = self.routing_target_names(); match self { - Self::LlmClassifier { config, .. } => names.push(&config.classifier_target), + Self::LlmClassifier { + config, subagents, .. + } => { + names.push(&config.classifier_target); + if let Some(subagents) = subagents { + names.extend(subagents.classifier_target_name()); + } + } Self::Passthrough { subagents: Some(subagents), .. @@ -865,6 +882,7 @@ fn build_algorithm( } AlgorithmSpec::LlmClassifier { config: classifier_config, + subagents, .. } => { let classifier = @@ -948,7 +966,8 @@ fn build_algorithm( error, ) })?; - Ok(Arc::new(algorithm)) + let parent: Arc = Arc::new(algorithm); + attach_subagent_router(route_name, parent, subagents.as_ref(), targets) } AlgorithmSpec::StageRouter { tiers, diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rs index c1fea1464..3371bc0e3 100644 --- a/crates/switchyard-runner/src/config.rs +++ b/crates/switchyard-runner/src/config.rs @@ -743,7 +743,7 @@ confidence_threshold = 0.5 } #[test] - fn passthrough_and_stage_accept_subagent_routing() -> RunnerResult<()> { + fn parent_routes_accept_subagent_routing() -> RunnerResult<()> { let stage = stage_config(); let stage_with_classifier = with_subagent_llm_classifier(&stage, "stage", ""); let parsed: DeploymentConfig = toml::from_str(&stage_with_classifier).map_err(|error| { @@ -757,11 +757,41 @@ confidence_threshold = 0.5 assert!(callable_targets.contains(&expected)); } + // An llm_classifier parent ends up with two judges once it nests a sub-agent route: + // its own and the child's. Renaming the parent's tells them apart. The exact vector + // pins that the child's targets are appended rather than merely present -- the child + // reuses the parent's own `strong`/`weak`, so `contains` cannot see the difference. + let base = VALID_CONFIG.replace( + "classifier_target = \"classifier\"", + "classifier_target = \"parent_judge\"", + ) + "\n[targets.parent_judge]\nid = \"parent-judge/model\"\nllm_client = \"primary\"\n"; + let classifier_with_classifier = with_subagent_llm_classifier(&base, "classifier", ""); + let parsed: DeploymentConfig = + toml::from_str(&classifier_with_classifier).map_err(|error| { + RunnerError::configuration(format!("failed to parse classifier config: {error}")) + })?; + let Some(classifier_route) = parsed.routes.get("classifier") else { + return Err(RunnerError::configuration("classifier route is missing")); + }; + assert_eq!( + classifier_route.callable_target_names(), + [ + "weak", + "strong", + "strong", + "weak", + "parent_judge", + "classifier" + ] + ); + for configured in [ with_subagent_llm_classifier(VALID_CONFIG, "passthrough", ""), with_subagent_passthrough(VALID_CONFIG, "passthrough"), stage_with_classifier, with_subagent_passthrough(&stage, "stage"), + classifier_with_classifier, + with_subagent_passthrough(VALID_CONFIG, "classifier"), ] { runner_from_toml(&configured)?; } @@ -1031,6 +1061,14 @@ classifier_magic = true ), "cannot use message_hash_fallback", ), + ( + with_subagent_llm_classifier( + VALID_CONFIG, + "classifier", + "\nmessage_hash_fallback = true", + ), + "cannot use message_hash_fallback", + ), ( with_subagent_llm_classifier(VALID_CONFIG, "passthrough", "") .replace("mode = \"custom\"", "mode = \"capability\""), diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md index 0193ca680..ce7690aac 100644 --- a/docs/reference/toml_schema.md +++ b/docs/reference/toml_schema.md @@ -176,6 +176,7 @@ Runs one of three judge-backed modes: `capability`, `escalation`, or `custom`. | `classifier_target` | Yes | — | Target the judge is called through. Not a routing destination. | | `max_output_tokens` | No | `4096` | Maximum completion tokens for the judge verdict. Must be at least `1`. | | `response_format_type` | No | `json_schema` | Structured-output mode for capability and escalation judges. Use `json_object` when the provider does not support JSON Schema; Switchyard adds the schema to the prompt and validates the verdict locally. Custom mode always uses its configured JSON Schema. | +| `subagents` | No | unset | Nested `passthrough` or custom `llm_classifier` policy used only for delegated sub-agent work. See [Sub-Agent-Aware Routing](../routing_algorithms/subagent_routing.md). | Capability mode classifies before serving. See [LLM Classifier Routing](../routing_algorithms/llm_classifier_routing.md). diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md index 648e6866b..6ad847e95 100644 --- a/docs/routing_algorithms/overview.md +++ b/docs/routing_algorithms/overview.md @@ -12,7 +12,7 @@ configuration and tuning. For the vocabulary these pages use, see | Strategy | Use it when | Route `type` | |---|---|---| -| [Sub-Agent-Aware Routing](subagent_routing.md) | Delegated sub-agents should use a separate routing policy from the parent agent. | `passthrough` or `stage_router` with `subagents` | +| [Sub-Agent-Aware Routing](subagent_routing.md) | Delegated sub-agents should use a separate routing policy from the parent agent. | `passthrough`, `llm_classifier`, `stage_router` or `composite` with `subagents` | | [Random Routing](random_routing.md) | You need a fixed traffic split for A/B tests, baselines, or cost experiments. | `random` | | [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm_classifier` | | [Stage-Router Routing](stage_router_routing.md) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` | diff --git a/docs/routing_algorithms/subagent_routing.md b/docs/routing_algorithms/subagent_routing.md index 2533e80ea..64c009c71 100644 --- a/docs/routing_algorithms/subagent_routing.md +++ b/docs/routing_algorithms/subagent_routing.md @@ -2,7 +2,8 @@ Sub-agent-aware routing leaves parent-agent traffic with its configured routing algorithm while routing delegated sub-agent work separately. It is available on -`passthrough` and `stage_router` routes through the optional `subagents` table. +`passthrough`, `llm_classifier`, `stage_router` and `composite` routes through the +optional `subagents` table. ```toml schema_version = 1