diff --git a/benchmark/SWE_ATLAS_MODEL_ORDER_CALIBRATION_REPORT.md b/benchmark/SWE_ATLAS_MODEL_ORDER_CALIBRATION_REPORT.md new file mode 100644 index 000000000..d64265a35 --- /dev/null +++ b/benchmark/SWE_ATLAS_MODEL_ORDER_CALIBRATION_REPORT.md @@ -0,0 +1,253 @@ +# SWE-Atlas model-order calibration findings + +## Summary + +The escalation router can decide that a trajectory needs rescue, but that signal does not prove +that the configured capable target is the better rescue model. The paired sample reported in +[PR #637](https://github.com/NVIDIA-NeMo/Switchyard/pull/637) made the distinction concrete: Opus +outscored GLM on RF, while GLM outscored Opus on TW. A trajectory-only router always interprets +rescue as GLM-to-Opus and cannot represent the second ordering. + +This experiment adds an optional deployment calibration gate named `expected_capable_gain`. A +positive value enables the existing trajectory judge. Zero or a negative value routes directly to +the efficient target, avoids judge calls, and prevents a judge-driven transition to a target whose +expected utility is not positive. Omitting the value preserves existing behavior. + +The live experiment verifies the mechanism, not the quality of the calibration estimate. With a +TW-derived value of `-0.10`, all 20 calibrated held-out trajectories stayed on GLM: 800 worker +calls, zero judge calls, and zero Opus calls. They scored 8/20. However, independently sampled +GLM-only arms ranged from 4/20 to 9/20, and held-out direct Opus beat direct GLM 7/20 to 4/20—the +opposite ordering from the calibration sample. A static gate is therefore a useful safety +constraint when its input is reliable, but these 20-task samples are too noisy to establish a +stable TW model order. + +This is a narrow vertical slice of the unified-routing proposal in +[issue #601](https://github.com/NVIDIA-NeMo/Switchyard/issues/601), not the proposed `auto` route. +A complete solution needs pool-owned, task-conditioned, uncertainty-aware utility estimates. + +## Problem statement + +The escalation judge observes one model's trajectory. It can detect repeated failures, false +progress, drift, desperation, or a capability gap. It cannot observe the counterfactual result of +calling a different model. In particular: + +- a failing GLM trajectory does not imply that Opus will solve the task; +- the labels `efficient` and `capable` do not establish workload-specific model ordering; +- switching may lower accuracy as well as increase cost when that ordering is reversed; and +- judge calls add cost even when deployment evidence says that switching has no expected value. + +The two signals should remain distinct: trajectory evidence answers whether rescue is needed; +calibration answers whether the proposed rescue target has positive expected utility. + +## Change + +`EscalationJudgeConfig` accepts an optional finite `expected_capable_gain`: + +```toml +[routes.calibrated_escalation] +id = "switchyard/calibrated-escalation" +type = "llm_classifier" +mode = "escalation" +classifier_target = "glm" +strong_target = "opus" +weak_target = "glm" +escalation = { confirmations = 2, expected_capable_gain = -0.10 } +``` + +| Calibrated gain | Behavior | +| --- | --- | +| Unset | Preserve the existing trajectory-judge behavior. | +| Greater than zero | Run the trajectory judge and allow its normal confirmed, one-way switch. | +| Zero or less | Serve the efficient target directly; do not call the judge or proactively select the capable target. | + +The setting is available in native TOML and the Python binding. Non-finite values are rejected +during route construction. Existing configurations remain backward compatible. The gate controls +judge-driven escalation; normal transport and context-window fallback behavior remains available. + +## Calibration rule + +For this experiment, expected capable gain is the signed difference in deployment utility: + +```text +expected_capable_gain = + (p_capable - p_efficient) + - lambda * (cost_capable - cost_efficient) +``` + +`lambda` expresses how the deployment trades one unit of task success against one synthetic +dollar. With `lambda = 0`, this reduces to the observed accuracy difference. On the original TW +sample in PR #637, direct GLM scored 8/20 and direct Opus scored 6/20, so the correctness-only +estimate was `0.30 - 0.40 = -0.10`. The configured gate therefore kept GLM and skipped the judge. + +That original sample is calibration data, not evidence that the estimate generalizes. The +held-out experiment below tests the estimate on a disjoint task sample. + +## Held-out experiment + +### Design + +- 20 tasks from `scale-ai/swe-atlas-tw@1`, disjoint from PR #637's 20-task TW sample. +- Tasks were selected deterministically by ranking the hash of + `20260905:tw-selector:`; they were not selected by outcome. +- One independent trial per task and arm. Comparison arms were direct GLM, direct Opus, patched + escalation from PR #637, and the existing capability selector. A fifth arm used calibrated + escalation with `expected_capable_gain = -0.10`. +- The live calibrated binary combined PR #637 with the directional gate so all five arms could use + one server build. The nonpositive gate returned before the escalation judge, so PR #637's judge + changes were not exercised by any calibrated trajectory. This PR's clean branch and validation + are independently based on `main`. +- GLM was `nvidia/zai-org/glm-5.2`; Opus was + `aws/anthropic/bedrock-claude-opus-4-8` with medium effort. GLM was also the classifier/judge. +- There was no agent-turn limit. Worker requests allowed up to 128,000 output tokens, the model + server timeout was 900 seconds, and the Harbor agent-timeout multiplier was 3. +- Infrastructure failures before a valid graded trajectory were excluded and rerun. A final cell + was accepted only when it had a verifier reward and at least one model request. Server-startup + and container-registry failures never entered the result matrix; all 100 final cells were valid. + +The exact task manifest, Switchyard configuration, and Slurm runner are checked into +`benchmark/tw-model-order/`. + +### Accuracy and routing + +| Arm | Correct | Accuracy | Judge/classifier calls | Opus worker trajectories | +| --- | ---: | ---: | ---: | ---: | +| Direct GLM | 4/20 | 0.200 | 0 | 0 | +| Always Opus | 7/20 | 0.350 | 0 | 20 | +| Patched escalation | 7/20 | 0.350 | 674 | 2 | +| Capability selector | 9/20 | 0.450 | 20 | 0 | +| **Calibrated escalation** | **8/20** | **0.400** | **0** | **0** | + +The calibrated arm enforced the intended routing invariant in every trajectory: all 800 recorded +model calls were GLM worker calls. The capability selector also selected GLM on every task, but it +spent one GLM classifier call per trajectory. The patched router switched on two tasks. Each had +one GLM-to-Opus transition followed by a latch with no hand-back. + +The accuracy results must not be read as a causal win for the gate. Direct GLM, calibrated +escalation, and the capability selector all used GLM for worker calls, yet scored 4/20, 8/20, and +9/20 respectively. The calibrated arm beat direct GLM on four tasks and lost none, despite having +the same worker model configuration. That gap is sampling and long-agent trajectory variance, not +a routing effect. Likewise, direct Opus's 7/20 versus direct GLM's 4/20 reverses the ordering that +produced the `-0.10` calibration value. + +The paired direct baselines tied on 17 tasks; Opus alone solved three and GLM alone solved none. +That directional result is still based on only three discordant task outcomes and should not be +treated as a stable model-ranking estimate. + +### Synthetic cost + +The inference service did not provide billable cost. These values are synthetic comparisons, not +NVIDIA prices or charges. The assumed rates match PR #637: + +| Model | Uncached input / 1M tokens | Cached input / 1M tokens | Output / 1M tokens | +| --- | ---: | ---: | ---: | +| GLM 5.2 | $0.50 | $0.05 | $2.00 | +| Opus 4.8 | $5.00 | $0.50 | $25.00 | + +For every worker, classifier, or judge call: + +```text +uncached_input_tokens = max(prompt_tokens - cached_tokens, 0) + +model_cost = ( + uncached_input_tokens * uncached_input_rate + + cached_tokens * cached_input_rate + + output_tokens * output_rate +) / 1,000,000 + +trajectory_cost = sum(worker costs) + sum classifier/judge costs +``` + +Cache-creation tokens are included in the uncached prompt remainder. The calculation includes +classifier and judge calls and excludes the Harbor verifier and cluster infrastructure. + +| Arm | Synthetic total | Cost/task | Cost/correct | +| --- | ---: | ---: | ---: | +| Direct GLM | $6.422 | $0.321 | $1.606 | +| Always Opus | $10.838 | $0.542 | $1.548 | +| Patched escalation | $12.479 | $0.624 | $1.783 | +| Capability selector | $5.036 | $0.252 | $0.560 | +| **Calibrated escalation** | **$7.520** | **$0.376** | **$0.940** | + +Skipping 20 classifier calls does not guarantee a lower per-task total in independently sampled +agent runs. The calibrated trajectories happened to make 800 worker calls, versus 573 worker calls +plus 20 classifier calls for the capability selector, so their synthetic total was higher. A +paired replay or multiple replicas would be needed to isolate judge-cost savings from worker-call +variance. + +The calibrated arm's synthetic total was 39.7% lower than patched escalation, but that comparison +also includes independently sampled worker trajectories. The patched arm spent $1.414 on 674 judge +calls. Its largest outlier, `task-6902ef3ab97fe23e2ad27253`, made 146 judge calls and 144 GLM +worker calls before two consecutive fresh `false_progress` verdicts triggered a late switch. It +then made 30 latched Opus calls, scored 0, and cost $7.392 by itself—59.2% of the patched arm's +total. The gate would remove judge and Opus spending under a negative utility estimate, but it +cannot guarantee a shorter independently sampled GLM trajectory. + +## Relationship to unified routing + +Issue #601 proposes that algorithms emit needs and confidence while a pool optimizer owns the +model catalog, cost, budget, and preferences. This change does not add the proposed model pool, +`auto` route, capability discovery, or learned/online policy. It establishes one decision boundary: + +```text +trajectory rescue evidence + + +externally calibrated model utility + | + v +allow or reject the proposed transition +``` + +A complete implementation should replace the scalar deployment constant with a pool-owned, +task-conditioned estimate, account for uncertainty in that estimate, and let the optimizer choose +any eligible model. It should preserve the current maximum-one-switch and latch behavior for a +coding-agent session unless a later design explicitly introduces safe hand-back. + +## Limitations + +- One trial per task has high variance for long stochastic coding-agent trajectories. +- Twenty paired tasks were insufficient to establish a stable TW model ordering: the held-out + direct-model result reversed the calibration sample's direction. +- A workload average does not estimate Opus's conditional value specifically on trajectories the + judge would escalate. +- A static calibration can become stale and does not include confidence intervals or exploration. +- The gate prevents a negative-value transition; it does not automatically reverse the transition, + learn model capabilities, or solve model selection by itself. +- The client requested up to 128,000 output tokens, but the longest GLM trajectory repeatedly + received length-terminated 32,768-token responses from the upstream service. No agent-turn cap + was imposed; this provider behavior contributed to the long-tail trajectory described above. + +## Validation + +The implementation was tested with Rust 1.96.1 on a Slurm compute node using: + +```text +uv sync --locked +cargo fmt --all --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test --workspace --all-features +uv run ruff check . +uv run mypy switchyard +uv run maturin develop --uv +uv run pytest tests/ -v -m "not integration" +make -C docs publish +``` + +All commands passed on the clean branch. Pytest reported 116 passed, 2 deselected, and 2 subtests +passed; the strict MkDocs build also completed successfully. + +Focused tests verify that a nonpositive calibrated gain consumes neither a judge response nor a +capable-model response, a positive gain preserves confirmed escalation, non-finite values are +rejected, and the TOML and Python surfaces carry the setting. + +## Conclusion + +The change solves the mechanical failure mode: Switchyard can now refuse an escalation when an +external utility estimate says the destination is not better, while preserving the existing route +when calibration is absent or positive. The held-out run verifies that the gate eliminates both +judge and Opus calls when disabled. + +It does not establish that `-0.10` is the right TW policy. The direct baselines reversed ordering +on the held-out tasks, and nominally identical GLM worker arms differed by five correct outcomes. +The practical next step toward issue #601 is replicated calibration with uncertainty bounds, +followed by a pool-owned task-conditioned utility estimate—not more trajectory-judge prompt +tuning. diff --git a/benchmark/tw-model-order/config.toml b/benchmark/tw-model-order/config.toml new file mode 100644 index 000000000..2a6f80b3f --- /dev/null +++ b/benchmark/tw-model-order/config.toml @@ -0,0 +1,62 @@ +schema_version = 1 + +[llm_clients.inference_api_chat] +format = "openai_chat" +base_url = "https://inference-api.nvidia.com/v1" +api_key_env = "INFERENCE_API_KEY" + +[llm_clients.inference_api_anthropic] +format = "anthropic_messages" +base_url = "https://inference-api.nvidia.com" +api_key_env = "INFERENCE_API_KEY" + +[targets.opus] +id = "aws/anthropic/bedrock-claude-opus-4-8" +llm_client = "inference_api_anthropic" + +[targets.opus.extra_body.output_config] +effort = "medium" + +[targets.glm] +id = "nvidia/zai-org/glm-5.2" +llm_client = "inference_api_chat" + +[routes.escalation] +id = "switchyard/escalation" +type = "llm_classifier" +mode = "escalation" +classifier_target = "glm" +strong_target = "opus" +weak_target = "glm" +max_output_tokens = 32768 +escalation = { confirmations = 2 } + +[routes.calibrated_escalation] +id = "switchyard/calibrated-escalation" +type = "llm_classifier" +mode = "escalation" +classifier_target = "glm" +strong_target = "opus" +weak_target = "glm" +max_output_tokens = 32768 +escalation = { confirmations = 2, expected_capable_gain = -0.10 } + +[routes.always_opus] +id = "switchyard/always-opus" +type = "passthrough" +target = "opus" + +[routes.direct_glm] +id = "switchyard/direct-glm" +type = "passthrough" +target = "glm" + +[routes.capability] +id = "switchyard/capability" +type = "llm_classifier" +mode = "capability" +classifier_target = "glm" +strong_target = "opus" +weak_target = "glm" +base_threshold = 0.5 +classify_trigger = "new_session" diff --git a/benchmark/tw-model-order/manifest.txt b/benchmark/tw-model-order/manifest.txt new file mode 100644 index 000000000..00e8415f9 --- /dev/null +++ b/benchmark/tw-model-order/manifest.txt @@ -0,0 +1,20 @@ +tw|task-6902ef3ab97fe23e2ad2726d +tw|task-6902ef3ab97fe23e2ad2721b +tw|task-6902ef3ab97fe23e2ad27277 +tw|task-6902ef3ab97fe23e2ad271f8 +tw|task-6902ef3ab97fe23e2ad27256 +tw|task-6902ef3ab97fe23e2ad2722c +tw|task-6902ef3ab97fe23e2ad27268 +tw|task-6902ef3ab97fe23e2ad2724d +tw|task-6902ef3ab97fe23e2ad2725c +tw|task-6902ef3ab97fe23e2ad2721f +tw|task-6902ef3ab97fe23e2ad2720a +tw|task-6902ef3ab97fe23e2ad271f5 +tw|task-6902ef3ab97fe23e2ad271ee +tw|task-6902ef3ab97fe23e2ad27207 +tw|task-6902ef3ab97fe23e2ad2726a +tw|task-6902ef3ab97fe23e2ad271fa +tw|task-6902ef3ab97fe23e2ad27243 +tw|task-6902ef3ab97fe23e2ad27253 +tw|task-6902ef3ab97fe23e2ad2727c +tw|task-6902ef3ab97fe23e2ad27264 diff --git a/benchmark/tw-model-order/run.sbatch b/benchmark/tw-model-order/run.sbatch new file mode 100644 index 000000000..2f6dc9518 --- /dev/null +++ b/benchmark/tw-model-order/run.sbatch @@ -0,0 +1,141 @@ +#!/bin/bash +#SBATCH --nodes=1 +#SBATCH --time=12:00:00 +#SBATCH --cpus-per-task=16 +#SBATCH --mem=60G +#SBATCH --job-name=swe-tw-selector +#SBATCH --output=slurm-%A_%a.out +#SBATCH --error=slurm-%A_%a.err + +set -uo pipefail + +root=${BENCHMARK_ROOT:?Set BENCHMARK_ROOT to a writable benchmark directory} +manifest=${MANIFEST:-"$root/manifest.txt"} +run_kind=${RUN_KIND:-matrix} +gym_dir=${GYM_DIR:?Set GYM_DIR to a NeMo Gym checkout} +credential=${INFERENCE_KEY_FILE:?Set INFERENCE_KEY_FILE to a mode-600 key file} +config=${CONFIG:-"$root/config.toml"} +compose="$gym_dir/responses_api_agents/harbor_agent/configs/docker-compose-clear-entrypoint.yaml" + +read -r -a arms <<<"${ARMS:-direct-glm always-opus patched-escalation capability-selector}" +arm_count=${#arms[@]} +task_index=$((SLURM_ARRAY_TASK_ID / arm_count + 1)) +arm_index=$((SLURM_ARRAY_TASK_ID % arm_count)) +arm=${arms[$arm_index]} +row=$(sed -n "${task_index}p" "$manifest") +IFS='|' read -r dataset task_slug <<<"$row" +if [ -z "${dataset:-}" ] || [ -z "${task_slug:-}" ]; then + echo "No manifest row $task_index for array index $SLURM_ARRAY_TASK_ID" >&2 + exit 2 +fi + +switchyard_bin=${SWITCHYARD_BIN:?Set SWITCHYARD_BIN to the server binary under test} +route=$arm +[ "$arm" = patched-escalation ] && route=escalation +[ "$arm" = capability-selector ] && route=capability +[ "$arm" = calibrated-escalation ] && route=calibrated-escalation + +task="scale-ai/$task_slug" +run_dir="$root/$run_kind-$SLURM_ARRAY_JOB_ID/$SLURM_ARRAY_TASK_ID-$dataset-$task_slug-$arm" +case_dir="$run_dir/case" +mkdir -p "$case_dir" "$run_dir/tmp" "$run_dir/xdg-cache" + +if [ ! -s "$credential" ] || [ "$(stat -c %a "$credential")" != 600 ]; then + echo "A non-empty mode-600 inference credential is required." >&2 + exit 2 +fi + +export INFERENCE_API_KEY +INFERENCE_API_KEY=$(<"$credential") +export OPENAI_API_KEY="$INFERENCE_API_KEY" +export OPENAI_API_BASE=https://inference-api.nvidia.com/v1 +export OPENAI_BASE_URL=https://inference-api.nvidia.com/v1 +export EVAL_MODEL=azure/anthropic/claude-opus-4-8 +export XDG_CACHE_HOME="$run_dir/xdg-cache" +export TMPDIR="$run_dir/tmp" +export PYTHONPATH="$gym_dir" + +port=$((20000 + (SLURM_ARRAY_JOB_ID * 97 + SLURM_ARRAY_TASK_ID) % 30000)) +session_id="$run_kind-$dataset-$task_slug-$arm-$SLURM_ARRAY_JOB_ID-$SLURM_ARRAY_TASK_ID" + +"$switchyard_bin" \ + --config "$config" \ + --host 127.0.0.1 \ + --port "$port" \ + --routing-log-file "$case_dir/routing.jsonl" \ + >"$case_dir/switchyard.log" 2>&1 & +switchyard_pid=$! +cleanup() { + kill "$switchyard_pid" 2>/dev/null || true + wait "$switchyard_pid" 2>/dev/null || true +} +trap cleanup EXIT INT TERM + +for _ in $(seq 1 60); do + curl -fsS "http://127.0.0.1:$port/health" >/dev/null && break + kill -0 "$switchyard_pid" 2>/dev/null || break + sleep 1 +done +if ! curl -fsS "http://127.0.0.1:$port/health" >/dev/null; then + echo "switchyard_start_failed" > "$case_dir/run-status.txt" + exit 3 +fi + +cd "$gym_dir" +set +e +uv run --with \ + 'harbor @ git+https://github.com/laude-institute/harbor.git@527d50deb63a5d279e8c20593c18a2cbc7f61f9e' \ + harbor run \ + --agent responses_api_agents.harbor_agent.custom_agents.terminus_2_nemo_gym:Terminus2NemoGym \ + --model "switchyard/$route" \ + --ak "api_base=http://127.0.0.1:$port/v1" \ + --ak "session_id=$session_id" \ + --ak encode_terminus_tool_history=true \ + --ak interleaved_thinking=true \ + --ak enable_summarize=true \ + --ak collect_rollout_details=false \ + --ak 'trajectory_config={"raw_content":true}' \ + --ak 'model_info={"max_input_tokens":200000,"max_output_tokens":128000,"input_cost_per_token":0.0,"output_cost_per_token":0.0}' \ + --ak 'responses_create_params={"max_output_tokens":128000}' \ + --ak nemo_model_server_timeout_sec=900 \ + --dataset "scale-ai/swe-atlas-$dataset@1" \ + --include-task-name "$task" \ + --extra-docker-compose "$compose" \ + --n-concurrent 1 \ + --max-retries 0 \ + --agent-setup-timeout-multiplier 3 \ + --agent-timeout-multiplier 3 \ + --yes \ + --jobs-dir "$case_dir/jobs" \ + --job-name "$dataset-$task_slug-$arm" \ + --verifier-env "OPENAI_API_KEY=$OPENAI_API_KEY" \ + --verifier-env "OPENAI_API_BASE=$OPENAI_API_BASE" \ + --verifier-env "OPENAI_BASE_URL=$OPENAI_BASE_URL" \ + --verifier-env "EVAL_MODEL=$EVAL_MODEL" \ + >"$case_dir/harbor.log" 2>&1 +harbor_status=$? +set -e + +curl -fsS "http://127.0.0.1:$port/v1/stats" > "$case_dir/stats.json" || true +echo "$harbor_status" > "$case_dir/harbor-exit-code" + +result=$(find "$case_dir/jobs" -mindepth 3 -maxdepth 3 -name result.json -type f -print -quit 2>/dev/null) +if [ -z "$result" ]; then + echo "missing_result" > "$case_dir/run-status.txt" + exit 4 +fi + +jq '{reward: (.verifier_result.rewards.reward // null), exception_info, agent_result}' \ + "$result" > "$case_dir/result-summary.json" +exception=$(jq -r '.exception_info // empty' "$result") +reward=$(jq -r '.verifier_result.rewards.reward // empty' "$result") +requests=$(jq -r '.total_requests // 0' "$case_dir/stats.json") +if [ "$harbor_status" -ne 0 ] || [ -n "$exception" ] || [ -z "$reward" ] || [ "$requests" -le 0 ]; then + printf 'invalid harbor=%s exception=%s reward=%s requests=%s\n' \ + "$harbor_status" "${exception:-none}" "${reward:-missing}" "$requests" \ + > "$case_dir/run-status.txt" + exit 5 +fi + +printf 'valid reward=%s requests=%s\n' "$reward" "$requests" > "$case_dir/run-status.txt" +echo "$dataset/$task_slug/$arm: reward=$reward requests=$requests" diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rs index 593c22925..a98a50fba 100644 --- a/crates/libsy/src/algorithms/escalation.rs +++ b/crates/libsy/src/algorithms/escalation.rs @@ -48,6 +48,8 @@ struct EscalationClassifier { efficient: ModelId, /// Consecutive escalate verdicts required to latch. confirmations: u32, + /// Signed deployment calibration for the capable target's expected utility gain. + expected_capable_gain: Option, } /// Builds the escalation classifier used by the shared LLM classifier route shell. @@ -60,6 +62,7 @@ pub(super) fn build_classifier( max_output_tokens: u64, ) -> Result>> { let confirmations = config.confirmations; + let expected_capable_gain = config.expected_capable_gain; let classifier: Arc> = Arc::new(EscalationClassifier { judge: escalation::build_judge( judge_target, @@ -72,6 +75,7 @@ pub(super) fn build_classifier( capable: capable_target.clone(), efficient: efficient_target.clone(), confirmations, + expected_capable_gain, }); Ok(classifier) } @@ -90,6 +94,20 @@ impl Classifier for EscalationClassifier { }); }; + // Calibration is a workload/model-pool prior. When the proposed rescue target has no + // positive expected gain, trajectory trouble is not a reason to spend a judge call or + // make a judge-driven switch; the surrounding route serves the efficient target directly. + if let Some(gain) = self.expected_capable_gain + && gain <= 0.0 + { + tracing::info!( + expected_capable_gain = gain, + target = %self.efficient, + "calibration kept the efficient tier" + ); + return Ok((decisive(&self.efficient), None)); + } + // A confirmed session stays capable without a judge call. if streak(state) >= self.confirmations { return Ok((decisive(&self.capable), None)); @@ -251,6 +269,12 @@ mod tests { /// Builds a router with escalation enabled (`confirmations=1` latches immediately). fn escalation_router() -> Result> { + escalation_router_with_gain(None) + } + + fn escalation_router_with_gain( + expected_capable_gain: Option, + ) -> Result> { Ok(Arc::new(LlmTaskClassifier::new( LlmClassifierConfig::Escalation { judge_target: ModelId::from("judge"), @@ -258,6 +282,7 @@ mod tests { capable_target: ModelId::from("capable"), contract: ClassifierContractConfig::default(), config: EscalationJudgeConfig { + expected_capable_gain, confirmations: 1, ..EscalationJudgeConfig::default() }, @@ -266,6 +291,23 @@ mod tests { )?)) } + #[test] + fn rejects_nonfinite_expected_capable_gain() { + let result = LlmTaskClassifier::new(LlmClassifierConfig::Escalation { + judge_target: ModelId::from("judge"), + efficient_target: ModelId::from("efficient"), + capable_target: ModelId::from("capable"), + contract: ClassifierContractConfig::default(), + config: EscalationJudgeConfig { + expected_capable_gain: Some(f64::NAN), + ..EscalationJudgeConfig::default() + }, + max_output_tokens: DEFAULT_JUDGE_MAX_OUTPUT_TOKENS, + }); + + assert!(matches!(result, Err(LibsyError::AlgorithmError { .. }))); + } + #[tokio::test] async fn serves_efficient_when_judge_declines() -> Result<()> { let judge = Queue::new([r#"{"escalate":false,"reason":"progressing"}"#]); @@ -286,6 +328,27 @@ mod tests { Ok(()) } + #[tokio::test] + async fn nonpositive_calibrated_gain_bypasses_judge_and_capable_target() -> Result<()> { + let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]); + let model = Queue::new(["efficient answer"]); + + let (selected_model, response) = test_drive( + escalation_router_with_gain(Some(0.0))?, + classify_request(), + queued(model, Arc::clone(&judge)), + ) + .await?; + + assert_eq!(selected_model, "efficient"); + assert_eq!( + response.llm_response.as_agg().map(completion_text), + Some("efficient answer".to_string()) + ); + assert_eq!(judge.0.lock().len(), 1, "judge response was not consumed"); + Ok(()) + } + #[tokio::test] async fn config_overrides_the_packaged_prompt() -> Result<()> { let prompts = Arc::new(Mutex::new(Vec::new())); @@ -327,12 +390,12 @@ mod tests { } #[tokio::test] - async fn upgrades_to_capable_when_judge_escalates() -> Result<()> { + async fn positive_calibrated_gain_allows_judge_escalation() -> Result<()> { let judge = Queue::new([r#"{"escalate":true,"reason":"stuck in a loop"}"#]); let model = Queue::new(["efficient draft", "capable answer"]); let (selected_model, response) = test_drive( - escalation_router()?, + escalation_router_with_gain(Some(0.1))?, classify_request(), queued(model, judge), ) diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rs index 54ded690c..1b8d260ac 100644 --- a/crates/libsy/src/algorithms/util/escalation.rs +++ b/crates/libsy/src/algorithms/util/escalation.rs @@ -39,13 +39,20 @@ const FIRST_USER_CHARS: usize = 2_000; /// Backstop on the assembled transcript; the per-message caps normally bind first. const MAX_REQUEST_CHARS: usize = 18_000; -/// The tuning surface for the trajectory judge. +/// The tuning surface for trajectory escalation. /// /// The routing settings retain their benchmarked defaults. Everything else is a fixed invariant /// (the constants above). #[derive(Clone, Debug, Deserialize)] #[serde(default, deny_unknown_fields)] pub struct EscalationJudgeConfig { + /// Externally calibrated expected utility gain from switching to the capable target. + /// + /// A positive value enables trajectory judging. Zero or a negative value keeps the + /// efficient target for normal serving and avoids judge calls. `None` preserves the legacy + /// uncalibrated behavior and enables judging. Transport and context-window fallbacks remain + /// available. + pub expected_capable_gain: Option, /// Consecutive escalate verdicts required before a turn moves to the capable tier, which /// is also the turn that latches the session. Any decline clears the streak. /// `1` escalates on the first verdict; the router's main cost dial. @@ -61,6 +68,12 @@ impl EscalationJudgeConfig { /// Rejects settings that would leave the judge with nothing useful to read. fn validate(&self) -> Result<()> { let reject = |message: String| Err(LibsyError::AlgorithmError { message }); + if self + .expected_capable_gain + .is_some_and(|gain| !gain.is_finite()) + { + return reject("expected_capable_gain must be finite when set".to_string()); + } if self.confirmations == 0 { return reject("confirmations must be at least 1".to_string()); } @@ -80,6 +93,7 @@ impl EscalationJudgeConfig { impl Default for EscalationJudgeConfig { fn default() -> Self { Self { + expected_capable_gain: None, confirmations: 2, recent_turn_window: 28, window_message_chars: 500, diff --git a/crates/switchyard-py/src/libsy_bindings.rs b/crates/switchyard-py/src/libsy_bindings.rs index 0571a3e12..8236618f8 100644 --- a/crates/switchyard-py/src/libsy_bindings.rs +++ b/crates/switchyard-py/src/libsy_bindings.rs @@ -88,6 +88,7 @@ impl PyEscalationClassifierConfig { #[new] #[pyo3(signature = ( *, + expected_capable_gain=None, confirmations=2, recent_turn_window=28, window_message_chars=500, @@ -97,6 +98,7 @@ impl PyEscalationClassifierConfig { ))] #[allow(clippy::too_many_arguments)] fn new( + expected_capable_gain: Option, confirmations: u32, recent_turn_window: usize, window_message_chars: usize, @@ -107,6 +109,7 @@ impl PyEscalationClassifierConfig { Ok(Self { contract: classifier_contract(prompt, response_format_type)?, judge: EscalationJudgeConfig { + expected_capable_gain, confirmations, recent_turn_window, window_message_chars, diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rs index c1fea1464..7c86a1c1f 100644 --- a/crates/switchyard-runner/src/config.rs +++ b/crates/switchyard-runner/src/config.rs @@ -804,6 +804,29 @@ confidence_threshold = 0.5 Ok(()) } + #[test] + fn escalation_accepts_finite_expected_capable_gain() -> RunnerResult<()> { + let calibrated = VALID_CONFIG.replace( + "base_threshold = 0.5", + "base_threshold = 0.5\nescalation = { confirmations = 2, expected_capable_gain = -0.1 }", + ); + + runner_from_toml(&calibrated)?; + Ok(()) + } + + #[test] + fn escalation_rejects_nonfinite_expected_capable_gain() { + let calibrated = VALID_CONFIG.replace( + "base_threshold = 0.5", + "base_threshold = 0.5\nescalation = { confirmations = 2, expected_capable_gain = nan }", + ); + + assert!( + error_message(&calibrated).contains("expected_capable_gain must be finite when set") + ); + } + #[test] fn classifier_judge_completion_caps_are_configurable() -> RunnerResult<()> { let capability = VALID_CONFIG.replace( diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md index 1ed9e8085..c8ae54153 100644 --- a/docs/routing_algorithms/escalation_router_routing.md +++ b/docs/routing_algorithms/escalation_router_routing.md @@ -54,7 +54,9 @@ the prompt in `json_object` mode. ## How the decision works -For each turn on an unlatched session, Switchyard: +When `expected_capable_gain` is set to zero or a negative number, Switchyard +routes directly to the weak target and skips the judge. Otherwise, for each turn +on an unlatched session, Switchyard: 1. Calls the weak target and buffers its reply. 2. Appends that reply to the transcript and asks the judge to rule on the @@ -74,7 +76,9 @@ A latched session routes straight to the strong target with no judge call: ```mermaid %%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%% flowchart LR - t["turn"] --> p{"streak >= confirmations?"} + t["turn"] --> g{"calibrated gain <= 0?"} + g -->|yes| w0["route weak; skip judge"] + g -->|no or unset| p{"streak >= confirmations?"} p -->|yes| s["route strong; skip judge"] p -->|no| c["call weak, buffer reply"] c --> j["judge the completed turn"] @@ -83,7 +87,7 @@ flowchart LR j -->|escalate, confirmed| l["discard weak reply; serve strong"] classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px; - class t,p,s,c,j,w,l box; + class t,g,w0,p,s,c,j,w,l box; ``` A judge that times out, errors, or returns an unparseable verdict fails open: the @@ -98,20 +102,33 @@ compatibility guidance as the LLM classifier judge. See ## Tuning options -The judge exposes three settings. Their defaults are the benchmarked +Escalation exposes four settings. Their defaults are the benchmarked configuration, so a bare `escalation = {}` is a valid, tuned route: | Key | Default | Meaning | |---|---|---| +| `expected_capable_gain` | unset | Externally calibrated, signed expected-utility gain from switching to the capable target. A positive value enables judging; zero or a negative value keeps the efficient target and skips judge calls. Must be finite. | | `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. | | `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. | | `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. | +The utility gate controls judge-driven escalation. The route's normal fallback +to the other target remains available if the weak target is unavailable or +exceeds its context window. + `confirmations` is the main cost dial. `1` latches sooner and spends more on the strong tier. `2` or higher requires a session identity, because the streak is retained per session — without one, every turn starts from zero and the route never latches. Clients supply it with `x-switchyard-session-id`. +`expected_capable_gain` is a deployment-level calibration input, not a value the +trajectory judge learns. Define utility for the deployment, estimate the +capable target's expected utility minus the efficient target's on comparable +traffic, and set the signed difference. This prevents judge-driven escalation +when model ordering is reversed or the capable target's quality gain does not +justify its cost. Omit it when no calibration exists to preserve the normal +trajectory judge behavior. + Anchor and transcript caps remain fixed. Set the route-level `max_output_tokens` key to change the judge's reply budget. Any decline still resets the streak to zero. diff --git a/tests/test_libsy_minimal_bindings.py b/tests/test_libsy_minimal_bindings.py index 02ac614e4..21d98c681 100644 --- a/tests/test_libsy_minimal_bindings.py +++ b/tests/test_libsy_minimal_bindings.py @@ -12,6 +12,7 @@ Algorithm, ContextWindowExceededError, CustomClassifierConfig, + EscalationClassifierConfig, LlmClassifierConfig, LlmResponse, RoutingOutcome, @@ -308,6 +309,28 @@ async def call(self, request: dict[str, Any]) -> dict[str, Any]: assert response["model"] == "weak" +async def test_escalation_config_nonpositive_gain_bypasses_judge() -> None: + """Verify that model-order calibration reaches the native escalation router.""" + + weak = EchoClient("weak") + algorithm = algorithms.llm_classifier( + LlmClassifierConfig.escalation( + "judge", + "weak", + "strong", + config=EscalationClassifierConfig(expected_capable_gain=0.0), + ), + ) + + selected_model, response = await run_algorithm(algorithm, {"weak": weak}) + + assert selected_model == "weak" + assert response["model"] == "weak" + assert len(weak.calls) == 1 + assert weak.calls[0]["model"] == "weak" + assert weak.calls[0]["messages"] == request_body()["messages"] + + def test_classifier_config_rejects_unknown_response_format() -> None: invalid_response_format: Any = "yaml"