feat(eval-routing): decouple judge/agent_judge backend+model from the agent's route

[akshaylive]

Summary

• Adds checker_context.api_route: {route, model} on TaskDefinition (4-layer merged like agent/simulation), letting a task/variant pick which backend and model the evaluation side (llm_judge/agent_judge/simulator) uses, independent of the agent under test — enabling cross-vendor grading (checking for self-preference bias) and cheaper judge models without editing every task YAML.
• The override is baked into the resolved ApiRoute's own model field before any criterion runs, so criteria stay task-blind — llm_judge only ever reads CheckContext.route.model, never checker_context/TaskDefinition directly.
• checker_context validates its own shape (unknown namespace, unknown key, or unrecognized backend name all raise) both at task-load time and after the experiment-layer merge, so a typo can't silently no-op an override.
• Separately: BedrockRoute/LiteLLMRoute no longer carry bearer_token/auth_token fields. Those secrets now flow only through the shared coder_eval.config.settings singleton, read directly by each consumer (ClaudeCodeAgent._build_sdk_env, judge_bedrock.invoke_bedrock_judge_async) at the point of use — a route object flowing through CheckContext/environment_info/logging never carries a credential.

Full design discussion in #136 (comment).

Review-driven fixes included in this PR

A multi-axis code review surfaced several real issues, fixed here:

• Bug: resolve_evaluation_route's reuse-agent-route branch shipped a bare, unqualified model alias straight to the Bedrock API when only checker_context.api_route.model was overridden (no backend override). Fixed via a new shared _bedrock_model_pair helper that also de-duplicates qualification logic previously copy-pasted across three call sites.
• Bug: judge_bedrock.invoke_bedrock_judge_async's missing-bearer-token check was a bare assert, which handle_criterion_errors(_async) was silently catching and downgrading to a scored 0.0 instead of escalating to FinalStatus.ERROR. Now raises JudgeInfrastructureError.
• Doc bug: the Checker Context guide's example paired llm_judge with route: litellm, which has no transport implemented yet — copy-pasting it would fail immediately. Fixed to route: bedrock, with the LiteLLM gap called out explicitly.
• Orchestrator._eval_route_overrides now returns a NamedTuple instead of a bare 2-tuple (closes a silent positional-swap risk); the two duplicated route-resolution call sites in Orchestrator._setup are now one _resolve_routes() helper; environment_info now records eval_model so a judge-model override is visible in run artifacts.

Known follow-up (not fixed in this PR — flagging transparently)

• The new checker_context/route-override mechanism has no dedicated test coverage yet (the diff's test-file changes are mechanical adaptations to the bearer_token/auth_token field removal, not new tests). Recommend adding coverage for resolve_evaluation_route's override paths and llm_judge's route.model fallback before relying on this in CI.
• agent_judge does not currently honor checker_context.api_route.model (its sub-agent's model comes from a hardcoded default in _default_judge_agent_config) — documented in the guide as a known gap rather than fixed, since the fix touches files outside this diff's scope.
• The LiteLLM judge transport itself is still unimplemented (tracked in llm_judge: support non-Anthropic judge routes (LiteLLM / OpenAI-compatible) #136) — route: litellm resolves cleanly but fails at grading time with a clear "not implemented yet" error.

Test plan

• make format / make check (ruff) — clean
• uv run pyright — 0 errors
• make lint (custom architectural rules incl. CE030 doc-schema-parity) — clean
• Full test suite: 4698 passed, 8 skipped (pre-existing, environment-gated, unrelated)
• Manually verified the Bedrock-qualification fix by direct execution (eu.anthropic.claude-opus-4-6-v1 now correctly qualified, was previously bare)

🤖 Generated with Claude Code

[bai-uipath]

Doesn't get us to luna — this is the selector, not the capability, and that gap is the whole reason #136 exists.

The judge can only be pointed at backends that reach Claude. The endpoint codex actually uses for luna isn't one of them, and the judge has no transport that could talk to it if it were — so nothing here moves the sonnet-4.6 → luna swap. Getting there needs a new backend on top of the transport work in #136. Fine as a prerequisite; worth saying so up front rather than merging this looking like it delivers the thing.

The simulator swap isn't covered either — its model comes from the simulation config, not the eval route.

Fix what you agree with on the rest:

• The default judge model silently changes. With no checker_context anywhere, the judge stops using the criterion default and picks up BEDROCK_MODEL — which the nightly sets to sonnet-5, which 400s on temperature (verified against live Bedrock). That's all 361 llm_judge tasks in the skills suite once the pin moves. Fix: only populate the eval route's model when an override was actually supplied.
• The unset-detection doesn't survive the docker driver — the task round-trip marks every field as set, so the model override is a silent no-op there while working on tempdir. Same fix retires it.
• No tests on any of the new paths. Every existing judge test uses a model-less route, which is why the suite stayed green through the above.
• The model half may be the wrong shape. The route half earns its keep; the model override is doing a job a judge: block on ExperimentDefaults would do more directly — set the criterion's model explicitly and the two bugs above stop existing. Worth considering landing the selector alone and bringing the model override back alongside the transport.

[bai-uipath]

LGTM overall. The transport landed, so this now does what the title says: a judge can be pointed at luna. The two grading-correctness bugs from the last pass are fixed and tested, and the judge-model floor holds. Fix what you agree with below.

• route: litellm breaks the simulator and agent_judge. Both run as Claude Code sub-agents on the resolved eval route, but only the judge reads params/env_params. The SDK env builder still falls back to the agent's own LiteLLM settings, so a task with a simulation block or an agent_judge criterion either gets an empty base URL, or silently runs on the agent's proxy: the exact misrouting LiteLLMRoute's own docstring says must never happen. Either reject that combination at load, or thread the params into the sub-agent env. This is the blocking one for me, since the simulator swap is half the reason we wanted this.
• checker_context is still an untyped bag. Two authored values, plus a hand-written allowlist validator called from two sites, plus a hand-rolled merge that records no config lineage. model's value type is never checked, so model: 5 loads clean and gets stringified into a model id. That was raised as blocking on the last review and the two new keys grew the validator instead of replacing it. Either land the typed model or say you are declining it, but it should not go quiet.
• Nothing in the repo exercises the feature. No task, no experiment variant. The luna config that proves this works lives in a Slack message. One committed variant would make it real and give CI something to catch regressions with.
• Pin api_version for the Azure path. The codex agent sets it explicitly for that deployment; the judge does not inherit it and litellm falls back to its own default, so judge and agent can hit different API versions of the same endpoint. Also: the docs say azure_ai/, the tested config says azure/. Different providers, different request shapes. Make the doc match what actually ran.

Minor: the LiteLLM secret assertion in the orchestrator tests is still vacuous, the token string it checks for is never set on the settings object it now patches.