Eval runner: --effort flag, fix --config auth bypass - #587
Merged
TheGreatAxios merged 2 commits intoAug 23, 2026
Merged
Conversation
--effort <level> threads reasoning effort through to the exec runner by writing it into the fixture workdir's local settings, the same path an interactive session uses. Rejects an effort/model pairing the target model does not accept via supportedEfforts, before any inference runs. Matrix cells can carry their own effort (provider:model:effort) alongside the run-level --effort default. Each result row records the effort used. --config was silently dropping the OAuth profile catalog (codex/xai credentials live in home-level auth stores, separate from settings.json), so any codex/xai run through --config reached the provider unauthenticated (HTTP 426/404 before a single turn). Fixed by merging OAuth profiles regardless of --config; only the programmatic globalSettingsPath test override (not a CLI flag) still opts out, for full test isolation.
…ejection test --config is a general CLI flag, not eval-only: it now composes with home-level codex/xai OAuth credentials instead of excluding them. Document that in docs/PRODUCT.md and docs/IMPLEMENTATION.md (prose and flag table) so an operator relying on --config to pin a single API-key provider on a shared/CI machine knows OAuth sessions in ~/.corbits/codex-auth.json / xai-auth.json now apply too when their config names a codex/* or xai/* provider. Export validateVariantEfforts and add a wiring-level test exercising the full parseArgs -> parseMatrix -> validateVariantEfforts path for an unsupported model/effort matrix cell (grok-composer-2.5-fast:xhigh), plus the matching accept case.
TheGreatAxios
enabled auto-merge
August 23, 2026 23:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes CL-6973.
Behavior change:
--confignow composes with home OAuth credentials--config <path>is a general CLI flag (corbits exec --config ..., not eval-only). Before this change, passing--configunconditionally excluded the home-level codex/xai OAuth profile catalog (~/.corbits/codex-auth.json,xai-auth.json) from provider resolution — a hardcodedconfigPath === undefinedcheck gated whether those credentials were merged in at all.Before: an operator on a shared or CI machine who used
--configto pin to a single API-key provider was implicitly also excluding any OAuth-authenticated codex/xai sessions on that machine —--configacted as an OAuth-exclusion mechanism, whether or not that was the intent.After:
--configonly overrides where provider definitions come from. It always composes with the home OAuth catalog. If the--configfile (or the CLI--provider/--modelflags on top of it) names acodex/*orxai/*provider, and a matching OAuth session exists on that machine, it now resolves and authenticates — where before it silently reached the provider with no credentials.What gets worse for someone: a shared or CI machine that relies on
--configspecifically to keep runs scoped to one settings file's providers, on a box where OAuth sessions also happen to exist, will now pick those up for anycodex/*/xai/*provider name in that config, rather than being excluded from them. No new opt-out flag is added for this — if isolation from home OAuth turns out to be needed, that's a separate ticket with its own design; this PR does not add a rider option for it. The only existing knob that still fully excludes OAuth is the programmaticglobalSettingsPathoption toloadConfig(not a CLI flag, used only by tests for full isolation).docs/PRODUCT.mdanddocs/IMPLEMENTATION.md(prose and the flag table) are updated in this diff to state the compose behavior plainly, and the stale code comment that asserted the opposite (src/config/index.ts) is corrected — that comment asserting--configwas meant to exclude OAuth, unvalidated, is how this shipped broken in the first place.Root cause
Validated with
git log -Stracing this line to the commit that introduced OAuth profiles: no documentation describes--configas an isolation/credential-exclusion mechanism, and no test depended on the exclusion. Confirmed via reproduction at v0.2.106:Fix: OAuth profiles are now merged in regardless of
--config. Only the programmaticglobalSettingsPathoption — never exposed as a CLI flag — still opts out.Also in this PR:
--efforton the eval runnerscripts/eval-capability.tsgains--effort <level>:src/agent/profile-types.ts.supportedEfforts(src/provider/reasoning-effort.ts) before any inference runs — a bad model/effort pairing fails the whole invocation immediately, naming the model and the levels it does accept.label=provider:model:effort(e.g.xai/thegreataxios:grok-4.6:xhigh), falling back to the run-level--effortwhen a cell omits it.reasoningEffortinto the fixture workdir's local settings — the same path an interactive session uses — so a run never touches the operator's real~/.corbits/settings.json.CaseResult.effort) so results are self-describing.Testing
No live inference anywhere in this diff.
--effortflag parsing, matrix-cell effort parsing/precedence: unit-tested against the pure parsers (evals/capability/lib.test.ts).scripts/eval-capability.test.tsnow exercises the fullparseArgs -> parseMatrix -> validateVariantEffortspath forgrok-composer-2.5-fast:xhigh, asserting the thrown message names the model and its accepted levels, plus the matching accept case for a valid pairing.--config/OAuth fix is proven intests/unit/config.test.tswith a test that stubs a synthetic OAuth profile in a fake home directory and assertsloadConfigstill resolves it asconfiguredwith a--configfile also passed — no network call, asserted on the resolved config/auth-catalog shape.bun run checkgreen (lint, typecheck, build, 5377 tests) run in the foreground.