Skip to content

Eval runner: --effort flag, fix --config auth bypass - #587

Merged
TheGreatAxios merged 2 commits into
mainfrom
cl-6973-eval-runner-needs-a-effort-flag-config-breaks-provider-auth
Aug 23, 2026
Merged

Eval runner: --effort flag, fix --config auth bypass#587
TheGreatAxios merged 2 commits into
mainfrom
cl-6973-eval-runner-needs-a-effort-flag-config-breaks-provider-auth

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Closes CL-6973.

Behavior change: --config now composes with home OAuth credentials

--config <path> is a general CLI flag (corbits exec --config ..., not eval-only). Before this change, passing --config unconditionally excluded the home-level codex/xai OAuth profile catalog (~/.corbits/codex-auth.json, xai-auth.json) from provider resolution — a hardcoded configPath === undefined check gated whether those credentials were merged in at all.

Before: an operator on a shared or CI machine who used --config to pin to a single API-key provider was implicitly also excluding any OAuth-authenticated codex/xai sessions on that machine — --config acted as an OAuth-exclusion mechanism, whether or not that was the intent.

After: --config only overrides where provider definitions come from. It always composes with the home OAuth catalog. If the --config file (or the CLI --provider/--model flags on top of it) names a codex/* or xai/* provider, and a matching OAuth session exists on that machine, it now resolves and authenticates — where before it silently reached the provider with no credentials.

What gets worse for someone: a shared or CI machine that relies on --config specifically to keep runs scoped to one settings file's providers, on a box where OAuth sessions also happen to exist, will now pick those up for any codex/*/xai/* provider name in that config, rather than being excluded from them. No new opt-out flag is added for this — if isolation from home OAuth turns out to be needed, that's a separate ticket with its own design; this PR does not add a rider option for it. The only existing knob that still fully excludes OAuth is the programmatic globalSettingsPath option to loadConfig (not a CLI flag, used only by tests for full isolation).

docs/PRODUCT.md and docs/IMPLEMENTATION.md (prose and the flag table) are updated in this diff to state the compose behavior plainly, and the stale code comment that asserted the opposite (src/config/index.ts) is corrected — that comment asserting --config was meant to exclude OAuth, unvalidated, is how this shipped broken in the first place.

Root cause

const useOAuthProfiles = configPath === undefined && options.globalSettingsPath === undefined;

Validated with git log -S tracing this line to the commit that introduced OAuth profiles: no documentation describes --config as an isolation/credential-exclusion mechanism, and no test depended on the exclusion. Confirmed via reproduction at v0.2.106:

# works — passes, 5 turns, real tokens
bun scripts/eval-capability.ts --provider xai/thegreataxios --model grok-4.6 --case tier-easy --repeats 1

# fails instantly, turns=0
bun scripts/eval-capability.ts --matrix "xai/thegreataxios:grok-4.6" \
  --config /tmp/settings-low.json --case tier-easy --repeats 1
# → FAIL (Upgrade Required)   [HTTP 426]

Fix: OAuth profiles are now merged in regardless of --config. Only the programmatic globalSettingsPath option — never exposed as a CLI flag — still opts out.

Also in this PR: --effort on the eval runner

scripts/eval-capability.ts gains --effort <level>:

  • Validated against the canonical literal set in src/agent/profile-types.ts.
  • Rejected against the target model's supportedEfforts (src/provider/reasoning-effort.ts) before any inference runs — a bad model/effort pairing fails the whole invocation immediately, naming the model and the levels it does accept.
  • Matrix cells can carry their own effort: label=provider:model:effort (e.g. xai/thegreataxios:grok-4.6:xhigh), falling back to the run-level --effort when a cell omits it.
  • Applied by writing reasoningEffort into the fixture workdir's local settings — the same path an interactive session uses — so a run never touches the operator's real ~/.corbits/settings.json.
  • Recorded on every row (CaseResult.effort) so results are self-describing.

Testing

No live inference anywhere in this diff.

  • --effort flag parsing, matrix-cell effort parsing/precedence: unit-tested against the pure parsers (evals/capability/lib.test.ts).
  • Unsupported model/effort rejection, wiring-level: scripts/eval-capability.test.ts now exercises the full parseArgs -> parseMatrix -> validateVariantEfforts path for grok-composer-2.5-fast:xhigh, asserting the thrown message names the model and its accepted levels, plus the matching accept case for a valid pairing.
  • The --config/OAuth fix is proven in tests/unit/config.test.ts with a test that stubs a synthetic OAuth profile in a fake home directory and asserts loadConfig still resolves it as configured with a --config file also passed — no network call, asserted on the resolved config/auth-catalog shape.

bun run check green (lint, typecheck, build, 5377 tests) run in the foreground.

--effort <level> threads reasoning effort through to the exec runner
by writing it into the fixture workdir's local settings, the same
path an interactive session uses. Rejects an effort/model pairing the
target model does not accept via supportedEfforts, before any
inference runs. Matrix cells can carry their own effort
(provider:model:effort) alongside the run-level --effort default.
Each result row records the effort used.

--config was silently dropping the OAuth profile catalog (codex/xai
credentials live in home-level auth stores, separate from
settings.json), so any codex/xai run through --config reached the
provider unauthenticated (HTTP 426/404 before a single turn). Fixed
by merging OAuth profiles regardless of --config; only the
programmatic globalSettingsPath test override (not a CLI flag) still
opts out, for full test isolation.
@linear-code

linear-code Bot commented Aug 23, 2026

Copy link
Copy Markdown

CL-6973

…ejection test

--config is a general CLI flag, not eval-only: it now composes with
home-level codex/xai OAuth credentials instead of excluding them.
Document that in docs/PRODUCT.md and docs/IMPLEMENTATION.md (prose
and flag table) so an operator relying on --config to pin a
single API-key provider on a shared/CI machine knows OAuth sessions
in ~/.corbits/codex-auth.json / xai-auth.json now apply too when
their config names a codex/* or xai/* provider.

Export validateVariantEfforts and add a wiring-level test exercising
the full parseArgs -> parseMatrix -> validateVariantEfforts path for
an unsupported model/effort matrix cell (grok-composer-2.5-fast:xhigh),
plus the matching accept case.
@TheGreatAxios
TheGreatAxios merged commit d1594a2 into main Aug 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant