Eval runner: --effort flag, fix --config auth bypass

docs/IMPLEMENTATION.md

@@ -283,6 +283,8 @@ OpenAI-compatible `baseURL` values are normalized during provider resolution. A
 
 `--config <path>` replaces the global settings file as the provider source (useful for CI to inject a provider per run). The per-repo `.corbits/settings.json` selection still applies on top of a `--config` source (definitions come from `--config`, selection from the local file; CLI `--provider`/`--model` override both). A provider must be defined in one of these settings files; there is no environment-variable fallback.
 
+`--config` composes with, rather than replaces, the home-level OAuth profile catalog: codex/xai credentials live in `~/.corbits/codex-auth.json` and `xai-auth.json`, entirely separate from settings.json, and are merged into the resolved provider catalog on every run regardless of `--config` (CL-6973). A `--config` file that names a `codex/*` or `xai/*` provider by ID does not by itself grant that provider's credentials — those come from the OAuth store whenever a matching profile exists there, independent of which settings file supplied the provider definitions. The only way to fully exclude the home OAuth catalog is the programmatic `globalSettingsPath` option to `loadConfig`, used by tests for full isolation; it is not exposed as a CLI flag.
+
 ### Profiles (`src/config/profiles.ts`)
 
 Profiles supply per-project or named-profile overrides for `model`, `maxTurns`, and `systemPromptExtensions` (the only allowed keys; any other key is rejected on load).
@@ -323,7 +325,7 @@ Printed by `corbits --help` / `-h` from `CLI_HELP_TEXT` in `src/config/index.ts`
 | `resume <session-id>`            | —                          | Reopen a specific session                                                                                                                                                                                                                                                    |
 | `resume --pick` / `--list`       | —                          | Interactive session picker                                                                                                                                                                                                                                                   |
 | `--cwd <dir>`                    | `process.cwd()`            | Working directory                                                                                                                                                                                                                                                            |
-| `--config <path>`                | `~/.corbits/settings.json` | Settings file to use                                                                                                                                                                                                                                                         |
+| `--config <path>`                | `~/.corbits/settings.json` | Settings file to use for provider definitions; composes with (does not exclude) home-level codex/xai OAuth credentials                                                                                                                                                       |
 | `--provider <name>`              | from settings              | Select a configured provider                                                                                                                                                                                                                                                 |
 | `--model <id>`                   | provider default           | Select a model for the active provider                                                                                                                                                                                                                                       |
 | `--profile <name>`               | —                          | Settings profile                                                                                                                                                                                                                                                             |

docs/PRODUCT.md

@@ -132,7 +132,7 @@ The exact turn thresholds are model-family-dependent (tighter for models with ob
 
 ## Configuration
 
-Providers and models are configured in `~/.corbits/settings.json` (holds providers + credentials), with a selection-only per-repo `.corbits/settings.json` override. Select at launch with `--provider` / `--model`, or point at an alternate file with `--config <path>`. Credentials are read only from these settings files — there is no environment-variable override and `.env` files are not loaded, so a stale or exported key can't shadow the configured provider. The agent is denied read access to both settings files.
+Providers and models are configured in `~/.corbits/settings.json` (holds providers + credentials), with a selection-only per-repo `.corbits/settings.json` override. Select at launch with `--provider` / `--model`, or point at an alternate file with `--config <path>`. `--config` only overrides where provider _definitions_ come from; it composes with, rather than replaces, credentials for codex/xai OAuth-profile providers, which live in separate home-level auth stores (`~/.corbits/codex-auth.json`, `xai-auth.json`) and are merged into the catalog regardless of `--config`. Credentials are read only from these settings files and the OAuth auth stores — there is no environment-variable override and `.env` files are not loaded, so a stale or exported key can't shadow the configured provider. The agent is denied read access to both settings files.
 
 ## Optional Capabilities (plugins)
 

evals/capability/lib.test.ts

@@ -78,6 +78,7 @@ function sampleResult(over: Partial<CaseResult> = {}): CaseResult {
     behaviors: over.behaviors ?? null,
     providerFallback: over.providerFallback ?? null,
     diagnostics: over.diagnostics ?? null,
+    effort: over.effort ?? null,
   };
 }
 
@@ -327,6 +328,41 @@ describe("parseMatrix", () => {
     const v = parseMatrix("xai:", { model: "grok-4.5" });
     expect(v[0]).toEqual({ id: "xai:grok-4.5", provider: "xai", model: "grok-4.5" });
   });
+
+  test("parses a third colon segment as effort", () => {
+    const v = parseMatrix("xai/thegreataxios:grok-4.6:xhigh", {});
+    expect(v[0]).toEqual({
+      id: "xai/thegreataxios:grok-4.6",
+      provider: "xai/thegreataxios",
+      model: "grok-4.6",
+      effort: "xhigh",
+    });
+  });
+
+  test("labeled cell can also carry an effort segment", () => {
+    const v = parseMatrix("fast=xai:grok-4.6:high", {});
+    expect(v[0]).toEqual({ id: "fast", provider: "xai", model: "grok-4.6", effort: "high" });
+  });
+
+  test("a trailing segment that is not a real effort literal falls through to the model", () => {
+    // "grok-4.6:not-an-effort" has no valid effort literal in the third slot,
+    // so the whole thing after the first colon is the model id.
+    const v = parseMatrix("xai:grok-4.6:not-an-effort", {});
+    expect(v[0]!.provider).toBe("xai");
+    expect(v[0]!.model).toBe("grok-4.6:not-an-effort");
+    expect(v[0]!.effort).toBeUndefined();
+  });
+
+  test("--effort fallback applies to a cell that doesn't specify its own", () => {
+    const v = parseMatrix("xai:grok-4.6,openai:gpt-5", { effort: "medium" });
+    expect(v[0]!.effort).toBe("medium");
+    expect(v[1]!.effort).toBe("medium");
+  });
+
+  test("a cell's own effort wins over the --effort fallback", () => {
+    const v = parseMatrix("xai:grok-4.6:high", { effort: "medium" });
+    expect(v[0]!.effort).toBe("high");
+  });
 });
 
 describe("expandMatrix", () => {

evals/capability/lib.ts

@@ -6,6 +6,7 @@
 import { readdir, readFile, stat } from "node:fs/promises";
 import { join, resolve } from "node:path";
 import { runWithEvalHttpEnv, evalHttpEnvGet } from "../../src/tools/eval-http-env.js";
+import { isReasoningEffort, type ReasoningEffort } from "../../src/provider/reasoning-effort.js";
 import {
   isNumericBehaviorMetric,
   parseBehaviorMetrics,
@@ -100,6 +101,8 @@ export interface EvalVariant {
   id: string;
   provider?: string;
   model?: string;
+  /** Reasoning effort for this cell; overrides the run-level --effort. */
+  effort?: ReasoningEffort;
 }
 
 /**
@@ -149,6 +152,8 @@ export interface CaseResult {
   providerFallback: ProviderFallbackInfo | null;
   /** Per-cell diagnostics for debugging eval failures; null when unavailable. */
   diagnostics: EvalDiagnostics | null;
+  /** Requested reasoning effort for this cell (--effort or matrix cell); null when unset. */
+  effort: ReasoningEffort | null;
   textPreview?: string;
 }
 
@@ -538,15 +543,17 @@ export function defaultVariantId(provider?: string, model?: string): string {
  * Parse a matrix string of variants.
  * Formats (comma-separated cells):
  *   - `provider:model`
+ *   - `provider:model:effort`
  *   - `provider/model` (slash only when no colon)
- *   - `label=provider:model`
- * Empty / omitted → single default variant (caller provider/model flags).
+ *   - `label=provider:model[:effort]`
+ * Empty / omitted → single default variant (caller provider/model/effort flags).
  * Each expanded cell must have both provider and model (after applying
- * `--provider`/`--model` as cell defaults when a side is omitted).
+ * `--provider`/`--model` as cell defaults when a side is omitted); effort
+ * falls back to `--effort` when the cell does not specify its own.
  */
 export function parseMatrix(
   matrix: string | undefined,
-  fallback: { provider?: string; model?: string },
+  fallback: { provider?: string; model?: string; effort?: ReasoningEffort },
 ): EvalVariant[] {
   if (matrix === undefined || matrix.trim().length === 0) {
     const id = defaultVariantId(fallback.provider, fallback.model);
@@ -555,6 +562,7 @@ export function parseMatrix(
         id,
         ...(fallback.provider !== undefined ? { provider: fallback.provider } : {}),
         ...(fallback.model !== undefined ? { model: fallback.model } : {}),
+        ...(fallback.effort !== undefined ? { effort: fallback.effort } : {}),
       },
     ];
   }
@@ -571,7 +579,7 @@ export function parseMatrix(
 function parseMatrixCell(
   cell: string,
   index: number,
-  fallback: { provider?: string; model?: string },
+  fallback: { provider?: string; model?: string; effort?: ReasoningEffort },
 ): EvalVariant {
   let label: string | undefined;
   let rest = cell;
@@ -582,8 +590,17 @@ function parseMatrixCell(
   }
   let provider: string | undefined;
   let model: string | undefined;
+  let effort: ReasoningEffort | undefined;
   if (rest.includes(":")) {
-    const [p, ...mParts] = rest.split(":");
+    const parts = rest.split(":");
+    // provider:model or provider:model:effort — the last segment is treated
+    // as effort only when it parses as a real reasoning-effort literal, so a
+    // model id that happens to contain a colon still falls through cleanly.
+    if (parts.length >= 3 && isReasoningEffort(parts.at(-1)!.trim())) {
+      effort = parts.at(-1)!.trim() as ReasoningEffort;
+      parts.pop();
+    }
+    const [p, ...mParts] = parts;
     provider = p!.trim() || undefined;
     model = mParts.join(":").trim() || undefined;
   } else if (rest.includes("/")) {
@@ -597,11 +614,12 @@ function parseMatrixCell(
   }
   provider = provider ?? fallback.provider;
   model = model ?? fallback.model;
+  effort = effort ?? fallback.effort;
   if (provider === undefined || model === undefined) {
     throw new Error(`matrix cell ${index + 1} "${cell}" must specify both provider and model`);
   }
   const id = label ?? defaultVariantId(provider, model);
-  return { id, provider, model };
+  return { id, provider, model, ...(effort !== undefined ? { effort } : {}) };
 }
 
 /** Cartesian product of cases × variants (cases outer for stable progress). */
@@ -744,6 +762,7 @@ function parseCaseResult(raw: unknown): CaseResult {
     behaviors: parseBehaviorMetrics(raw.behaviors),
     providerFallback: parseProviderFallback(raw.providerFallback),
     diagnostics: parseEvalDiagnostics(raw.diagnostics),
+    effort: isReasoningEffort(raw.effort) ? raw.effort : null,
     ...(typeof raw.textPreview === "string" ? { textPreview: raw.textPreview } : {}),
   };
 }

scripts/eval-capability.test.ts

@@ -5,7 +5,14 @@ import { join } from "node:path";
 import { execFile } from "node:child_process";
 import { promisify } from "node:util";
 
-import { initEvalGitRepo, mapPool, parseArgs, buildEvalDiagnostics } from "./eval-capability.js";
+import {
+  initEvalGitRepo,
+  mapPool,
+  parseArgs,
+  buildEvalDiagnostics,
+  validateVariantEfforts,
+} from "./eval-capability.js";
+import { parseMatrix } from "../evals/capability/lib.js";
 import type { Config } from "../src/config/index.js";
 
 const execFileAsync = promisify(execFile);
@@ -91,6 +98,22 @@ describe("parseArgs", () => {
     expect(() => parseArgs(["--matrix", ":grok-4.5"])).toThrow(/both provider and model/);
   });
 
+  test("--effort accepts a canonical literal", () => {
+    const opts = parseArgs(["--provider", "foo", "--model", "bar", "--effort", "high"]);
+    expect(opts.effort).toBe("high");
+  });
+
+  test("--effort rejects an unknown literal", () => {
+    expect(() => parseArgs(["--provider", "foo", "--model", "bar", "--effort", "bogus"])).toThrow(
+      /--effort must be one of/,
+    );
+  });
+
+  test("--matrix cell can carry its own effort as a third segment", () => {
+    const opts = parseArgs(["--matrix", "xai/thegreataxios:grok-4.6:xhigh"]);
+    expect(opts.matrix).toBe("xai/thegreataxios:grok-4.6:xhigh");
+  });
+
   test("parsed defaults never equal xai/thegreataxios", () => {
     const help = parseArgs(["--help"]);
     const pair = parseArgs(["--provider", "foo", "--model", "bar"]);
@@ -158,6 +181,35 @@ describe("parseArgs", () => {
   });
 });
 
+describe("validateVariantEfforts", () => {
+  // Wiring-level regression: parseArgs -> parseMatrix -> validateVariantEfforts,
+  // the same path main() runs before any inference. A matrix cell pairing an
+  // effort the model does not accept must fail fast, naming the model and its
+  // accepted levels, rather than silently falling back to the provider default
+  // and poisoning the matrix.
+  test("rejects an unsupported model/effort matrix cell before any inference runs", async () => {
+    const opts = parseArgs(["--matrix", "xai/thegreataxios:grok-composer-2.5-fast:xhigh"]);
+    const variants = parseMatrix(opts.matrix, {
+      ...(opts.provider !== undefined ? { provider: opts.provider } : {}),
+      ...(opts.model !== undefined ? { model: opts.model } : {}),
+      ...(opts.effort !== undefined ? { effort: opts.effort } : {}),
+    });
+    await expect(validateVariantEfforts(variants, opts)).rejects.toThrow(
+      /grok-composer-2\.5-fast.*does not support reasoning effort "xhigh".*supported: low, medium, high/s,
+    );
+  });
+
+  test("accepts a supported model/effort matrix cell", async () => {
+    const opts = parseArgs(["--matrix", "xai/thegreataxios:grok-4.6:xhigh"]);
+    const variants = parseMatrix(opts.matrix, {
+      ...(opts.provider !== undefined ? { provider: opts.provider } : {}),
+      ...(opts.model !== undefined ? { model: opts.model } : {}),
+      ...(opts.effort !== undefined ? { effort: opts.effort } : {}),
+    });
+    await expect(validateVariantEfforts(variants, opts)).resolves.toBeUndefined();
+  });
+});
+
 describe("mapPool", () => {
   test("N overlapping jobs with concurrency N finish in ~one job duration", async () => {
     const jobMs = 80;