Problem
Real runs spend API calls, and the full arms multiply quickly (models x cases x cues x conditions, plus committee rounds). We need a first-class way to bound and account for that spend, and to subsample cases deterministically so a cheap pilot and the full run are drawn from the same, reproducible pool. The gateway already has retry/rate-limit handling (#35); this issue is the policy and accounting layer on top.
What to build
- A
run_budget config block: max_cases, max_calls, seed for subsampling. Subsampling must be deterministic given the seed (same seed -> same case subset), so a pilot is a strict subset of the larger run.
- A lightweight call counter/accountant that tallies calls (and, if the provider returns it, tokens) per model and writes it into the run summary, so we can see what each arm cost.
- A
--max-calls guard that stops a run cleanly (flushing partial results) when the budget is hit, rather than running unbounded.
- Log, do not silently drop: if a run is truncated by budget, the summary must say how many cases were skipped so a truncated run is never mistaken for full coverage.
Acceptance
- Two runs with the same subsampling seed select the same cases; a larger
max_cases is a superset of the smaller.
- The run summary reports calls-per-model and total, and flags any budget truncation.
- Unit tests cover deterministic subsampling and the max-calls stop with partial-result flush.
Gemini is the default backend for now; the accountant reads whatever usage the gateway exposes and degrades gracefully when a provider reports none.
Problem
Real runs spend API calls, and the full arms multiply quickly (models x cases x cues x conditions, plus committee rounds). We need a first-class way to bound and account for that spend, and to subsample cases deterministically so a cheap pilot and the full run are drawn from the same, reproducible pool. The gateway already has retry/rate-limit handling (#35); this issue is the policy and accounting layer on top.
What to build
run_budgetconfig block:max_cases,max_calls,seedfor subsampling. Subsampling must be deterministic given the seed (same seed -> same case subset), so a pilot is a strict subset of the larger run.--max-callsguard that stops a run cleanly (flushing partial results) when the budget is hit, rather than running unbounded.Acceptance
max_casesis a superset of the smaller.Gemini is the default backend for now; the accountant reads whatever usage the gateway exposes and degrades gracefully when a provider reports none.