fix(ci): support explicit refresh experiments

[mattrossman]

Motivation

When developing experiments you often need to run specific experiment(s) on your PR to validate it.

Before

If you tried to dispatch the workflow on a PR with:

experiments: codex-gpt-5.6,codex-gpt-5.6-no-skills
experiment_suite: benchmark,no-skills
merge: true

It would try to run this in 4 jobs:

# works
pnpm eval -- --experiment codex-gpt-5.6 --experiment-suite benchmark --eval example

# fails: this experiment is not in no-skills
pnpm eval -- --experiment codex-gpt-5.6 --experiment-suite no-skills --eval example

# fails: this experiment is not in benchmark
pnpm eval -- --experiment codex-gpt-5.6-no-skills --experiment-suite benchmark --eval example

# works
pnpm eval -- --experiment codex-gpt-5.6-no-skills --experiment-suite no-skills --eval example

There were 2 problems:

• some of the jobs are invalid permutations so workflow fails
• it would --merge with whatever was on main instead of results already on that branch.

After

Now, the experiments input overrides experiment_suite so it runs:

pnpm eval -- --experiment codex-gpt-5.6 --eval example
pnpm eval -- --experiment codex-gpt-5.6-no-skills --eval example

Verified by manually dispatching against this PR branch with 2 explicit experiments, to verify this overrides experiment suite:
https://github.com/supabase/evals/actions/runs/30658091594

And once more with a 3rd explicit experiment to verify it merges results with the aggregated results on the branch:
https://github.com/supabase/evals/actions/runs/30658826318

The resulting commits (now reverted) correctly refreshed results on all 3 experiments in the combined diff.

Closes AI-1005

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview	Jul 31, 2026 7:35pm