Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
bb95da8
feat: add opencode harness running Kimi K3 via Vercel AI Gateway
mattrossman Jul 24, 2026
88e765a
Merge remote-tracking branch 'origin/main' into feat/opencode-kimi-k3…
mattrossman Jul 24, 2026
1b19fb4
refactor: drop redundant model declaration from opencode config
mattrossman Jul 24, 2026
ddaf4af
fix: provide AI_GATEWAY_API_KEY to eval refresh runs
mattrossman Jul 24, 2026
b33bc2a
chore: refresh eval results
github-actions[bot] Jul 25, 2026
d79b20e
chore: refresh eval results
github-actions[bot] Jul 25, 2026
8079586
chore: trim workflow guard comment
mattrossman Jul 25, 2026
ca0f910
fix: parse opencode's real error envelope shape
mattrossman Jul 27, 2026
53c0032
chore: shorten harness debug error log prefix
mattrossman Jul 27, 2026
43c0997
fix: enable opencode thinking so reasoning records are emitted
mattrossman Jul 27, 2026
cbb1e3b
chore: comment about models.dev live dep
mattrossman Jul 27, 2026
e8d0879
fix: disable opencode's title agent to avoid an unmeasured cross-vend…
mattrossman Jul 27, 2026
4f40366
chore: fix formatting
mattrossman Jul 27, 2026
48b7154
chore: refresh eval results
github-actions[bot] Jul 27, 2026
7e861b0
chore: refresh eval results
github-actions[bot] Jul 27, 2026
fd064d8
Merge remote-tracking branch 'origin/main' into feat/opencode-kimi-k3…
mattrossman Jul 27, 2026
36b20d8
chore: refresh eval results
github-actions[bot] Jul 28, 2026
472ad26
chore: refresh eval results
github-actions[bot] Jul 28, 2026
df58341
fix: capitalize Kimi K3's model label in the results dashboard
mattrossman Jul 28, 2026
07e5f30
Merge remote-tracking branch 'origin/feat/opencode-kimi-k3-via-gatewa…
mattrossman Jul 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
@@ -1,3 +1,10 @@
# AI SDK Core direct-provider credentials.
# Claude Code harness and Anthropic ai-sdk experiments.
Comment thread
mattrossman marked this conversation as resolved.
ANTHROPIC_API_KEY=

# Codex harness and OpenAI ai-sdk experiments. The default LLM judge is an
# OpenAI model, so rubric-scored evals need this key regardless of which
# harness produced the run.
OPENAI_API_KEY=

# Vercel AI Gateway. The opencode harness routes through this.
AI_GATEWAY_API_KEY=
8 changes: 8 additions & 0 deletions .github/workflows/eval-refresh.yml
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,7 @@ jobs:
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
steps:
- name: Checkout
uses: actions/checkout@9f698171ed81b15d1823a05fc7211befd50c8ae0 # v6.0.3
Expand Down Expand Up @@ -286,6 +287,7 @@ jobs:
{
echo "ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}"
echo "OPENAI_API_KEY=${OPENAI_API_KEY}"
echo "AI_GATEWAY_API_KEY=${AI_GATEWAY_API_KEY}"
} > .env

- name: Run evals
Expand All @@ -300,6 +302,12 @@ jobs:
--runs "${{ needs.prepare.outputs.runs }}" \
--timeout-sec "${{ needs.prepare.outputs.timeout_sec }}"

# A skipped experiment (e.g. missing API key) exits 0 without writing results.
if [ ! -f "results/${{ matrix.experiment }}/${{ matrix.eval_id }}.json" ]; then
echo "::error::no result written for ${{ matrix.experiment }} x ${{ matrix.eval_id }} (experiment skipped?)"
exit 1
fi

- name: Upload raw results
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
Expand Down
Loading