feat(agents): add OpenCode harness with opt-in [opencode] extra

README.md

@@ -11,14 +11,14 @@
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
-Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
+Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Not an "agentic coding" benchmark: it measures how
 effective your CLI and skills are when used by coding agents.
 
 Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
-prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
+prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
@@ -33,14 +33,14 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, and Antigravity (Gemini) today, extensible via a plugin SPI
+- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -209,6 +209,7 @@ alone.
 | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
+| [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

docker/Dockerfile

@@ -49,15 +49,21 @@ COPY src/ ./src/
 COPY experiments/default.yaml ./experiments/default.yaml
 
 # Codex, Antigravity, and litellm are always baked into the image -- codex/
-# antigravity are peers to the claude-code agent installed above, so all
-# built-in agents ship in every build; litellm backs the `checker_context.
-# api_route.route: litellm` judge dispatch (see pyproject.toml's `litellm`
-# extra), which every DockerRunner-isolated task needs available IN-container
-# since the checker runs there too, not just on the host. `openai-codex`
-# (+ its pinned cli-bin), `google-antigravity` (which bundles its
-# `localharness` binary as a manylinux wheel), and `litellm` all come from
-# public PyPI, so this needs no private-index credentials. The RUN below
-# always passes `--extra codex --extra antigravity --extra litellm`.
+# antigravity are peers to the claude-code agent installed above; litellm backs
+# the `checker_context.api_route.route: litellm` judge dispatch (see
+# pyproject.toml's `litellm` extra), which every DockerRunner-isolated task
+# needs available IN-container since the checker runs there too, not just on the
+# host. `openai-codex` (+ its pinned cli-bin), `google-antigravity` (which
+# bundles its `localharness` binary as a manylinux wheel), and `litellm` all
+# come from public PyPI, so this needs no private-index credentials. The RUN
+# below always passes `--extra codex --extra antigravity --extra litellm`.
+#
+# NOT every built-in agent ships here: `opencode` is registered unconditionally
+# but its CLI is a Node package (`npm install -g opencode-ai`), absent from this
+# image, and no OpenCode credentials are in SandboxConfig.env_passthrough -- so
+# `--driver docker` does not support it. Adding it means a pinned version that
+# travels with the release tag (as CLAUDE_CODE_VERSION does) plus an
+# env_passthrough block; see docs/agents/OPENCODE.md "Running in Docker".
 #
 # CODER_EVAL_UV_EXTRAS carries ADDITIONAL opt-in extras on top of those; it
 # defaults to none. `make docker-image-full` passes `--extra uipath`, which

docs/EXTENDING.md

@@ -285,8 +285,8 @@ The base package ships **no** plugin rates; only the built-in table.
 ## See also
 
 - [Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) ·
-  [Antigravity](agents/ANTIGRAVITY.md) — the built-in agents, each registered via
-  this same SPI
+  [Antigravity](agents/ANTIGRAVITY.md) · [OpenCode](agents/OPENCODE.md) — the
+  built-in agents, each registered via this same SPI
 - [Task Definition Guide](TASK_DEFINITION_GUIDE.md) — the criterion catalogue
 - [CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md) — architecture
   and extension points in depth

docs/USER_GUIDE.md

@@ -39,7 +39,7 @@ coder-eval run tasks/hello_date.yaml --stream full  # live LLM output
 | `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, token/USD budget caps, tools, plugins, and SDK options. |
 | `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-5`) |
 | `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) |
-| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, or a plugin kind). |
+| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, `opencode`, or a plugin kind). |
 | `--repeats` | Run each `(task, variant)` N times (≥1); overrides experiment/variant `repeats:`. See [Replicates](#replicates). |
 | `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. |
 | `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [Bring Your Own Dataset](DATASETS.md). |

docs/agents/HARNESS_PARITY.md

@@ -1,6 +1,6 @@
 # Run-Limit Parity
 
-One task file, run on three harnesses, must be the same task. `run_limits.max_turns`
+One task file, run on any harness, must be the same task. `run_limits.max_turns`
 was the field that broke that promise hardest: Claude Code enforced it, and Codex and
 Antigravity accepted it and never read it, so `max_turns: 6` ran capped on one
 backend and unbounded on the other two.
@@ -10,12 +10,12 @@ This page is the contract for what each run limit means per harness, plus the sh
 
 ## The table
 
-| Limit | claude-code | codex | antigravity |
-|---|---|---|---|
-| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) |
-| `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) |
-| `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
-| `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` |
+| Limit | claude-code | codex | antigravity | opencode |
+|---|---|---|---|---|
+| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) | native step cap (the CLI's own agent-loop steps) |
+| `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group |
+| `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
+| `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) |
 
 ## `max_turns` counts visible turns on Codex and Antigravity
 
@@ -42,6 +42,14 @@ same number bounds very different amounts of work: under a prompt that encourage
 batching, a cap of N here permits many more than N tool calls, where it buys exactly
 N on the other two.
 
+**OpenCode also keeps a native unit — its stream's own steps.** Unlike Codex and
+Antigravity, `opencode run` executes a real multi-step agent loop per invocation
+and streams it (`step_start` / `step_finish`), so the natural agent-loop unit
+exists and is honored: `max_turns: N` allows N complete steps and cuts the run
+when step N+1 begins, with the completed steps' tokens intact. A step is one
+assistant generation and may carry several tool calls — so, as with claude-code,
+the same number is a looser tool-call budget than on the visible-turn backends.
+
 **So holding `max_turns` constant across harnesses does not hold the budget
 constant.** If you are A/B-ing across backends and the cap is close to binding, that
 is the number to distrust.
@@ -162,9 +170,10 @@ so the same requirement applies there and is unlinted.
 `tasks/run_limits/` holds one fixture per limit: `max_turns_cap.yaml` asks for more
 sequential work than its cap allows, and `turn_timeout.yaml` runs a command that
 outlives its watchdog. Run either with `--type claude-code` / `--type codex` /
-`--type antigravity` to check a backend against the contract above.
+`--type antigravity` / `--type opencode` to check a backend against the contract
+above.
 
 ## Related
 
-- [Claude Code](CLAUDE_CODE.md) · [Codex](CODEX.md) · [Antigravity](ANTIGRAVITY.md)
+- [Claude Code](CLAUDE_CODE.md) · [Codex](CODEX.md) · [Antigravity](ANTIGRAVITY.md) · [OpenCode](OPENCODE.md)
 - [Task Definition Guide](../TASK_DEFINITION_GUIDE.md) — the full `run_limits` schema