feat(eval): add Cursor CLI agent harness - #76
Draft
Rodriguespn wants to merge 1 commit into
Draft
Conversation
AI-889 Add Cursor CLI agent support to the eval framework
Add a Need to ask for a Cursor enterprise account key |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Adds `cursor-cli` (Cursor's `cursor-agent`) as an eval agent alongside
claude-code and codex, on the same per-agent framework (runner + parser +
index + experiment):
- Runner: pinned standalone-binary install (not npm) — fetches the version's
tarball from Cursor's download host and symlinks the binary onto PATH; runs
`--print --output-format stream-json --model <m> --trust --force
--approve-mcps` with the prompt on stdin; MCP written to ~/.cursor/mcp.json.
- Parser: Cursor's stream-json (SDK messages) — tool type is the `*ToolCall`
key, started/completed paired by call_id, {success}/{error} result shape,
terminal result for the stop reason; skill tracking via the shared extractor.
- Model id is a free-form string; provider "cursor".
- Experiment cursor-cli.ts: composer-2.5 (Max mode off ⇒ no reasoningEffort),
suite benchmark. CURSOR_API_KEY wired into eval-refresh CI + .env.example.
- reasoningEffort is agent-scoped: cursor accepts only `max` (Max mode);
claude-code/codex now exclude `max` (it's Cursor-specific).
Verified end-to-end: cursor-cli (composer-2.5, CLI 2026.05.01-eea359f) passes
investigate-db-001 (3/3) — install, MCP, stream parse (29/29 tools paired),
skill loading, and scoring all work.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rodriguespn
force-pushed
the
pedrorodrigues/add-cursor-cli-agent-support
branch
from
July 2, 2026 23:31
a2fd157 to
a9ff80b
Compare
Rodriguespn
marked this pull request as draft
July 29, 2026 10:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
cursor-cli(Cursor'scursor-agent) as an eval agent.Cursor CLI is a standalone binary, not a npm package like claude code/codex. We fetch the pinned version's tarball from Cursor's download host and symlinks it onto PATH. To run the CLI headless:
cursor-agent --print --output-format stream-json --model <m> --trust --force --approve-mcps, prompt fed on stdin. The MCP config is written to~/.cursor/mcp.json.Using
composer-2.5with Max mode off. Cursor models don't have the samereasoningEffortparam as claude code/codex have (low,medium,high). Instead, it has a max mode that can be toggled to imporve the agent reasoning capabilities. We keep thereasoningEffortconfig in createCursorCLIAgent for consistency and it acceptsnoneormax. Default tonone.Note
A
CURSOR_API_KEYsecret must be set to run the benchmark and get a preview.Closes AI-889