feat(eval): add Gemini CLI agent harness - #57
Draft
Rodriguespn wants to merge 6 commits into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Rodriguespn
force-pushed
the
pedrorodrigues/add-gemini-cli-agent-support
branch
from
July 2, 2026 00:17
e2c8a92 to
e0f4b2e
Compare
AI-862 Add Gemini CLI agent support to the eval framework
Add a The Gemini CLI runs headless via Shipped as a runner + parser + registry entry + factory + experiments ( Rate limitsWe will likely need a way around the Gemini models' rate limits. The Google AI Studio / Gemini API tiers impose aggressive per-minute and per-day request/token limits, and the refresh matrix (many evals × attempts, run concurrently) will hit them — surfacing as 429s that look like agent failures rather than genuine task failures. Options to consider:
|
Rodriguespn
force-pushed
the
pedrorodrigues/add-gemini-cli-agent-support
branch
from
July 2, 2026 00:28
e0f4b2e to
1b6b730
Compare
Rodriguespn
marked this pull request as ready for review
July 2, 2026 00:58
Rebuild the Gemini CLI agent on the current agents/<name>/ architecture
(runner + parser + index), mirroring the Claude Code and Codex harnesses:
- packages/core/src/agents/gemini-cli/{runner,parser,index,parser.test}.ts
- Register geminiCliDefinition and route gemini-cli -> "google" provider
- Add "gemini-cli" / "google" to the agent + provider enums, export
geminiCliAgent, and render both in the web UI (label + Gemini model formatter)
- Prompt is fed on stdin (like Codex), not a positional arg — avoids the
ARG_MAX / shell-expansion surface
- Skill-activation tracking wired via the shared extractor, same as the others
- experiments/gemini-cli-3.5-flash.ts + gemini-cli-3.1-pro.ts (suite: benchmark;
auto-discovered by CI), GEMINI_API_KEY wired into eval-refresh + .env.example
- Add @ai-sdk/google (catalog) for the model-id type
Verified: core tests + full typecheck pass; gemini-cli-3.5-flash passes
investigate-db-001-table-row-counts (3/3 checks) end-to-end in the sandbox.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Drop the gemini-cli-3.5-flash experiment; keep only gemini-3.1-pro-preview (Google's comparable to GPT-5.5 / Opus 4.8) and make it the runner default. - Record reasoningEffort: "high" for it. gemini-cli has no flag to set effort — gemini-3.1-pro-preview thinks at high by default (inherited from chat-base-3) — so geminiCliAgent's reasoningEffort is display-only metadata (the runner ignores it) for benchmark parity with the other high-effort agents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rodriguespn
force-pushed
the
pedrorodrigues/add-gemini-cli-agent-support
branch
from
July 2, 2026 21:36
bb4372e to
221e191
Compare
Validated gemini-cli 0.46.0's --output-format stream-json against the parser: the event schema (init/message/tool_use/tool_result/result, field names, delta merge, result.status) is unchanged from 0.20, so no parser/schema changes. Two 0.46-specific fixes: - Runner sets GEMINI_CLI_TRUST_WORKSPACE=true. Since 0.46 gemini refuses to run headless in an "untrusted" folder (exit 55) and silently downgrades --approval-mode yolo to prompt-for-approval — either of which breaks every sandbox run. The env var is ignored by versions that lack it. - Map the new 0.46 built-in `update_topic` (session housekeeping) to agent_task instead of falling through to "unknown". Verified end-to-end: gemini-cli-3.1-pro (CLI 0.46.0) passes investigate-db-001 (3/3), tool calls paired, stop reason + effort metadata correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`max` is not a Gemini effort level; constrain the factory option to the levels Gemini actually reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rodriguespn
marked this pull request as draft
July 29, 2026 10:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
gemini-cli(Google's Gemini CLI) as an eval agent alongsideclaude-codeandcodex, built on the same per-agent framework (packages/core/src/agents/<name>/): a runner + a parser + a registry entry + a factory + an experiment. Orchestration is unchanged and the transcript is parsed back into the same surface scorers already use. Runs in both modes (tools + local-stack), like Claude Code and Codex.gemini-cli runs headless via
gemini --approval-mode yolo --output-format stream-json(CLI pinned to 0.46.0) in a Docker sandbox, with the prompt fed on stdin.GEMINI_CLI_TRUST_WORKSPACE=true. Since 0.46 gemini refuses to run headless in an "untrusted" folder (and silently downgrades--approval-mode yoloto prompt-for-approval), which would fail/hang every sandbox run. The env var is ignored by versions that lack it.$HOME/.gemini/settings.json(outside the scored workspace).result.status. Streamed assistantdeltachunks are merged into one message so step count / report stay correct.loadedSkill), the same way asclaude-code/codex.gemini-3.1-pro-previewthinks athighby default (inherited from chat-base-3). The factory recordsreasoningEffort: "high"in the experiment metadata for benchmark parity with the other high-effort agents (Opus 4.8), but does not pass it to the CLI.Closes AI-862