Add codex myagent eval - #1
Open
dev-ankit wants to merge 9 commits into
Open
Conversation
Set up the repo with uv (pyproject.toml, .python-version pinned to 3.10,
uv.lock) covering eval + retrieve + data dependencies.
Add eval/myagent/: a drop-in BEAVER text-to-SQL method whose agent delegates
generation to the local Codex CLI (codex exec). Includes the full pipeline
(execute -> unify) mirroring the existing baselines, plus two gold-blind,
execution-guided enhancements in agent.py:
- self-fix (CODEX_SQL_FIX): run own SQL read-only; repair execution errors
from DB error feedback only.
- explore/verify (CODEX_SQL_EXPLORE): run read-only queries against the real
tables, inspect the rows its own queries return, self-check, then finalize.
Neither ever sees gold; all DB access is mediated read-only by the agent process.
RESULTS.md records the study on dw (100q): one-shot 23% -> setting 2 +
explore/verify + fix 34%, with lever analysis (hints > explore > fix/effort).
Raw per-run outputs and unified-output/ (gold SQL from the gated dataset) are
gitignored; results are captured as aggregate metrics in RESULTS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add eval/claudeagent/: a second drop-in BEAVER text-to-SQL method, backend = Claude Code headless (`claude -p`). Mirrors myagent's pipeline and the same gold-blind, execution-guided modes (self-fix, explore/verify), with CLAUDE_* config incl. --model (CLAUDE_MODEL) and --effort (CLAUDE_EFFORT). RESULTS.md records the head-to-head at the best config (setting 2 + explore/verify + fix, effort high) on dw/100q: Codex gpt-5.5 34% vs Claude Opus 32% (tied within noise, both 0 SQL errors). Key finding: the backends are complementary — only 23 shared correct, union 43 — so cross-model ensembling is the standout next lever. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add to myagent/RESULTS.md: - Ablation (Codex, high + explore + fix): setting 2 with decomp 34% vs without 28%; per-question decomp helps 13 / hurts 7 -> net +6. Decomposition is net-positive; keep it. - Failure analysis of two setting-2 "values mismatch" cases (dw_2933 scope via a "top 10" scaffold; dw_104 rolling-avg vs overall-avg, a Codex-right/ Claude-wrong divergence), with the caveat that the ablation shows these are a minority failure mode, not a net drag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
agent.py: add two optional, gold-blind, off-by-default modes to the Codex agent: - CODEX_DECOMPOSE: prepend guidance to self-decompose the question and validate each sub-step with SQL during the explore loop. - CODEX_REVIEW: a final subagent pass that reviews the answer against the question for intent capture (grounded in the query's own results, not gold). RESULTS.md: record the isolation ablation (Codex, high, explore+fix, setting 2 minus decomp). Both modes are net-negative: self-decompose -5, subagent review -7 vs the 28% baseline (review dominates; both = review-only = 21%). This is the third "more reasoning" hypothesis the controlled runs refuted this study; the oracle decomposition hint (34%) beats the agent's own reasoning. Best config stays hints + explore + fix. README documents the modes with the caveat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GPT-5.6 rerun of the best config matches 5.5 on pass@3 (49%) but ranks candidates worse (pass@1 33% vs 39%). 5.6 HTML-escapes operators inside <ans> spans on some answers (8/100, 7 became 1064 syntax errors); clean_sql now unescapes entities so this can't recur. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.