feat: add slow query CPU spike red-herring eval (AI-841)

[Rodriguespn]

Harder follow-up to the slow query CPU spike eval (AI-824). Same scenario and scored dimensions, but the slow-query logs are a deliberate red herring so the agent can no longer read the answer straight out of them — it has to do real CPU diagnosis via pg_stat_statements.

Why this is different from AI-824 (and why the logs mislead in real life)

Postgres only writes a statement to the server logs when its execution time exceeds log_min_duration_statement. It is disabled by default (-1); this scenario's seed explicitly enables it at 1s (ALTER DATABASE ... SET log_min_duration_statement = '1s'), so the logs capture only statements slower than 1s. That means slow-query logs surface long-running statements and never a cheap, high-frequency query — even when that frequent query is the real CPU driver. In AI-824 the offending query was both the slowest-per-execution and the most-frequent, so the seeded logs named it outright and an agent could fix it without ever consulting pg_stat_statements (gpt-5.4-mini did exactly that). This variant breaks that shortcut.

Additions

• Benchmark eval resolve-performance-002-slow-query-cpu-spike-red-herring. The CPU is driven by a fast-but-very-frequent recent-events lookup on events (~22 ms × ~486k calls) that stays under the 1s log_min_duration_statement threshold and therefore never appears in the logs. The logs are instead dominated by a genuinely slow but rare audit_log report query (~3 s × 14 calls) whose total CPU cost is negligible. The offending query is only identifiable by ranking pg_stat_statements on total_exec_time (calls × mean_exec_time) — sorting by mean/max time, or reading the logs, leads to the wrong table.
• Seed (remote/project.sql) enables slow-query logging at 1s, ships a pg_stat_statements-style table whose stats diverge by ranking dimension, an events table (the hog) and an audit_log table (the decoy), plus a logs.jsonl that contains only the above-threshold slow decoy. The prompt is intentionally generic (no "recent events" hint) so the agent must diagnose rather than pattern-match.

To run the eval

pnpm eval -- --eval resolve-performance-002-slow-query-cpu-spike-red-herring --experiment claude-sonnet-4.6 --force --runs 1

What the eval checks

• The agent inspected pg_stat_statements / query performance data.
• The agent ranked query stats by total (cumulative) execution time (total_exec_time), not per-call latency.
• The agent ran EXPLAIN on the expensive events query.
• The database has an index on events covering (user_id, created_at).
• The high-frequency hog query uses an index in its plan and no longer sequential-scans events — an agent that only "fixed" the slow audit_log decoy fails here.
• Inserts into events still work after the schema change.

Results & variance
This is an LLM-agent benchmark, so per-run outcomes vary. Across the first two run-evals passes, gpt-5.4-mini and gpt-5.4-nano each passed once and fell for the decoy once — e.g. in the latest run mini indexed the audit_log decoy and failed every diagnostic check, while nano diagnosed via pg_stat_statements/total_exec_time and passed. The trap reliably catches capable agents; score with --runs N for a stable pass rate rather than a single sample.

References

• https://supabase.com/docs/guides/troubleshooting/high-cpu-usage
• https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-MIN-DURATION-STATEMENT
• https://supabase.com/docs/guides/database/query-optimization
• https://supabase.com/docs/guides/database/inspect

🤖 Generated with Claude Code

Closes AI-841

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview, Comment	Jun 15, 2026 8:38pm

[linear-code]

AI-841 Benchmark: Optimize slow query CPU spike (red-herring variant)

Harder follow-up to AI-824 (supabase/evals#33).

AI-824's scenario can be solved without the intended diagnostic path: because the offending query is both the slowest-per-execution and the most-frequent, the seeded slow-query logs name it outright, so an agent can read the logs, add the index, and pass every state check without ever consulting pg_stat_statements. (gpt-5.4-mini did exactly this in PR #33's run: it diagnosed from get_logs and skipped pg_stat_statements, so the logs are an unintended bypass of the workflow the eval is meant to test.)

This variant adds a red herring so the logs alone lead to the wrong fix and pg_stat_statements is required for a correct diagnosis:

• Diagnose a CPU spike where the slow-query logs are misleading: a genuinely slow but rare report query (e.g. ~3 s, runs a handful of times) dominates the logs, while the real CPU hog is a fast-but-very-frequent query (e.g. ~25 ms × hundreds of thousands of calls) that never crosses the slow-query log threshold. The agent must find the offending statement in pg_stat_statements by ranking on total_exec_time (calls × mean_exec_time), confirm the sequential scan with EXPLAIN, and add the index that fixes it — not the index suggested by the log decoy.
  • journey: investigate, resolve · interface: mcp · product: database · topic: observability, sql
  • Same dimensions/values as AI-824; differs only in the seed (adds a decoy query + frequency-skewed query stats) so it actually exercises the pg_stat_statements find-then-fix workflow.

Scoring should keep AI-824's deterministic checks (index on the correct table/columns, plan uses an index + no seq scan, inserts still work) and add a check that the agent indexed the high-frequency CPU hog rather than the slow decoy.

Review in Linear