feat: supabase-js SDK evals (auth flow, relational report, key migration)

[mandarini]

Adds four evals for supabase-js: dedicated coverage for the SDK's two most-used surfaces (auth and the PostgREST query builder), a regression canary for the API key migration, and a second Data API scenario probing an SDK-adoption gap the first one turned up.

• build-auth-001-email-password-flow (benchmark): signup/sign-in/profile flow against stubbed app code. Gates on agents reaching for @supabase/supabase-js unprompted (the build-functions-005 pattern); the profile display name only survives if sent as signup user metadata.
• build-dataapi-001-relational-report (benchmark): backend reporting script over a seeded relational schema (embedded relations, aggregation). Gates on using the SDK query builder via the Data API rather than psql or a raw Postgres driver.
• build-dataapi-002-restock-alert-report (benchmark): same shape as build-dataapi-001 — bare prompt, empty package.json, no SDK named — but a different schema/domain (inventory restock alerts vs. a sales report). Added after refreshed results on build-dataapi-001 showed every claude-code variant (4/4) reaching for @supabase/supabase-js unprompted while every codex variant (0/4) hand-rolled raw HTTP instead, despite otherwise-correct, RLS-safe implementations. One scenario isn't enough to tell a real model tendency from an artifact of that specific prompt, so this companion checks whether the split generalizes.
• resolve-sdk-001-legacy-key-migration (regression): seeds a working app on the legacy demo JWTs; agents must migrate it to sb_publishable_/sb_secret_ keys without breaking behavior or leaking the secret key into client code.

All four were sanity-checked locally against a real stack on the pinned CLI (2.109.1): migrations apply, reference solutions pass every scorer check, and the checks discriminate (missing signup metadata, publishable-key impersonation, leftover legacy keys, and SDK vs. raw-HTTP implementations all fail as intended). A scoped .gitignore exception ships the key-migration eval's seeded .env, which holds only the well-known local demo keys.

Fix along the way: build-dataapi-001-relational-report and resolve-sdk-001-legacy-key-migration initially failed 100% of agents identically — a harness bug, not an agent gap. Both had omitted gotrue from their services: frontmatter, so the local stack's auth service never started and supabase status couldn't emit PUBLISHABLE_KEY/SECRET_KEY. Fixed by adding gotrue to services: (matching resolve-dataapi-002-secure-default-grants's pattern); results refreshed and now produce real signal.

build-dataapi-002-restock-alert-report still needs a live run (no API keys in this pass) to confirm whether the adoption split holds.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview, Comment	Jul 28, 2026 3:44pm