refactor(framework): run framework tests through Vitest - #135
Open
barryroodt wants to merge 6 commits into
Open
Conversation
AI-934. The 659-line smoke-framework.ts script becomes Vitest files: six
scorer tests next to the EVAL.ts they cover, four harness tests next to the
harness module they cover, shared plumbing in harness/scorer-test-kit.ts.
Adds a CI job so these actually run on PRs, which is the point of the ticket.
Colocation lets each scorer test import its scorer directly, so the old
dynamic loadScorer(path) is gone and the scorer/test contract is typechecked.
Two things the old script hid:
- Every failure was invisible. It stubbed console.error globally to silence
expected-fail scorer noise, then reported failures through console.error, so
a failing run printed nothing and exited 1. Vitest reports failures itself,
so the stub moves to an onConsoleLog filter that cannot swallow a result.
- The frontend build test was failing for real (AI-975). Fixed here because a
CI job that ships red is worse than no CI job: project-runner hardcoded
<repo>/node_modules/{vite,vitest}, which pnpm's isolated layout never
creates, and the frontend toolchain was declared in apps/framework where a
scored workspace under results/ cannot resolve it. Bins now resolve via each
package's exported package.json rather than a deep subpath that exports may
stop publishing, and the workspace-facing toolchain moves to the root
manifest, matching the documented contract that scoring runs repo-root
vite/vitest so the sandbox need not carry it. Nothing in apps/framework
imported those packages; they appeared only in generated workspace config
and fixture strings. build-frontend-001-todos-app has never produced a
result row, which is why nobody noticed.
The suite is hermetic: investigate-security-001's scorer calls a real OpenAI
judge, so that test mocks judge() alone and keeps every other core export
real. Verified green with OPENAI_API_KEY/ANTHROPIC_API_KEY unset.
AI-975 Framework smoke test fails on main with no failure output
By elimination it is Reproduce: cd apps/framework
node --import tsx/esm scripts/smoke-framework.ts; echo "exit=$?"Output ends at There are two separate problems here:
Noticed while working on AI-923. Confirmed pre-existing by stashing that branch and rerunning on clean AI-934 Refactor smoke framework tests into per-scorer Vitest files
The evals repo still has a monolithic |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Review follow-ups on the Vitest migration.
Eval tests were importing `../../apps/framework/harness/{scorer-test-kit,types}.js`
while the EVAL.ts sitting next to them imported `@supabase-evals/core`: two
conventions for the same types, side by side, and six eval directories reaching
across a package boundary into an app's internals. The framework barrels they
reached for are re-exports anyway (platform-backend.ts is two lines).
The kit now lives at top-level `test-utils/`, not in core: it hardcodes the
monorepo layout (ROOT, seedPath) and is test-only, so a production package
should not own it. Canonical types come straight from core.
Also drops the `onConsoleLog` stderr filter. It was carried over reflexively
from the old script's global console.error stub, and never measured: Vitest
attributes console output per test, and the suite runs clean without it.
Suppressing output was the very thing that made the old script's failures
invisible, so it does not get reintroduced without evidence.
The frontend reference solution stops being an 80-line template string and
becomes evals/build-frontend-001-todos-app/reference/App.tsx, genuinely
typechecked via its own tsconfig extending the eval's jsx-capable one, plus
@types/react at the root. It stays outside local/ so the agent never receives
it. Note tsc cannot catch data-testid drift, only API misuse; the withheld
tests remain what proves the testids still line up.
Each colocated scorer test hardcoded `EVAL_DIR = 'evals/<its own directory>'`, restating what the file's path already says. Copying a test into a new eval and missing that constant would seed a different eval's fixtures and score against them, which does not throw and can pass: a wrong-but-valid path, in a suite whose whole point is tests that do not lie. seedPath now takes the caller's module URL, so a colocated test always seeds its own scenario. platform-backend.test.ts genuinely borrows another eval's logs fixture, so that one path stays spelled out to keep the exception visible.
Frontend evals instruct the agent to read import.meta.env.VITE_SUPABASE_URL and _ANON_KEY (PROMPT.md:29-30), but nothing ever supplied them: no .env.local in the workspace, none written by the runner or the sandbox. createClient threw at module import, vitest collected zero tests, and the eval scored 1/2 no matter what the agent produced. The values already existed, but only as text inside the string setupSource() returns, so the generated config could not reference them. Hoists them to real module scope and interpolates them into both generated files, so the setup and the app under test agree on one definition. Also drops the .env.local write from project-runner.test.ts. A real agent workspace has no such file, so writing it made the test green while the live path failed, which is how this defect stayed hidden. Verified end to end: claude-code-sonnet-5 x build-frontend-001-todos-app now passes 2/2 with numTotalTests 2, up from 1/2 with zero tests collected. First passing result this eval has ever produced.
barryroodt
marked this pull request as ready for review
July 29, 2026 08:27
The previous commit supplied VITE_SUPABASE_* to the generated vitest config only. viteBuild invoked vite with no env, so `import.meta.env.VITE_SUPABASE_URL` compiled to undefined and the bundle called createClient(void 0, void 0). The build exited 0 and the eval scored 2/2 while shipping an artifact that throws on load: the scorer was green, the app was not. Both tools now read one PROJECT_ENV object, delivered the way each expects, so a build and its tests cannot disagree about which project the app targets. project-runner.test.ts asserts the built bundle actually contains the project URL rather than just checking vite's exit code. Falsified by reverting the fix: the assertion fails with "built bundle is missing the Supabase project URL". Verified end to end: the real agent bundle from a sandboxed run now embeds supabase-evals.local, where the previous passing run's bundle did not.
The previous commit gave Vite the env via process env and Vitest via an `env:` block interpolated into the generated config, then added a comment explaining why there were two mechanisms. A comment justifying complexity is a good sign the complexity is not needed. Vite exposes VITE_-prefixed variables that already exist in the environment on import.meta.env, and Vitest inherits that behaviour, so passing PROJECT_ENV through runNodeBin covers both. Drops the generated env block and the Object.entries interpolation. Verified with the block removed: the smoke workspace's generated config has no `env:` key and its withheld tests still run 2/2, and a full sandbox run of build-frontend-001-todos-app passes 2/2 with the project URL still embedded in the agent's bundle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refactors
smoke-framework.tsinto Vitest files, plus the frontend scoring fixes needed to get the new CI job green.EVAL.tsthey cover, four harness tests next to their module, shared helpers intest-utils/.VITE_SUPABASE_*were all broken, so the root manifest now owns the frontend toolchain.Verified:
build-frontend-001-todos-apppasses 2/2 on a real sandboxed run, its first ever.Closes AI-934 + AI-975