feat: add Vercel Sandbox eval runner (AI-912 spike) - #114
Draft
Rodriguespn wants to merge 9 commits into
Draft
Conversation
AI-912 Spike: Vercel sandbox runner for evals
Motivation: https://supabase.slack.com/archives/C0BAGJBL49E/p1783455828540249 Consider replacing our GitHub Actions based refresh workflow, with one that dispatches eval runs to Vercel Sandbox (perhaps with a durable queue?). References: |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Rodriguespn
marked this pull request as draft
July 23, 2026 19:03
Rodriguespn
added
run-evals-sandbox-changed
Dispatch (experiment x eval) pairs to Vercel Sandbox microVMs instead of GitHub Actions matrix jobs. Each pair gets its own Firecracker VM that plays the runner's role: clone at the pushed commit, dnf-install Docker and start dockerd, pnpm install, run the same `pnpm eval` command line as eval-refresh.yml, then pull results/ back (tar + downloadFile) and run the publish-results export step scoped to the dispatched pairs. packages/sandbox is untouched: the agent container and the Supabase CLI's sibling containers run against the VM's own dockerd exactly as they do against a GitHub runner's daemon. Vercel credentials go to Sandbox.create explicitly — the SDK reads no VERCEL_* env vars, and CI has no cached dev credentials. Validated E2E: two pairs in parallel; build-cli-001-bootstrap-app (CLI/ local-stack) passed 7/7 inside the VM; investigate-db-001 scored identically to local controls on haiku (FAIL 0/3) and sonnet (PASS 3/3). Findings recorded in packages/vercel-runner/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single dispatcher job (pnpm eval:vercel) with the same inputs, gating, and publish steps as eval-refresh.yml, which stays untouched. Triggered by manual dispatch or the new run-evals-sandbox / run-evals-sandbox-changed PR labels; own concurrency group and results-PR branch. Validated in CI: dispatch runs on haiku and sonnet (job body byte-identical, run under the eval-refresh.yml name pre-move) and a live label-triggered run on PR #114. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
force-pushed
the
feat/replce-docker-for-vercel-sandboxes
branch
from
July 23, 2026 19:31
4eefc03 to
dba1398
Compare
Rodriguespn
added
the
run-evals-sandbox
Experiment discovery (pnpm eval -- list) shells into framework scripts whose --env-file hard-requires a repo-root .env — the full-matrix label path crashed in CI without it. Write the same .env the old run-evals job wrote. Also: concurrency becomes a dispatch input (default 16 — the matrix fanned as wide as the runner pool), sandbox creation retries on a short backoff (rate limits at full-matrix scale), and raw results upload with !cancelled() so passing pairs survive a failed sibling, like per-job matrix artifacts did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
removed
the
run-evals-sandbox
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
added
the
run-evals-sandbox
A once-a-minute scoreboard line (done/pass/fail/error, per-phase in-flight counts, longest-running pair) keeps wide CI fan-outs legible; the summary now shows each pair's scored eval verdict (job ok != eval pass); sandboxes carry run/experiment/eval tags so the Vercel dashboard filters a CI run's fleet live. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
removed
the
run-evals-sandbox
… fan-out The full-matrix run (166 pairs) failed on a single bootstrap step dying with 'Stream ended before command finished' — only the eval step had been moved off live log streams. All steps now run detached with polled status/log files. Pairs that end in a job error (never scored FAILs) get one fresh-sandbox retry: at matrix scale even a sub-percent transient rate would fail most runs. Fan-out defaults to 64 with sandbox creations spaced ~1.3s apart, ramping under Pro's 200 vCPUs/min allocation rate instead of bouncing off it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build a Vercel Sandbox snapshot once per input key (lockfile, sandbox Dockerfile, pinned CLI versions; ~3 min): dnf docker, pnpm+node_modules, the agent sandbox base image, and the full Supabase stack pulled via a throwaway supabase start. Pairs boot from it, restart dockerd, fetch the target revision, and run — measured 3.6 min total for a CLI pair that took 7.4 cold (the eval itself halves too: supabase start pulls nothing). Lookup is by builder-sandbox name via Snapshot.list; any failure falls back to cold git-source boots (--no-snapshot forces it). Two traps found live: sandboxes auto-snapshot on stop by default (persistent: false on throwaway VMs — 435 GB had accumulated), and a snapshot inherits the builder's /tmp, so step files carry a per-session nonce lest a poller read a stale exit file and declare an unrun step done. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
added
run-evals-sandbox
A pair's VM is never reused: stop + delete so nothing lingers (the builder is deliberately kept — its snapshot is the warm-boot cache and is resolved by its name; verified deleting a from-snapshot sandbox leaves the source snapshot intact). Sandboxes are also named <experiment>--<eval>--<rand4> — tags only show on the detail page, so the list view was unreadable random names; the suffix keeps retries and overlapping runs collision-free (names are project-unique). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rodriguespn
marked this pull request as ready for review
July 24, 2026 00:29
Rodriguespn
marked this pull request as draft
July 29, 2026 10:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Check this thread before doing any change to this PR. We might want to explore Vercel workflows to couple with sandboxes
Dispatch eval runs to Vercel Sandbox instead of GitHub Actions matrix jobs.
Two pieces:
packages/vercel-runnerreproduces theeval-refresh.ymlpipeline end to end: matrix discovery, one isolated machine per(experiment x eval)pair running in parallel, the samepnpm evalcommand line, result collection, and thepublish-resultsexport step.github/workflows/eval-refresh-vercel.ymlis a manually-dispatched twin ofeval-refresh.ymlthat runs the dispatcher from a single job. Same inputs, gating, and publish steps; Can be triggered by adding therun-evals-sandboxlabel to this PR.Validation
Local vs sandbox vs CI, same commit:
Limits (Pro plan)
The binding limit is the 200 vCPUs/min creation rate (Sandbox pricing & limits); self-throttled to ~46 × 4-vCPU sandboxes/min, a full 166-pair fan-out ramps in ~4 min; the 2,000-concurrent cap is never close. Cold start is solved by the warm-boot snapshot (docker + node_modules + Supabase stack images pre-baked; rebuilt in ~3 min when the lockfile/CLI key changes; 4.2 GB ≈ $0.33/mo): pairs reach their eval in under a minute and CLI evals run ~2× faster. Pair sandboxes are deleted after use and created with
persistent: false— the default otherwise banks a multi-GB auto-snapshot per stopped VM.How to run
# .env: VERCEL_TOKEN, VERCEL_TEAM_ID, VERCEL_PROJECT_ID, GITHUB_TOKEN, ANTHROPIC_API_KEY pnpm eval:vercel -- --experiment claude-haiku-4.5 \ --eval investigate-db-001-table-row-counts,build-cli-001-bootstrap-app --runs 1Repo secrets
VERCEL_TOKEN/VERCEL_TEAM_ID/VERCEL_PROJECT_IDare already set for the workflow.Ref AI-912