feat: add docs homepage quickstart eval

[czenko]

Closes DOCS-1199

Problem

The docs homepage shows a "Help me get set up with Supabase" prompt. That flow installs the Supabase CLI and Supabase plugin, detects or initializes the project, and suggests next steps, but no eval covers it. CI wouldn't catch a regression in the CLI or plugin install steps, and the sandbox pre-satisfying those steps would mask a failure even in a new eval.

Solution

• Add evals/build-cli-004-quickstart-nextjs, seeded with a bare Next.js app, to exercise the docs homepage quickstart prompt end-to-end.
• Add per-eval skills: [] and skipCliInstall: true overrides so the eval tests the CLI and plugin install steps directly, instead of relying on the sandbox to pre-satisfy them.
• Add an experiment-level skipEval predicate so the no-skills experiment skips evals whose skills list is already empty, avoiding a duplicate run.
• Fix the eval-refresh workflow's pair discovery to respect skipEval. Without this fix, publish-results tried to download an artifact for a pair that skipEval prevents from ever being produced. This broke CI on the first run of this PR.

Manual testing

• Run pnpm eval:dry -- --eval build-cli-004-quickstart-nextjs --experiment claude-code-sonnet-5,claude-code-sonnet-5-no-skills. Only claude-code-sonnet-5 should be planned. The no-skills experiment should print SKIP ... (skipEval).
• Check evals/build-cli-004-quickstart-nextjs/EVAL.ts for the checks that confirm the CLI and plugin were installed, rather than already present. That's the part most likely to break if the sandbox's base image changes.
• Add the run-evals-changed label to confirm publish-results completes without the artifact-download failure.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
evals	Ready	Preview, Comment	Jul 28, 2026 7:36pm

[nrichers]

@czenko as mentioned on Slack, I think you just broke the ice on getting docs into Evals! 👏

I tested your PR locally and for the intent of adding a happy path smoke test, LGTM. 👍 I will defer to the Evals team for approval of the actual code changes as these are outside of my wheelhouse.

This is a great first step towards our a AEO testing aspirations. Way cool and thank you so much! 🚀

---

Verdict: Approve with a note

Check	Result
pnpm --filter @supabase-evals/framework typecheck	✅ pass
pnpm --filter @supabase-evals/core test	✅ 47/47 pass
pnpm --filter @supabase-evals/sandbox test	✅ 35/35 pass
SANDBOX_DOCKER_TESTS=1 ... test:docker -t skipCliInstall	✅ pass — CLI genuinely absent, shim doesn't crash setup
pnpm eval:dry --eval build-cli-004-quickstart-nextjs --experiment claude-code-sonnet-5,claude-code-sonnet-5-no-skills	✅ matches PR claim exactly: SKIP claude-code-sonnet-5-no-skills ... (skipEval), PLAN claude-code-sonnet-5 ...
CI	Only Vercel preview checks run on this repo; no required checks configured

Verified:

• skills/skipCliInstall frontmatter plumbing is threaded correctly end-to-end: eval-metadata.ts → run-eval.ts (ev.metadata.skills ?? exp.skills) → agent-environment.ts → local-stack-runtime.ts/supabase.ts (skip both CLI install and the CLI wrapper when skipCliInstall).
• skipEval on claude-code-sonnet-5-no-skills and the new list --eval filtering in run-eval.ts/eval-refresh.yml correctly avoid planning a redundant duplicate run — confirmed live via eval:dry.
• New eval's scorer (EVAL.ts) checks are sound: supabase/config.toml existence, supabase --version runnable, an LLM-judge rubric for "relevant next steps," and a regex check for the plugin-install command actually being attempted (explicitly not asserting install success, with a comment explaining why — reasonable scoping).

Notes (non-blocking):

• The checked-in apps/web/src/data/regression-eval-results.json currently contains a stale entry: claude-code-sonnet-5-no-skills × build-cli-004-quickstart-nextjs shows passed: true, but per this PR's own skipEval logic, that experiment should no longer run against this eval at all. Tracing the two chore: refresh eval results bot commits: the first (ab5421a, 2026-07-22 23:22 UTC) ran before the skipEval fix (cc8b18a, 2026-07-23 19:18 UTC) and recorded that result; the second (f93b41a, 19:24 UTC) ran after the fix but used the PR-triggered run-evals-changed path, which calls export-results --merge. export-results.ts's merge logic (apps/framework/scripts/export-results.ts:274-289) only overwrites experiment::eval keys present in the new results — it never prunes keys that stop being produced. Since the skipped combo now produces no artifact at all, the stale pre-fix "passed" result will persist through every future merge-mode refresh, only self-correcting on the next full (non-merge) scheduled regression run. This is a pre-existing gap in export-results.ts's merge semantics, not something this PR broke, but this PR is the first to exercise skipEval and is shipping a visibly-stale row into main as a result. Worth a follow-up (either prune-on-merge in export-results.ts, or manually drop that row before merging).
• claude-code-sonnet-5's own real (non-dry) regression result for the new eval shows passed: false (supabase CLI is installed and runnable failed — CLI not found after the agent's npm install -g supabase), while everything else passed (init, plugin-install attempt, next-steps judge). Worth confirming with the author whether this is expected/tracked behavior (a real capability gap the eval is meant to surface) or a PATH-related environment artifact — not a reason to block the PR either way, since the eval mechanically does exactly what it's designed to do.

[linear-code]

DOCS-1199 Add one happy-path quickstart Eval test

Ship one narrow happy-path quickstart test for a popular framework using the shared docs E2E harness.

The goal for this milestone is one reliable smoke path by the deadline rather than broader cross-framework coverage. Include enough structure or documentation so additional framework tests can be added later without reshaping the setup.

Text to Evaluate

From Supabase.com/docs

Help me get set up with Supabase. Do the following: 1. Install the Supabase CLI globally with `npm install -g supabase`. 2. Install the Supabase Plugin with `npx plugins add supabase-community/supabase-plugin`. 3. Review my project and determine whether Supabase is already initialized. If it is not initialized, run `supabase init`. 4. Suggest the most relevant next steps.

Slack thread context

Miranda asked whether evaluating the AI prompt on the docs homepage should live in the evals repo and how to get started. The scenario is described as validating that the prompt can successfully complete the action, starting with the recently added homepage prompt.

Relevant guidance from matt.rossman:

• This looks like a good stress test for the evals repo and is similar to AI-884, which covers a from-scratch Supabase setup scenario.
• The awkward part is that experiments currently define skills and CLI installation at the experiment level, which makes it hard to represent an eval that specifically tests whether the agent installs them itself.
• Proposed repo changes to support this cleanly:
  • per-eval skills override, such as skills: []
  • per-eval CLI-skip override, such as skipCliInstall: true
  • experiment-level skipEval predicate so *-no-skills experiments can skip evals with an empty skills override list
• For an initial implementation, copy the homepage prompt into a PROMPT.md for the eval.
• Review the evals contributing guide for how to add, submit, and run evals in CI.

Original question from the thread:

"We at docs want to evaluate AI prompts in our docs to see if they successfully complete the action, starting with the one recently added right on the home page. Is this something I contribute to the evals repo? How do you recommend I get started?"

See the Slack thread for the full discussion.

[czenko]

@nrichers Thanks for flagging the merge-mode gap. Fixes in claude/nervous-allen-83c06e.

export-results.ts now accepts a --requested experiment::eval,... flag; merge mode drops an existing row when its key was requested this run but produced no result file , while leaving rows untouched for pairs that simply weren't part of this run.

[nrichers]

LGTM, thank you for addressing my comment. 🚀

[mattrossman]

Hey @czenko I tested running this new eval locally. During that local run, the agent failed to install supabase CLI with npm because the Node image installs node as root and doesn't expose sudo access, so it gets stuck on the npm install -g supabase step:

$ npm install -g supabase 2>&1

Exit code 243
npm error code EACCES
npm error syscall mkdir
npm error path /usr/local/lib/node_modules/supabase
npm error errno -13
npm error Error: EACCES: permission denied, mkdir '/usr/local/lib/node_modules/supabase'
...
npm error It is likely you do not have the permissions to access this file as the current user
npm error
npm error If you believe this might be a permissions issue, please double-check the
npm error permissions of the file and its containing directories, or try running
npm error the command again as root/Administrator.

I believe the CI run failed for the same reason:

https://github.com/supabase/evals/pull/108/changes#diff-3b6aa05c44031b2fe9a909f86aa07cf051bd13e1ee6fb18c71d159b5f7383ce6R29-R31

I figure a typical user's machine will have Node configured in a way that npm install -g "just works" for a user without needing sudo, maybe we should update the Dockerfile to configure it this way to make the scenario more fair?

I found this Global npm dependencies section in the docs for the official Node.js docker images, which recommends the following:

ENV NPM_CONFIG_PREFIX=/home/node/.npm-global

ENV PATH=$PATH:/home/node/.npm-global/bin # optionally if you want to run npm global bin without specifying path

The ENV PATH line may need to be adapted for our use case, because we override PATH with an explicit SANDBOX_PATH list here, so we can instead add it like:

--- a/packages/sandbox/src/docker-sandbox.ts
+++ b/packages/sandbox/src/docker-sandbox.ts
@@
 const SANDBOX_PATH = [
+  "/home/node/.npm-global/bin",
   "/usr/local/sbin",
   "/usr/local/bin",
   "/usr/sbin",
   "/usr/bin",
   "/sbin",
   "/bin",
 ].join(":");

I think we'll also need to make sure the /home/node/.npm-global folder isn't created as root during image build, otherwise subsequent npm install -g may not work. Note the npm install -g skills line implicitly initializes this folder as whichever user is active (root` unless otherwise specified in the Dockerfile), so we could explicitly create the folder + set up the npm config prefix beforehand.

--- a/packages/sandbox/Dockerfile
+++ b/packages/sandbox/Dockerfile
@@
 FROM node:22-slim

 RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
     ca-certificates curl git docker.io postgresql-client \
   && rm -rf /var/lib/apt/lists/*

+USER node
+RUN mkdir -p /home/node/.npm-global
+ENV NPM_CONFIG_PREFIX=/home/node/.npm-global
+
 # Vercel's agent-skills CLI, used to install agent skills from local sources
 # into each session's workspace (both modes). Baked in (pinned) so installs are
 # offline and instant rather than an npx download per session.
 ARG SKILLS_CLI_VERSION
 RUN npm install -g "skills@${SKILLS_CLI_VERSION}"

I tested this change locally and verified it allows the new eval to pass 4/4 checks, vs failing one of the checks otherwise.

[mattrossman]

Also may want to add a test to lock in the behavior:

it(
  "npm install -g lands on PATH without sudo/workarounds",
  { timeout: TEST_TIMEOUT_MS },
  async () => {
    const image = await ensureSupabaseSandboxImage();
    const sandbox = await DockerSandbox.create({ image, network: "host" });
    try {
      const install = await sandbox.runShell("npm install -g supabase 2>&1");
      expect(install.ok).toBe(true);

      const version = await sandbox.runShell("supabase --version");
      expect(version.ok).toBe(true);
    } finally {
      await sandbox.stop();
    }
  },
);

[czenko]

@mattrossman Thanks for digging into this. Thoughtful, thorough feedback.

Applied basically as you proposed:

• packages/sandbox/Dockerfile: switch to USER node, set NPM_CONFIG_PREFIX=/home/node/.npm-global before installing the skills CLI, so it and the agent's own npm install -g supabase both land in a node-owned prefix.
• Added /home/node/.npm-global/bin to SANDBOX_PATH.
• Also added your suggested regression test
• Cleaned up commit history

[mattrossman]

I experienced a local run that failed because the agent tried to run:

npx --yes plugins add supabase-community/supabase-plugin

The regex didn't match because of the --yes, perhaps it should be more forgiving?

[czenko]

Thanks. Will look into this on Monday.

[czenko]

Pushing a fix for the regex, @mattrossman.
As another note, if there is more feedback that seems to be low-hanging, feel free to commit it directly so this can be merged. Or maybe we can review it together live? I admit that a lot of this PR is AI-assisted and your context and expertise will be valuable for me.