TypeScript local-dev runner: pnpm local (run/compare/experiments/docs)

.gitignore

@@ -6,3 +6,7 @@ dist/
 results/*/
 .sync-tmp/
 
+
+# local-dev runner (apps/framework/scripts/local.ts)
+/results-local/
+/.local-docs/

AGENTS.md

@@ -0,0 +1,112 @@
+# AGENTS.md — supabase/evals
+
+Instructions for coding agents working in this repo. Humans: start with
+[README.md](README.md) and [CONTRIBUTING.md](CONTRIBUTING.md).
+
+## What this repo is
+
+Evals for Supabase AI agents. An eval run is `agent + inputs -> score`. The
+three inputs a change usually targets: the **skills** tree (in this repo),
+the **MCP server** (external checkout), and **docs** content (external
+supabase/supabase checkout).
+
+## Verifying a change against the evals (`pnpm local`)
+
+Use the local runner for all "did my change help / did it regress?" work.
+It never mutates git state, so it is safe alongside in-flight work.
+
+```bash
+pnpm local run <eval-id...>     [--experiment <id>] [--runs N] [--mcp <path>] [--content-api <url>]
+pnpm local compare <eval-id...> [same flags]   # + diff vs latest published result on origin/main
+pnpm local experiments                         # experiments + which have published baselines
+pnpm local docs <up|seed|api|down> --docs <path-to-supabase-monorepo>
+```
+
+Per input:
+
+- **Skill edited** (in `skills/`): no sync step — `pnpm local compare <eval>`.
+- **MCP server edited** (external checkout): `pnpm build` in that checkout,
+  then `pnpm local compare <eval> --mcp <checkout-path>`.
+- **Docs page edited** (external supabase/supabase checkout):
+  `pnpm local docs seed --yes` to re-embed (**~$0.12 OpenAI — see spend
+  rules**), keep `pnpm local docs api` running in a separate terminal, then
+  `pnpm local compare <eval> --content-api http://127.0.0.1:3001/docs/api/graphql --mcp <mcp-checkout>`.
+  `--content-api` also needs `--mcp`: the env var it sets is only read by an
+  mcp build carrying supabase/mcp#343 (merged, unreleased). With the published
+  package `search_docs` would query production docs while the receipt claimed
+  otherwise, so the runner refuses pre-spend.
+  **`docs seed` currently fails against a vanilla docs checkout**: the pipeline
+  unconditionally loads its lint-warnings source, which needs a GitHub App
+  (`DOCS_GITHUB_APP_*`, no token fallback), and one `Promise.all` makes that
+  fatal. It aborts before embedding, so a retry costs nothing but achieves
+  nothing — don't loop on it. The leg needs an index seeded another way until a
+  skip flag lands upstream.
+  Score docs evals on retrieval (`docs.calls`, canary content coming back out of
+  `search_docs`), not on the answer text: tools mode also exposes
+  `WebSearch`/`WebFetch`, and an edit that contradicts the live page invites the
+  agent to fetch production and reject the local content as injection (observed).
+
+Receipts land in `results-local/` (git-ignored): treatment provenance (host
+SHA + dirty state, override git state) and, for `compare`, the published
+arm's result commit + parent + age.
+
+## Interpreting results — rules, not suggestions
+
+- **`compare` is a screen, not causal proof.** The published arm ran in the
+  scheduled CI world (published MCP package, prod docs index, model state at
+  refresh time). Never report a flip as caused by the edit; report it as a
+  signal consistent with the edit.
+- **Single runs are noisy.** Before claiming improvement or regression, run
+  `--runs 3` and read check-level results, not just pass/fail.
+- **MCP changes: judge by tool-call activation.** An eval can pass without
+  ever calling the tool you changed. Confirm the changed tool was actually
+  exercised (the result JSON records tool calls) before concluding anything.
+- **Docs changes: the eval must be able to see the docs.** Use a tools-mode
+  (`interface: mcp`) eval whose answer lives in the edited page and is
+  reached via `search_docs`. CLI-scaffold evals can pass regardless of docs.
+- **No published baseline?** Use `pnpm local run` (custom evals included).
+  For a before/after, run once before the edit and once after.
+
+## Validating a dependency PR (e.g. supabase/mcp)
+
+1. **Baseline-proof first**: build the dependency's MAIN and run the chosen
+   eval(s) against it before the PR build — version pins hide fixture drift
+   (platform-lite tracks the pinned `MCP_SERVER_VERSION`, not your local
+   build's line; the runner warns on version mismatch).
+2. Fixture or eval support living in an unmerged evals PR? Apply it into the
+   worktree as plain working-tree state: `gh pr diff <n> | git apply`.
+   Receipts record the dirty tree, so runs stay attributable.
+3. Run the PR build with `--mcp <checkout>`; a main-FAIL -> PR-PASS flip with
+   everything else constant is a true two-arm comparison on the dependency
+   axis (stronger than the published screen).
+4. **Judge by tool-call activation, not pass/fail**: confirm the changed tool
+   was called, and unwrap `<untrusted-data-…>` envelopes in `toolCalls[]`
+   before reading results — errors hide inside them. Note that claude-code
+   records endpoints with an `mcp__<server>__` prefix; match with
+   `.endsWith('<tool>')`.
+
+## Spend rules
+
+- Eval runs cost model tokens; `pnpm local docs seed` costs **~$0.12 OpenAI
+  per invocation**. State the cost and get user confirmation before
+  running paid steps the user did not explicitly request.
+- The runner refuses pre-spend on invalid eval metadata, unknown
+  experiments, and bad `--mcp` paths — do not work around these gates.
+- Zero-cost checks: `pnpm --filter @supabase-evals/framework test:local`
+  (runner self-test), `pnpm local experiments`, `pnpm eval:dry`.
+
+## Conventions
+
+- Keys live in `.env` at the repo root: `ANTHROPIC_API_KEY`, plus
+  `OPENAI_API_KEY` for the docs loop AND for any eval whose scorer uses the
+  LLM judge (an OpenAI grader model runs even when the agent under test is
+  Claude). Never hardcode or echo key values.
+- Model/agent selection = experiment id. To test an unlisted model, add a
+  small `experiments/<name>.ts` (copy an existing file's shape) rather than
+  editing a published experiment in place.
+- `results/`, `results-local/`, and `.local-docs/` are outputs — never
+  commit their contents.
+- Verify with `pnpm check` (typecheck + core/sandbox tests) and
+  `pnpm format:check` (biome) before pushing.
+- New evals: follow [CONTRIBUTING.md](CONTRIBUTING.md) (suite choice,
+  `motivation:` frontmatter, scorer shape).

CLAUDE.md

@@ -0,0 +1 @@
+See [AGENTS.md](AGENTS.md) for agent instructions in this repo.

README.md

@@ -78,6 +78,65 @@ Start the web app development server:
 pnpm web
 ```
 
+## Local development loop (`pnpm local`)
+
+Testing a change to an agent input — a skill, a local build of
+[`mcp-server-supabase`](https://github.com/supabase/mcp), or an edited docs
+page — against the evals, without touching git state:
+
+```bash
+pnpm local run <eval-id> [--experiment <id>] [--mcp <path>] [--content-api <url>]
+pnpm local compare <eval-id> [same flags]   # + diff vs the latest published result on main
+pnpm local experiments                      # list experiments + published-baseline availability
+```
+
+- **Skills**: edit the skills tree in this repo and just `run` — the harness
+  reads it as-is.
+- **MCP**: clone + build the mcp repo anywhere, then `--mcp <path-to-checkout>`
+  (sets `SUPABASE_MCP_SERVER_PATH`, so `search_docs` and friends run your build).
+- **Docs**: serve a local docs content API from your own supabase/supabase
+  checkout, then point runs at it:
+
+  ```bash
+  pnpm local docs up --docs <path-to-supabase-monorepo>
+  pnpm local docs seed        # full embed via the docs app's pipeline (~$0.12 OpenAI; asks first)
+  pnpm local docs api         # keep running in a separate terminal
+  # --content-api needs a local mcp build too: SUPABASE_CONTENT_API_URL support
+  # is merged (supabase/mcp#343) but unreleased, so the published server ignores
+  # it and search_docs would silently hit production docs. Refused pre-spend.
+  pnpm local run <eval-id> --content-api http://127.0.0.1:3001/docs/api/graphql --mcp <mcp-checkout>
+  ```
+
+  **Known limitation — `docs seed` needs a docs checkout containing
+  [supabase/supabase#48364](https://github.com/supabase/supabase/pull/48364).**
+  Without it, `fetchAllSources()` unconditionally awaits the lint warnings source,
+  whose loader requires the docs GitHub App, and one shared `Promise.all` turns
+  that into a full abort before any embedding (so it costs nothing). That PR adds
+  a token rung below the App, `GH_TOKEN` then `GITHUB_TOKEN`, which is all a
+  contributor needs: `export GH_TOKEN=$(gh auth token)`. Until it merges, check
+  that branch out in the checkout you pass to `--docs`.
+
+  Verified end to end against a checkout carrying it, with the `NEXT_PUBLIC_MISC_*`
+  wiring `docs seed` supplies: the seed completes (1901 sources, 7890 sections) and
+  a tools-mode eval's `search_docs` returns content that exists only in the local
+  index. Two rough edges to expect, both upstream: the seed exits 0 while silently
+  failing 22 `/reference/{javascript,dart}` pages whose sections exceed the
+  embedding model's 8192-token limit, and a local index has no partner-integration
+  pages, since that source reads the hosted misc project. Neither blocked the
+  tested guide-page eval, but an eval whose answer lives in those reference pages
+  would find them missing from the index.
+
+Every run writes a provenance receipt to `results-local/` (host SHA + dirty
+state, override paths and their git state). `compare` records the published
+arm's result commit, parent, and age — and a pass/fail flip against published
+is a **screen**, not causal proof: the published run happened in the scheduled
+CI world (published mcp package, prod docs index, model state at refresh time).
+
+Keys go in `.env` at the repo root: `ANTHROPIC_API_KEY`, and `OPENAI_API_KEY`
+for the docs loop and for judge-scored evals (the LLM judge is an OpenAI
+grader model, regardless of the agent under test). Zero-cost self-test: `pnpm --filter
+@supabase-evals/framework test:local`.
+
 ## Eval Shape
 
 Every eval contains:

apps/framework/harness/run-eval.ts

@@ -32,6 +32,7 @@ import {
   buildSkillResult,
   rehydrateTruncatedDocsResults,
   getExperimentDisplayMetadata,
+  supabaseMcpServerMounts,
 } from '@supabase-evals/core';
 import type {
   ExperimentConfig,
@@ -439,6 +440,7 @@ async function runOne(
           // (the session folds the discovery listing into its promptAddendum),
           // so no skill text is injected into the prompt here.
           skills: skillSources,
+          mounts: supabaseMcpServerMounts(),
         })
       );
 
@@ -507,7 +509,12 @@ async function runOne(
     // platform-lite via host.docker.internal (so platform-lite binds 0.0.0.0).
     // An in-process agent runs host-side with no sandbox.
     await using cliSandbox = agentRunsInSandbox
-      ? disposable(await createBareSandbox({ skills: skillSources }))
+      ? disposable(
+          await createBareSandbox({
+            skills: skillSources,
+            mounts: supabaseMcpServerMounts(),
+          })
+        )
       : undefined;
     await using session = disposable(
       await exp.runtime.startSession({

apps/framework/package.json

@@ -12,7 +12,9 @@
     "test:framework": "node --env-file-if-exists=../../.env --import tsx/esm scripts/smoke-framework.ts",
     "export-results": "node --import tsx/esm scripts/export-results.ts",
     "demo:mcp": "node --env-file=../../.env --import tsx/esm scripts/mcp-demo.ts",
-    "demo:executor": "node --env-file=../../.env --import tsx/esm scripts/executor-demo.ts"
+    "demo:executor": "node --env-file=../../.env --import tsx/esm scripts/executor-demo.ts",
+    "local": "node --env-file-if-exists=../../.env --import tsx/esm scripts/local.ts",
+    "test:local": "node --env-file-if-exists=../../.env --import tsx/esm scripts/smoke-local.ts"
   },
   "dependencies": {
     "@ai-sdk/anthropic": "catalog:",

apps/framework/scripts/docs/content-api-server.ts

@@ -0,0 +1,71 @@
+// fallow-ignore-file unused-file -- loaded at runtime (spawned/injected by local-docs.ts), never statically imported
+/**
+ * Standalone docs content GraphQL API for `search_docs`.
+ *
+ * Serves the docs app's own route handler (apps/docs/app/api/graphql/route.ts
+ * in a supabase/supabase checkout) over plain node:http — no Next server.
+ * Launched by `pnpm local docs api` with the docs checkout's tsx so the
+ * route's TS + tsconfig conditions resolve; DOCS_ROUTE_PATH points at the
+ * checkout, PORT picks the listen port.
+ */
+import { createServer } from 'node:http';
+import { pathToFileURL } from 'node:url';
+
+const routePath = process.env.DOCS_ROUTE_PATH;
+if (!routePath) {
+  console.error(
+    'DOCS_ROUTE_PATH not set — run this through `pnpm local docs api`'
+  );
+  process.exit(1);
+}
+// The docs checkout location is user-supplied at runtime; a static import
+// cannot name it.
+const route = await import(pathToFileURL(routePath).href);
+const handlers: Record<string, (req: Request) => Promise<Response>> = {
+  GET: route.GET,
+  OPTIONS: route.OPTIONS,
+  POST: route.POST,
+};
+const port = Number(process.env.PORT ?? 3001);
+
+createServer(async (incoming, outgoing) => {
+  const url = new URL(
+    incoming.url ?? '/',
+    `http://${incoming.headers.host ?? `127.0.0.1:${port}`}`
+  );
+  const handler = handlers[incoming.method ?? ''];
+  if (url.pathname !== '/docs/api/graphql' || !handler) {
+    outgoing.writeHead(404).end();
+    return;
+  }
+
+  const headers = new Headers();
+  for (const [name, value] of Object.entries(incoming.headers)) {
+    if (Array.isArray(value))
+      for (const item of value) headers.append(name, item);
+    else if (value !== undefined) headers.set(name, value);
+  }
+
+  const chunks: Buffer[] = [];
+  for await (const chunk of incoming) chunks.push(Buffer.from(chunk));
+  const body =
+    incoming.method === 'GET' || incoming.method === 'HEAD'
+      ? undefined
+      : Buffer.concat(chunks).toString('utf8');
+  const response = await handler(
+    new Request(url, { method: incoming.method, headers, body })
+  );
+
+  outgoing.writeHead(
+    response.status,
+    Object.fromEntries(response.headers.entries())
+  );
+  outgoing.end(Buffer.from(await response.arrayBuffer()));
+  // Bind every interface, advertise loopback — same rule as platform-lite in
+  // tools mode (see run-eval.ts: sandboxed CLI agents run their MCP servers
+  // INSIDE the container and reach host-side services via
+  // host.docker.internal, which arrives on the host's bridge interface, not
+  // loopback; a 127.0.0.1-only listener refuses those connections).
+}).listen(port, '0.0.0.0', () => {
+  console.log(`Docs content API: http://127.0.0.1:${port}/docs/api/graphql`);
+});

apps/framework/scripts/docs/sentry-stub-loader.mjs

@@ -0,0 +1,11 @@
+// fallow-ignore-file unused-file -- registered at runtime by sentry-stub-register.mjs via module.register()
+// Loader-thread resolve hook: '@sentry/nextjs' -> the no-op stub.
+const stubUrl = new URL('./sentry-stub.mjs', import.meta.url).href;
+
+// fallow-ignore-next-line unused-export -- Node loader-hook contract: the module system calls `resolve`
+export async function resolve(specifier, context, next) {
+  if (specifier === '@sentry/nextjs') {
+    return { url: stubUrl, shortCircuit: true };
+  }
+  return next(specifier, context);
+}

apps/framework/scripts/docs/sentry-stub-register.mjs

@@ -0,0 +1,9 @@
+// fallow-ignore-file unused-file -- loaded at runtime (spawned/injected by local-docs.ts), never statically imported
+// Registers a resolve hook that short-circuits '@sentry/nextjs' to the local
+// no-op stub. Injected via NODE_OPTIONS from `pnpm local docs api`; chains with tsx's
+// own hooks (ours only intercepts the one specifier). Uses module.register()
+// (Node 20.6+) rather than registerHooks() (22.15+) — mise pins node "22",
+// which an older 22.x install satisfies.
+import { register } from 'node:module';
+
+register('./sentry-stub-loader.mjs', import.meta.url);

apps/framework/scripts/docs/sentry-stub.mjs

@@ -0,0 +1,9 @@
+// fallow-ignore-file unused-file -- loaded at runtime (spawned/injected by local-docs.ts), never statically imported
+// No-op @sentry/nextjs stand-in for the standalone docs content API.
+// The route handler calls Sentry.captureException/flush; under plain tsx
+// (outside Next's Sentry instrumentation) the real package's ESM build
+// resolves without those functions and every request crashes. A local dev
+// adapter has no business sending telemetry anyway. Wired up by
+// sentry-stub-register.mjs (see local-docs.ts).
+export const captureException = () => '';
+export const flush = async () => true;