Fold the eval-source workspace into evals (M2) - #118
Closed
barryroodt wants to merge 10 commits into
Closed
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 08:15
086371f to
a0f6e6a
Compare
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 12:05
a766305 to
e4a639c
Compare
SUPABASE_CONTENT_API_URL (or the contentApiUrl option) adds --content-api-url to the server args, pointing search_docs at an alternative docs index (e.g. a locally built one). Requires a server that understands the flag — the SUPABASE_MCP_SERVER_PATH build; inert otherwise. Retires the eval-workspace enabler patch of the same name.
Verbatim copy of the eval-workspace repo's scripts/, patches/, and manifest (minus the two evals enabler patches: the local-build override shipped with the submodule PR, and content-api-url just landed as a real commit), plus the mise task file at the root and gitignore entries for the glue's artifacts. Re-rooting the scripts for life inside evals is the next commits' job — this one is the reviewable copy boundary.
evals is the host now, not a managed clone: every evals/ path prefix drops, 'cd evals && pnpm eval' becomes a root pnpm eval, and the evals manifest entry + marker plumbing die. The mcp clone is replaced by the pinned submodules/mcp working tree (patches apply there as marker commits; submodule.submodules/mcp.ignore=all keeps that patch dirt out of host status/commits — pin bumps are a deliberate git add). supabase stays the opt-in sparse clone. Keychain service name is unchanged so existing stored keys keep working. Self-tests from the repo root: status.test 19/0, hooks.test 8/0 (real install/chain/reinstall lifecycle against the initialized mcp submodule), ab.test 22/0.
…ee self-test workflow workspace/README.md is the ops doc (bootstrap, the three loops, A/B, provenance, self-tests); the root README points at it. The self-test workflow runs the three shell-only suites on workspace/** PRs: no keys, no model spend, no docker.
- ab.test.sh cleanup could destroy a user's unstaged stdio.ts edit: the EXIT trap reverted the mcp fixture file unconditionally while the fixture block only runs on a CLEAN file - exactly the state where the revert nukes user work. Cleanup now only reverts what the test itself wrote (verified: planted user edit survives a skipped-fixture run). - fold demo/canary-eval (ab-demo died at cp: fixtures were never folded). - affected.ts emits source-selecting commands: mcp/server-wide paths get mise run mcp-eval; docs paths get SUPABASE_CONTENT_API_URL + mcp-eval (a bare 'mise run eval' measured production, not the local edit). - workspace/README.md documents the docs-loop measurement step. - workspace-selftest.yml: pin setup-node to the repo's SHA convention, node-version-file instead of an inline 22. - dedupe the paid docs-embed preamble (docs-embed-env.sh sourced by docs-index/docs-seed) and provenance's untracked-hashing (one hashUntracked helper for receipt + stamp). Self-tests: status 19/0, hooks 8/0, ab 22/0.
- canary EVAL.ts imports @supabase-evals/core (real package; was elided at runtime but broke type resolution) - SUPABASE_CONTENT_API_URL env fallback now applies only alongside the SUPABASE_MCP_SERVER_PATH override: a stray env var can no longer feed --content-api-url to the published npx server (which rejects unknown flags). Explicit contentApiUrl option stays unconditional. Tests adjusted, +1 covering the stray-env npx case. - affected.ts: skips an eval dir missing PROMPT.md instead of aborting; the docs command's '# needs:' hint moved to its own line - content-API URL literals cross-referenced at each site (4-way sync note) - PUBLISH_REPOS identity alias collapsed into PATCH_REPOS everywhere - mise.toml pins pnpm 10.24.0 (matches packageManager/lockfile) - status.sh drops the stale 'incl. in agent-skills' guard comment - ab.sh fullwidth parens -> ASCII; .gitignore double blank dropped - status.test.sh anchors on the exact 'Ready. Try:' banner - hooks.test.sh/ab.test.sh fail actionably on zero checks instead of passing vacuously when submodules are missing Verified: status 19/0, hooks 8/0, ab 22/0, core mcp-server 12/12, framework typecheck clean, affected smoke on a real docs path.
apply-patches verified 'state matches the canonical patch' two ways:
byte-exact tree-OID reconstruction for an existing marker commit, but a
grep-payload diff for a fresh build - the weaker check, blind to an
identical +/- line reattributed across files within a multi-file patch.
Extract patch_tree() (throwaway index: parent-tree + patch -> tree OID)
and compare tree identity on BOTH paths; the payload-diff mechanism is
deleted and its blind spot closes as a side effect.
Verified live: existing-commit path ('already committed'), fresh path
(marker dropped via rebase --onto, rebuilt through the new check);
hooks 8/0, ab 22/0, status 19/0.
1. mcp-build built the UNPATCHED server and orphaned the marker commit: it delegated to the root 'pnpm mcp:build', whose first step is another 'git submodule update --init' - on an initialized submodule that re-checks out the pin, discarding the marker apply-patches created a step earlier (and any user commits). The task now runs init -> install -> apply-patches -> build directly; the root script remains the evals-native unpatched flow. Verified: after mcp-build the marker is HEAD and the built stdio.js carries --content-api-url. 2. Arm receipts mislabeled the run: ab.sh passed RUN_ENV to the eval but not to the 'provenance --embed' step, so a local-build arm recorded mcp_override: null. The embed now runs under the same env; ab.test gains a receipt-records-override assertion (23/0). Found by a real cold-clone demo: GitHub clone -> setup -> unstaged mcp edit -> live A/B (baseline PASS 5/5 -> treatment FAIL 3/5, REGRESSED).
…e next release + pin bump
barryroodt
force-pushed
the
workspace-fold
branch
from
July 24, 2026 12:09
e4a639c to
25b5f73
Compare
barryroodt
marked this pull request as draft
July 27, 2026 07:21
Contributor
Author
|
Parking this alongside #119 while we redo the local-dev story TypeScript-first (cross-platform matters once this repo is open-sourced, and the bash + macOS-specific glue doesn't get us there). The core piece here (SUPABASE_MCP_SERVER_PATH + tests) is exactly what the new PR re-lands, minus the submodule, and the review feedback from this thread carries over. Branch stays for reference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #109 (retarget to main when it merges). Moves the eval-workspace glue into this repo, per the direction agreed in the workspace-layout Slack thread: the sources under test (docs, skills, MCP server) now live beside the harness that tests them.
What's here (commit per step)
packages/core):SUPABASE_CONTENT_API_URL/ acontentApiUrloption adds--content-api-urlto the server args, pointingsearch_docsat an alternative docs index. Requires a server that understands the flag (theSUPABASE_MCP_SERVER_PATHbuild; upstream PR: mcp#343); inert otherwise. Retires the last evals enabler patch.workspace/+ themisetask file at the root (Supabase convention) — the reviewable copy boundary, no adaptation.evals/path prefixes drop,cd evals && pnpm evalbecomes a rootpnpm eval, the evals manifest entry + marker plumbing die, and the mcp clone is replaced by the pinnedsubmodules/mcpworking tree (enabler patches apply there as marker commits;submodule."submodules/mcp".ignore=allkeeps patch dirt out of host status, so pin bumps are a deliberategit add). The supabase docs monorepo stays an opt-in sparse clone.workspace/README.mdis the ops doc (bootstrap, the three edit→eval loops, head-to-head A/B, provenance receipts); a free self-test workflow (shell + node stdlib, no keys/docker/model spend) runs onworkspace/**PRs.Verification
git clone→statusprints the exact bootstrap steps →setup.shreaches Ready in ~16s (install, selective submodule init, patch markers, keychain keys) → all three self-tests pass in the fresh clone.Not in scope
Archiving the eval-workspace repo (after this merges), the docs-monorepo submodule (M3, planned), and the pin-sync/canary companions.