TypeScript local-dev runner: pnpm local (run/compare/experiments/docs) - #128
Open
barryroodt wants to merge 24 commits into
Open
TypeScript local-dev runner: pnpm local (run/compare/experiments/docs)#128barryroodt wants to merge 24 commits into
barryroodt wants to merge 24 commits into
Conversation
Re-lands the core piece of #118 without the submodule: createConfig resolves a local MCP build (path canonicalized against the evals checkout root, symlinked dist/ supported, mount fallback for containerized agents) when SUPABASE_MCP_SERVER_PATH is set. Includes the review fixes from the #118 thread (canonicalize base once, receipt-visible mounts) and the full test suite (13 mcp-server tests).
A cross-platform, treatment-only workflow for running evals against local inputs — no bash, no macOS keychain, no git mutation, no submodules: pnpm local run <eval...> [--experiment <id>] [--runs N] [--mcp <path>] [--content-api <url>] pnpm local compare <eval...> [same] # + diff vs the latest published result on origin/main pnpm local experiments # agent/model/effort + published-baseline availability pnpm local docs <up|seed|api|down> --docs <path> # minimal local docs content API - run/compare execute the harness with explicit input overrides (local MCP build via SUPABASE_MCP_SERVER_PATH, docs via SUPABASE_CONTENT_API_URL) and write provenance receipts (host SHA + dirty state, mcp override git state, content-api URL) to results-local/. compare records the published arm's result commit, parent SHA, and age, and says explicitly that a flip is a screen, not causal proof. - pre-spend gates: eval metadata parsed via the harness's own parseEvalMarkdown, experiment existence checked, --mcp path validated — all before any model call. - docs loop is deliberately minimal: full seed only, from YOUR supabase/supabase checkout (--docs), run through the docs app's own pipeline into a generated .local-docs workdir (own project id, 443xx ports — off the evals local-stack range and below the macOS ephemeral range). Incremental re-embed waits for the upstream pipeline fixes. - smoke test (test:local): 8 checks, zero model/docker cost. Verified live end to end: docs stack boots from the generated workdir, the standalone content API serves the docs route under tsx with the Sentry no-op stub, and searchDocs returns ranked results against a seeded index.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Self-contained brief for a design agent: content (final copy for all three workflows), page structure, comparison table, visual direction, and constraints for a single-file HTML page presenting the pnpm local skills/mcp/docs workflows.
Contributor
Author
|
Added docs/local-workflows-design-brief.md: a self-contained design brief for a single-page HTML presentation of the three workflows (skills / mcp / docs), for future reference. |
Operational rules for coding agents: how to verify skills/mcp/docs changes with pnpm local, interpretation rules (screen not proof, --runs 3 before claims, tool-call activation for mcp, tools-mode evals for docs), spend rules (docs seed ~$0.12, confirm before paid steps, don't bypass pre-spend gates), and repo conventions. CLAUDE.md points at it.
- Decompose the two functions over the cognitive-complexity threshold: cmdRunOrCompare 36 -> 7 (buildOverrideEnv + runTreatment + reportComparison) and resolveBaselines 28 -> under threshold (loadPublishedFile + scanFileRows + refuseMissingBaselines), with named PublishedFile type. - Suppress the four runtime-loaded docs files (content-api-server, sentry stubs) as unused-file false positives with reasons: they are spawned/injected at runtime (tsx argv, NODE_OPTIONS --import, module.register), never statically imported; same for the loader-hook resolve export. fallow audit vs origin/main after this: introduced dead-code 0, introduced duplication 0, no introduced function above the cognitive threshold. Remaining audit findings are inherited (main's unused deps, run-eval complexity, pre-existing dup groups) or CRAP-from-zero-coverage on script files, consistent with the repo baseline. Behavior verified: typecheck clean, smoke-local 8/8, experiments/compare output unchanged.
Four structural fixes from a strict self-review: - Delete both bespoke arg parsers (local.ts parseArgs, local-docs.ts flag loop) for node:util parseArgs — two fewer concepts, typed values without casts, and unknown flags now fail loudly instead of being silently swallowed. - Delete the hand-rolled PublishedRow shape for the repo's own canonical contract: rawEvalResultSchema (+ new RawEvalResult type exported from eval-metadata, the module that owns the schema). Treatment results are now VALIDATED against the contract instead of blind-cast, so a malformed result file fails with a schema error rather than propagating undefined. - Restructure resolveBaselines declaratively (per-eval candidates -> freshest match), deleting the scanFileRows accumulator threading and the seen-map bookkeeping. - Drop the redundant mode param from runTreatment (baseline presence already encodes it). Behavior identical: typecheck clean, core 57/57, smoke-local 8/8, error paths verified verbatim. fallow: introduced dead-code 0, duplication 0, nothing over complexity thresholds.
…ver and missing skills Two failures from the first manual validation run: - The core resolver wants the server package dir, but every natural invocation passes the checkout root. resolveMcpServerPath now accepts either (root resolves to packages/mcp-server-supabase) and refuses pre-spend when dist/transports/stdio.js is missing, with the build command — previously the harness discovered that only inside eval setup. - An uninitialised skills submodule made the harness warn-and-skip the experiment's declared skills: a treatment silently running skill-less against a skills-enabled published baseline is a world mismatch, not a comparison. validateSkills refuses pre-spend with the submodule command. smoke-local: 10 checks (new: unbuilt-checkout refusal with hint, root -> package-dir resolution asserted via the receipt's mcpOverride.path). Proven against a real mcp checkout: root path resolves, receipt records the canary commit sha.
Observed on a manual run: the agent (claude-code) completed its full paid run, then scoring failed at assertProviderReady — the eval's scorer grades with the LLM judge, which is an OpenAI model regardless of the agent under test. 16 evals use the judge today. validateJudgeKeys now scans the selected evals' scorers and refuses BEFORE any agent spawn when OPENAI_API_KEY is missing, naming the affected evals. smoke-local: 11 checks; the new one selects a known-judged eval from the published set (the generic pick may be unjudged, which would reach a real agent spawn — that was a test-design bug caught while writing it). README/AGENTS key docs updated.
- pnpm local compare --suite <regression|benchmark> expands to every eval the published export carries for the experiment (printed with count before running: one model run each). Refused in run mode, with explicit ids, and for unknown suites. - The smoke suite now redirects every output into a mkdtemp sandbox (LOCAL_RESULTS_ROOT seam). Previously it faked results and cleaned up under the checkout's real results/ + results-local/ paths — which can clobber an in-flight manual run's receipts for the same eval id. Observed live. smoke-local: 15 checks.
…t fake Second strict-review pass. The incremental fix rounds left suite-mode fetching origin/main twice (suite branch + inside resolveBaselines) and parsing both multi-MB published exports twice per run, plus SUITE_FILE duplicating PUBLISHED_FILES' literals. Now: PUBLISHED_EXPORTS is the one suite->file map (values double as the load list), compare mode does fetchMain() + loadPublished() exactly once, and resolveBaselines/ expandSuite are pure over the preloaded files (the fetch param is gone). smoke-local's per-platform quoted FAKE one-liner pair becomes a script file written into the sandbox — one string, no shell-quoting dialects. 15/15 checks; typecheck, biome clean; behavior verified unchanged.
Two frictions from validating mcp#333 through the runner: - Missing ANTHROPIC_API_KEY made the harness SKIP with exit 0, so the runner failed only at the no-result check. validateAgentKey refuses pre-spend — and distinguishes set-but-EMPTY (a stray `export KEY=` silently shadows .env because node --env-file never overrides an existing var; observed live) from genuinely unset. - A local mcp build from a newer release line can call endpoints the platform-lite fixture (which tracks the MCP_SERVER_VERSION pin) does not serve: get_logs moved logs.all -> /logs in 0.9.0 while pin + fixture sat at 0.8.x, so every logs eval 404s against local builds with no hint why. resolveMcpServerPath now compares the build's package version to the pin and prints a judge-by-activation warning on mismatch.
SUPABASE_CONTENT_API_URL support merged in supabase/mcp#343 but ships in no release yet (newest v0.9.0; pin is 0.8.1). Ungated, `--content-api` with the published package was a silent no-op: search_docs queried PRODUCTION docs while collectProvenance still stamped contentApiUrl into the receipt, so a paid run measured the wrong world and reported the right one. Refuse pre-spend when --content-api arrives without --mcp, or when the supplied build's dist does not read the env var. Shape check, so it also covers fake runs. +3 smoke checks (both refusals and the honoured-build accept path).
The gate added in 419b5bc stopped the silent-prod case but nothing delivered the override: createConfig returns only {command,args}, and CLI agents spawn that command INSIDE the sandbox container, which inherits nothing from the harness process. So SUPABASE_CONTENT_API_URL never reached search_docs. - createConfig now bakes --content-api-url into serverArgs, so rewriteLoopback maps 127.0.0.1 -> host.docker.internal for the containerised case (the same path --api-url already takes). Flag landed in supabase/mcp#343. - The docs content API binds 0.0.0.0 (still advertised on 127.0.0.1), matching platform-lite in tools mode (run-eval.ts): on native Linux docker, host.docker.internal arrives on the bridge interface, which a loopback-only listener refuses. - README/AGENTS: --content-api requires --mcp, and say why. Proven hermetically (no model spend): harness config -> real mcp build (main, #343) -> stdio -> local stub received both the schema load and the agent's searchDocs query, canary surfaced in the tool result. Negative control with the flag suppressed: zero stub requests, i.e. it went to production docs.
generate-embeddings.ts hard-requires NEXT_PUBLIC_MISC_URL and NEXT_PUBLIC_MISC_ANON_KEY before doing any work (they name the hosted "misc" project, read by sources/partner-integrations.ts). docs seed passed only the NEXT_PUBLIC_SUPABASE_* pair, so a seed against current supabase/supabase main aborted at the gate before embedding anything. Found running the loop end to end against a real checkout. NOT sufficient on its own: the same seed then fails on prod-only sources (lint-warnings and github discussions both hard-require DOCS_GITHUB_APP_* with no token fallback), which is what the old eval-workspace carried a local-only skip patch for. Upstreaming a skip flag remains the gate for a vanilla full seed.
…working The docs leg is not usable from a vanilla supabase/supabase checkout: the embedding pipeline fails closed on lint-warnings + github-discussion, which require a GitHub App (DOCS_GITHUB_APP_*, no token fallback). Verified by running it. The run/compare half IS verified against an externally seeded index, so say exactly that rather than implying the whole loop works. Also record the retrieval-vs-answer scoring rule: tools mode exposes WebSearch/WebFetch, and a docs edit contradicting the live page can be detected and rejected by the agent as prompt injection (observed, not theorised).
Only fetchLintWarningsGuideSources() blocks: it is awaited unconditionally and throws without the GitHub App creds, and one shared Promise.all makes that fatal. github-discussion has zero callers in the pipeline, so naming it as a blocker was wrong. Also state why the skip belongs upstream (this runner takes an arbitrary checkout; per-user patches are not distributable) and that a skip must preserve the skipped source's rows, since the purge deletes every page the run did not stamp.
Verified both: lint-warnings-skip only returns [], and fail-closed gates the purge on counted preparation/page/checksum failures. An intentional skip increments none of them, so the old setup did NOT protect that source's rows — the purge still ran. It never bit because a fresh index had no such rows. Minimal upstream fix is the opt-in skip alone. Skip-aware purge narrowing is a separate, newly identified retention requirement for re-running against an index that already holds those rows.
supabase/splinter is public and the loader makes 30 API calls, which fits the unauthenticated 60/hr budget for a single run; a PAT or gh auth token gives 5,000. So the gate is a rate-limit guard, not access control, and the better upstream fix is App creds -> GITHUB_TOKEN/GH_TOKEN -> unauthenticated. That keeps the 29 Database Advisor pages in the local index (no missing corpus, no purge-retention question) and leaves production on the first rung.
#43015 documents it explicitly: unauthenticated calls caused flaky failures on shared CI runners (failing job linked). And #44274 deliberately migrated GitHub content fetches TO the authenticated Octokit client after raw.githubusercontent failed in a preview, so proposing raw URLs would re-litigate that. Revised recommendation: keep the authenticated client, add one PAT rung (GITHUB_TOKEN/GH_TOKEN), no unauthenticated rung.
Contributor
Author
|
Carrying forward the pin-provenance reasoning from #109 (now closed in favour of this PR), since it outlives the mechanism that PR proposed.
It isn't academic. platform-lite tracks the pinned version, so a local build from a newer line can call endpoints the fixture doesn't serve: Cheapest follow-up, if we want it: record the release tag next to the version, so the fixture surface has a source revision to point at without carrying a submodule. |
The block described the auth ladder as a proposal; it is now an open PR. Replaces
the design discussion with the actual requirement (a --docs checkout containing
#48364, or that branch checked out until it merges) and records what the
end-to-end run showed: the seed completes, and a tools-mode eval's search_docs
returns content that exists only in the local index.
Keeps the two upstream rough edges visible rather than implying the seed is
healthy: it exits 0 while silently failing 22 /reference/{javascript,dart} pages
whose sections exceed the 8192-token embedding limit, and a local index has no
partner-integration pages because that source reads the hosted misc project.
Also drops 'PAT': the rung accepts any token Octokit takes, including a gh OAuth
token or an Actions installation token.
`docs up` streamed the CLI's own output, which meant every run printed the stack's ANON_KEY, PUBLISHABLE_KEY, SERVICE_ROLE_KEY, SECRET_KEY, and JWT_SECRET as JSON, plus the workdir line, a config deprecation warning, the stopped-service list, and the update-notifier nag. Our own one-line status was the last thing in a 20-line wall, and the keys are not something to leave on a screen recording. run() gains a quiet option that buffers and replays on failure, used for the three supabase/docker calls. capture() now swallows stderr for the same reason (execFileSync attaches output to the error, so failures still surface). docs up now reports the API URL it actually read back from the stack, which also means the status line only prints once the stack answers. docs down says where the seeded index went, since stopping keeps the volume. starting the docs stack (project evals-local-docs)... docs stack up on http://127.0.0.1:44321; next: pnpm local docs seed
raulb
reviewed
Jul 28, 2026
AI-939 Setup eval-skills-doc workspace
Set up a workspace that makes it easy to test how changes to docs, skills, and the MCP server affect evals: locally first, then via CI on PRs to any of the source repos. What we want
Where things stand
Plan (still the design reference): Eval workspace plan (AI-939) +++ Agent context (verified, current as of 2026-07-23)
+++ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the parked #118/#119 approach with a TypeScript-first, cross-platform implementation of the same concepts. No bash, no macOS keychain, no submodules, no patches, no git mutation anywhere.
What this adds
Concept: treat the three agent inputs as explicit overrides instead of managed clones. Skills = the tree in this repo. MCP = your own checkout, built, passed via
--mcp(re-lands #118's coreSUPABASE_MCP_SERVER_PATHresolution + its 13 tests, including the review fixes from that thread — thanks @Rodriguespn). Docs = a content GraphQL API served from your own supabase/supabase checkout viapnpm local docs, passed via--content-api.compareis a screen, not causal proof — the published arm ran in the scheduled CI world. The output says so, and the receipt records the published result commit, its parent (the ref the scheduled run built on), and its age, so the confound is explicit rather than silent.runneeds no published baseline at all (custom evals work).Receipts: every run writes
results-local/<eval>.treatment.jsonwith host SHA + dirty state, the mcp override's git state, and the content-api URL.Pre-spend gates: eval metadata parses via
parseEvalMarkdown, experiment existence and--mcppath are validated — all before any model call.Docs loop is deliberately minimal: full seed only, run through the docs app's own embeddings pipeline into a generated
.local-docsworkdir with its own project id and a 443xx port block (off the evals local-stack range, and below the macOS ephemeral port range — a transient outbound socket stealing a listen port is a real failure mode we hit). Incremental re-embed is deferred until the upstream pipeline fixes land in supabase/supabase (guide checksums are never set, so incremental re-embeds every guide; a source skipped for missing creds gets its still-valid rows purged).Verified
test:localsmoke: 8 checks, zero model/docker cost (fakes the run, reads real published baselines)searchDocsreturns ranked results against a seeded indexpnpm local experimentsreads experiment configs via dynamic import (accurate model/effort, no regex)