Skip to content

M3: docs monorepo as a sparse+partial submodule - #119

Closed
barryroodt wants to merge 19 commits into
workspace-foldfrom
docs-submodule
Closed

M3: docs monorepo as a sparse+partial submodule#119
barryroodt wants to merge 19 commits into
workspace-foldfrom
docs-submodule

Conversation

@barryroodt

@barryroodt barryroodt commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Stacked on #118 (M2 fold) — retarget to main after it merges.

Implements M3 from the migration plan agreed in the workspace-layout thread: the docs monorepo becomes submodules/supabase, a sparse+partial submodule pinned by evals, replacing the managed clone-docs flat clone. All three source repos (agent-skills, mcp, supabase) now share one mechanism.

What this buys

  1. One mechanism for all three loops — the managed-clone plumbing (manifest remote, flat clone) is gone; supabase is a submodule entry like mcp.
  2. In-repo docs provenance — the pinned SHA records exactly which docs corpus evals ran against; CI phase 6 (per-arm ephemeral index) can diff against the pin.
  3. Opt-in stays opt-in — a plain clone leaves the gitlink empty; the seed is mise run clone-docs.

Mechanics (all verified live)

  • Seed: blob-filtered clone without checkout → cone sparse-checkout (apps/docs + 13 packages) → explicit checkout of the gitlink pin. ~56s, 311M .git + 622M worktree (vs 2.9G flat clone). Re-runs refresh the sparse set only, never move HEAD.
  • update = none in .gitmodules: a recursive submodule update --init would materialize the entire monorepo tree (sparse config can't be injected before the atomic clone+checkout). Verified: git submodule update --init prints Skipping submodule 'submodules/supabase', so CI's submodules: recursive checkout stays safe.
  • ignore = all: marker commits and pin-bump rebases in the working tree never show as dirt in evals.
  • Pin bumps: mise run update keeps the fetch + rebase-markers flow and prints the deliberate record step (git add submodules/supabase && git commit). The generic managed-clone update flow is deleted — nothing used it.
  • Pin: 846706b — the base the current local index was embedded from, so the existing content DB carries over with zero re-embed drift.
  • pnpm toolchain split: the docs monorepo pins packageManager: pnpm@11.13.1; evals' mise pins 10.24. Docs-loop pnpm calls run corepack pnpm with cwd inside the submodule, so the monorepo's own pin wins. (This would have bitten the flat clone on its next reinstall too.)
  • Sentry stub (second commit): pre-existing drift — the GraphQL route calls Sentry.flush/captureException, which don't exist in the package's ESM build under standalone tsx; every request crashed. A node:module resolve hook no-ops the specifier. The old workspace's docs-api only worked on stale node_modules and fails today as well.

Verification

  • Seed + apply-patches (5 markers) + pre-push guards in the submodule; trap check green.
  • Self-tests: status 19/0, hooks 16/0 (supabase patch paths now exercised), ab 23/0.
  • docs-api serves searchDocs from the seeded index (1769 pages) through the submodule route; A/B dry-run resolves the docs loop with both env overrides.
  • status renders supabase as a third nested submodule row + git version floor check (seed needs git ≥ 2.26).

Open follow-ups (per plan)

  • Docs pin-bump cadence/owner (manual mise run update vs renovate-style) — same ownership question as the mcp pin-sync companion.
  • CI phase 6 containerizes docs-api per arm (PLAN §3e); local stays a host process — discussed, the toolchain split makes a local container net-negative.

@vercel

vercel Bot commented Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
evals Ignored Ignored Preview Jul 25, 2026 11:03am

Request Review

@barryroodt
M3: docs monorepo as a sparse+partial submodule (submodules/supabase)
7694f7f 
Replaces the managed clone-docs flat clone with a pinned submodule, so all
three source repos (agent-skills, mcp, supabase) share one mechanism and
the docs pin gives in-repo corpus provenance.

- Seed recipe (verified: ~56s, 311M .git + 622M worktree vs 2.9G flat):
  blob-filtered clone without checkout, cone sparse-checkout, explicit
  checkout of the evals gitlink pin. Re-runs refresh sparse only.
- Guardrails: 'update = none' (a recursive submodule update would
  materialize the ENTIRE monorepo tree - verified: 'git submodule update
  --init' and CI's recursive checkout skip it; the seed owns the checkout)
  and 'ignore = all' (marker commits and pin-bump rebases never show as
  dirt in evals).
- Pin bumps: update.sh keeps the fetch+rebase-markers flow and now prints
  the deliberate-record step (git add submodules/supabase). The generic
  managed-clone flow is deleted - nothing used it anymore.
- Pinned at 846706b, the base the current local index was embedded from
  (zero re-embed drift; the shared content DB carries over as-is).
- status.sh: supabase rendered as a third nested submodule row + a git
  version floor check (seed needs partial clone + cone sparse, git 2.26+).
- pnpm split: the docs monorepo pins its own pnpm (packageManager 11.13.1)
  vs evals' mise-pinned 10.24 - all docs-loop pnpm calls go through
  'corepack pnpm' with cwd inside the submodule so the monorepo wins.
- docs-content-api.ts / docs-api.sh: submodule path is one level deeper;
  fixed the route import and tsx entry path depth.

Verified: seed + apply-patches (5 markers) + pre-push guards; trap check;
self-tests 19/16/23 green; docs-api serves searchDocs from the seeded
index (1769 pages) through the submodule route.
@barryroodt
fix: stub @sentry/nextjs in the standalone docs content API
5df8415 
Pre-existing drift exposed by the fresh seed: the GraphQL route calls
Sentry.captureException/flush (upstream since 2025-06), but under plain
tsx (outside Next instrumentation) the package's ESM build resolves
without those functions - every request crashed. The old workspace only
worked on stale node_modules; its docs-api fails today too.

A node:module resolve hook (NODE_OPTIONS --import in docs-api.sh) short-
circuits '@sentry/nextjs' to a no-op stub - no upstream-file patch to
maintain, and a local dev adapter should not ship telemetry anyway.
@barryroodt
fix: Sentry stub via module.register() (Node 20.6+), tsx invoked dire…
9706e47 
…ctly

registerHooks() needs Node >= 22.15 while mise pins node "22" - an older
22.x install would fail before startup (review catch). The register()
loader API covers all supported nodes. Direct node_modules/.bin/tsx
invocation replaces 'corepack pnpm exec': NODE_OPTIONS must reach only
the server process - pnpm's own node chokes on the loader hook while
probing pnpmfiles. corepack stays install-only (clone-docs/update).

Re-verified live: docs-api serves searchDocs from the seeded index.
@barryroodt
fix: pin the docs gitlink at the upstream base, not the local marker tip
fb1ad8d 
Caught by the cold-install demo: the committed gitlink recorded db16c89,
the marker-commit tip of the local submodule (an explicit 'git add
submodules/supabase' bypasses ignore = all). Markers never leave the
machine (pre-push guard), so every cold seed died with 'upload-pack: not
our ref' - and the aborted seed left a half-state (clone done, checkout
not) that re-runs never repaired.

- gitlink re-pinned at 846706b (the upstream base).
- clone-docs.sh: the pin checkout now runs whenever the worktree isn't
  materialized (fresh seed AND recovery from an aborted one); an
  unfetchable pin fails loud with the marker-gitlink diagnosis instead
  of half-seeding.
- hooks.test: new check that the gitlink's subject is not an
  [eval-workspace-*] marker (the accident class ignore = all can't see).
@barryroodt
fix: ab-demo guide revert used a stale strip prefix (caught by the li…
a18151b 
…ve demo)

The M3 path swap left ${GUIDE#supabase/} unstripped (GUIDE now starts
with submodules/), so cleanup passed a nonexistent pathspec to the
submodule checkout, '|| true' swallowed it, the canary stayed in the
guide, and the 'de-embed' re-indexed the canary. The preflight clean-file
check was silently vacuous for the same reason. Both prefixes fixed; a
failed revert now warns loudly instead of masking itself.
@barryroodt
fix: close the skip->purge hole in the docs embed pipeline
4c38d6a 
Two rows lost their protection when a source went missing:

- lint-warnings-skip (LOCAL-ONLY): a skipped source returned [] and
  vanished before preparation, so purgeOldPages deleted its still-valid
  rows as stale. The skip now registers its path scope
  ('/guides/database/database-advisors?%' - the '?' keeps the base guide
  purgeable) and the purge excludes it. Scoped, not global: a blanket
  purge suppression would leave treatment-added pages contaminating A/B
  baselines forever.
- index-fail-closed (upstream candidate): fetchPartners() swallowed the
  query error, so a transient API failure looked like zero partners and
  the purge deleted every /partners/integrations/* row. It now throws,
  aborting the run before the purge.

Marker commits rebuilt; tree identity + hooks.test (17/0) green.
@barryroodt
fix: embed-env patch check by marker commit, not textual reverse-apply
2132df6 
The reverse-apply check broke as soon as a later patch legitimately
touched the same lines (lint-warnings-skip now extends fail-closed's
purgeOldPages), failing every docs-seed/index on a correctly-patched
tree. Check the marker-commit subject instead - the same invariant
apply-patches verifies with tree identity. Caught by cold run #2.
@barryroodt
fix: marker check must consume the log stream (pipefail SIGPIPE hazard)
23d9ac7
@barryroodt
fix: partners source skips with purge protection under DOCS_EMBED_ALL…
ce467ac 
…OW_MISSING_SOURCES

The fail-loud partners fix immediately caught a real, previously silent
failure: the misc API rejects the legacy anon key locally ('Legacy API
keys are disabled'), so partners has ALWAYS quietly indexed as zero on
this machine - strong candidate for the 27 purged rows. Upstream keeps
the hard throw (abort before purge); locally the ALLOW flag now treats
partners like lint-warnings: warn, register '/partners/integrations/%'
in the shared scope registry (moved to base.ts, both sources push), and
skip. Caught by cold run #3.
@barryroodt

barryroodt commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Cold run is green! Fresh clone, no containers, no volumes, zero interventions on the final pass.

Phase Time
Setup (clone → seeded index → serving API) 5:59 + $0.12 one-time embed
Demo (mise run ab-demo, 2 Sonnet runs) 2:42
Teardown (back to nothing) 0:30 
baseline   passed=false checks=0/1
treatment  passed=true  checks=1/1
-> edit IMPROVED the eval (FAIL->PASS)

Cold here means fresh workspace + fresh Docker data; host prerequisites (mise, Docker images, keys, SSH) were already in place. Cleanup verified clean afterwards: guide reverted, canary de-embedded, corpus back to exactly 1742 pages. I think the real test is a cold run on someone else's machine.

@barryroodt barryroodt self-assigned this Jul 24, 2026
@barryroodt
review: receipt coverage, vacuous test, dead clone concept (thermo pass)
8c57a80 
Self-review findings on this branch, all verified live:

- provenance.mjs recorded working-tree state only for non-submodule
  entries - now the empty set, so the docs submodule's dirty state (the
  very thing a docs A/B varies) silently left receipts, and mcp never
  had it. The model was wrong: what earns a tree record is 'carries
  patches' (a deliberately floating tree), not 'not a submodule'.
  submodules.<name> now enumerates from git (status --cached) and
  records the PIN, so a new submodule can't drop out of receipts;
  repos.<name> records working reality for patch repos. hooks.test
  asserts both (19/0).
- ab.test's docs mixed-path check still used the pre-M3 path, so it
  exited 2 at loop selection instead of the scope check it claims to
  test - vacuous. Fixed + a positive docs-mapping assertion (24/0).
- docs-embed-env hardcoded the marker subject; patches-lib owns that
  format (patch_subject) - now reused.
- The 'plain clone' concept is dead: repo_remote() had zero callers,
  manifest.mjs carried a dead schema branch + stale comments. Deleted;
  kind: submodule is now required.
@barryroodt
chore: merge workspace-fold (repo-wide biome formatting)
6d8bb86
@barryroodt
fix: A/B cancellation recovery + pre-spend seeded-index gate
2905cc5 
Two gaps observed live during warm-demo runs:

- A hard kill mid-baseline bypassed the EXIT trap and stranded the
  user's edit in an unmarked stash; the next run then failed with a
  cryptic 'no unstaged edit'. Now: catchable signals (HUP/INT/TERM)
  route through exit so the EXIT trap restores exactly once (never
  attached directly - signal + EXIT would double-pop); the baseline
  stash carries an identifying message; and preflight detects a
  stranded marked stash and prints the exact recovery command (covers
  SIGKILL, which no trap can). Proven live: TERM mid-baseline restores
  the edit with no stranded stash. 3 new ab.test checks (27/0).
- The docs preflight checked docs-api but not the index: an
  empty-but-running DB (another session's teardown, volume recreated)
  would turn the treatment sync into a full paid seed instead of
  failing pre-spend. Now refused with a pointer at docs-seed and the
  no-args readiness probe.

README: restore guarantee stated precisely (incl. the SIGKILL case)
+ discriminator-eval design tip (vague questions run for minutes).
@barryroodt
fix: exact stash ref in recovery; zero-cost eval validation preflight
5a57d5d 
Two review catches on the cancellation hardening:

- The recovery message printed a plain 'git stash pop', which takes the
  NEWEST stash - a user's own unrelated stash above the stranded one
  would get popped instead. Detection now resolves and prints the exact
  marked ref(s), highest index first so remaining refs stay stable.
  Tested with a decoy stash above the stranded one (ab.test 28/0).
  (Also: the refactor moved the detection grep out of an if-condition,
  where an empty stash list + set -e silently killed every run - guarded.)

- Eval schema/typo errors surfaced only at harness discovery, AFTER the
  treatment re-embed had spent (observed live: an invalid topic enum cost
  an embed). ab.sh now validates evals/<id>/PROMPT.md through the
  harness's own parseEvalMarkdown before any paid sync; skipped under
  the AB_EVAL_CMD test hook. Probed live: missing eval and the exact
  invalid-topic case both refuse pre-spend.
@barryroodt
fix: mise ab wrapper refuses a path in the experiment slot
ce5f869 
'mise run ab <eval> <p1> <p2>' silently fed p2 as the experiment name,
which would only fail at harness discovery - after the paid docs sync.
Refuse pre-spend with the direct-script form for multiple paths.
@barryroodt
feat: vs-main screen + per-worktree docs isolation
1c3b14e 
vs-main — treatment-only screen against published results (evals main):
  mise run vs-main <eval-id>… [--experiment <id>] [--runs N]
Runs eval(s) in the edited world (mcp/docs/skills edits combined; dirty
trees auto-detected and synced) and diffs against the freshest bot-refreshed
row in apps/web/src/data/{regression-,}eval-results.json. No baseline arm,
no stashes, no git mutation. Receipts record the published arm's provenance
(result commit + parent SHA + age + attempts) and the treatment's usual
provenance embed. Refuses pre-spend on unknown eval/experiment (lists the
published alternatives) and reuses ab.sh's zero-cost eval-metadata gate.
A published-vs-local flip is a screen, not causal proof; the output says so
and points at mise run ab.

docs-isolate — per-worktree docs stacks for interference-free sessions:
  mise run docs-isolate && mise run docs-up && workspace/scripts/docs-copy-index.sh
Writes an UNTRACKED overlay workdir (workspace/.docs-stack: rewritten
config.toml + symlinks into the submodule) so no tracked file in any clone
changes. Slots (project id + 55[1-9]21 port block + docs-api 300[1-9]) are
allocated through a machine-shared registry under an atomic mkdir lock,
keyed by canonical worktree path — concurrent isolations cannot collide
(a docker-only scan would race: containers appear at docs-up, not isolate).
docs-copy-index seeds the new stack from a sibling's warm index via
pg_dump/restore: zero OpenAI spend, checksums included.

All docs scripts + ab preflights now resolve the stack through
docs-profile.sh (workdir, project id, container, ports) instead of
hardcoded names/ports; the primary worktree keeps its exact current
identity (slot 0). Self-tests: vs-main.test 13/0, ab.test 28/0.
@barryroodt
fix: slot ports below the macOS ephemeral range; live-proof fixes
63e30af 
The 55xxx slot blocks sat inside the macOS ephemeral port range (49152+):
observed live, a transient outbound connection held 55422 and the slot-1 db
container could not bind. Slot ports now live at 43k21-43k24; the docs-api
port derives from the project-id suffix instead of port arithmetic, and the
profile's port regex accepts any block.

Also from the live proof (isolate -> docs-up -> copy-index on slot 1,
primary running throughout): fresh numeric slot allocation was colorized by
FORCE_COLOR into NaN ports (String() it), an apostrophe broke the allocator
quoting, and setval noise is silenced. Verified: overlay stack boots on the
symlinked workdir, free copy lands 1742 pages/$0.00, both stacks coexist,
teardown leaves zero containers/volumes.
@barryroodt
feat: vs-main --no-compare — run dirty trees without a published base…
e55d2f3 
…line

The auto-detect+sync+receipt machinery was locked behind published-baseline
existence, which excluded exactly the evals you write yourself for a docs
edit (custom ones aren't in any published export). --no-compare skips
baseline resolution entirely: same treatment sync, same provenance receipt,
result-only report. Published attempts drive the default run count only in
compare mode (a stale published.json from an earlier run must not leak in).
vs-main.test: 18/0.
@barryroodt
feat: mise run experiments — list experiments with model/effort + pub…
f4a1468 
…lished-baseline marker
@barryroodt
fix: experiments lister — restore effort column (clobbered by the mod…
4b2a1de 
…el-regex fix)
@barryroodt
barryroodt marked this pull request as draft July 27, 2026 07:21
@barryroodt barryroodt mentioned this pull request Jul 27, 2026
Closed
@barryroodt

barryroodt commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Parking this for now. The workspace glue grew very macOS-shaped (keychain, BSD sed, bash everywhere) and heavier than the job needs, and with open-sourcing on the horizon that's the wrong foundation. We're redoing the same concepts TypeScript-first in a new PR: treatment-only runs compared against the published results on main, provenance receipts, and explicit input overrides instead of submodules + patches. Branch stays for reference. A few fixes found while building this should land upstream in supabase/supabase regardless (guide checksums, the skip->purge interaction in generate-embeddings, fail-closed partner fetch).

@barryroodt barryroodt mentioned this pull request Jul 27, 2026
Open
@raulb
raulb deleted the branch workspace-fold July 29, 2026 11:10
@raulb raulb closed this Jul 29, 2026
@raulb
raulb deleted the docs-submodule branch July 29, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@barryroodt barryroodt

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@barryroodt @raulb