Skip to content

release: v2.0 public-release cleanup - #45

Open
seanrivera wants to merge 38 commits into
mainfrom
release/v2-cleanup
Open

release: v2.0 public-release cleanup#45
seanrivera wants to merge 38 commits into
mainfrom
release/v2-cleanup

Conversation

@seanrivera

@seanrivera seanrivera commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Scope note: this PR lands the whole fix/drop-key-bookkeeping feature line into main (it was ~100 commits ahead), topped by the 17 cleanup commits above. Squash-merge at release cut per the runbook below.

Consolidated release-cleanup PR derived from five independent model audits (fable / opus / sonnet / codex-gpt5 / gpt5.6-terra, 2026-08-05; review reports are local-only under docs/release_cleanup_reviews/). One commit per workstream, plus a live-main integration (below):

  • Hygiene: tracked vim swap + generated prompt dump removed; .gitignore hardened (editor swaps, .claude/, scratchpad/); frontier-probe DRAFT fixtures removed; completed-campaign launch scripts relocated to the results repo
  • Correctness: mazes/exp_maze_jsons/ retired — a stale, byte-drifted duplicate of the ogbench corpus carrying a maze.goal != goal.target violation in D3 that the D1-scoped assertion couldn't catch; maze tests now read the corpus the R1 manifests actually resolve; new corpus-wide goal-invariant test; one-shot disjointness scan extended to the live corpus
  • First contact: --recurse-submodules install docs; .gitmodules tracks the branch containing the pin; all 10 console entry points work (regression-tested); perf benchmarks opt-in (-m slow) so pytest is a deterministic install check; repo-root conftest.py makes unpackaged modules importable in any pytest invocation
  • Packaging/CI: v2.0.0 + MIT metadata + project URLs; wheel no longer ships mazes/deploy; minimal GitHub Actions workflow (submodules recursive)
  • Docs: 14 campaign/ops docs archived to the results repo's docs-archive/; docs index, RUNME, and public README rewritten around the canonical scripts.run_pipeline stack; smoke quickstart verified to pass the manifest guard end-to-end

Suite: 1034 passed, 21 perf tests deselected by default (opt-in via pytest -m slow). Verified from a fresh --recurse-submodules clone with no API keys.

🔀 Live-main integration (playable demo)

While this branch was in flight, main gained the playable demo (PRs #39/#40: demo/ package, play_task.py rewrite, watchdog/checkpoint refactor, +463-line default-manifest catalog). This PR merges it in rather than clobbering it:

  • Merge conflicts resolved: RUNME kept from the release rewrite (demo got its own §7, flags verified against the real CLI); the in-tree D3 maze stayed deleted (main's goal fix is byte-identical to the canonical submodule copy).
  • The demo's 42 default-manifest rows, its fallback default task, and a UI hint pointed at the deleted duplicate corpus — all repointed to the canonical ogbench corpus (same filenames; every target verified to exist).
  • Two pre-existing demo bugs fixed so a fresh clone works: the R1-comparison table loader hard-required a sibling Multinet-v2-results checkout and crashed play_task.py at startup (now sibling-or-nested, degrades to comparison-off), and episode end raised for any task outside the R1 table (now renders a plain end card; --record no longer loses trajectories).
  • Packaging: demo* added to the wheel (play_task became a shim over it); watchdog auto-merge verified coherent (single ProgressStallWatchdog drives live, resume, and demo paths — DROP-era progress signature preserved).
  • PR APIs for Demo UI #44 (demo_api) merged as well: demo/api/ HTTP subpackage + r1_config/r1_tasks modules; pyproject resolved as the union (release metadata/trims kept, their web extra adopted); the no-R1-table raise-removal re-applied where their refactor relocated it.
  • ⚠️ Note: the default gridworld/fixtures/manifest.json catalog grew 16 → 58 tasks (incl. the R1 panel) — it's a demo browse catalog; always pass explicit --run-config/--manifest for pipeline runs (RUNME §3 now says so).

⚠️ Evaluation-data callout (deliberate decision)

The current maze panels — mazes/validation_10, the conditional blind probe, gridworld/tasks, and the ogbench procgen corpus — are public by design for R1 reproducibility. These are NOT long-term held-out targets: evaluation panels rotate at the R3 release. When building the proper data pipeline, design for panel rotation and treat everything currently public as burned for held-out purposes.

✅ Corpus defects: fixed at source (was: frozen + tracked)

Retiring the duplicate corpus surfaced real defects in the pinned ogbench corpus; they are now fixed in the fork (fix/d3-goal @ 8d8cecd, pin bumped) and the KNOWN_CORPUS_DEFECTS tracking registry is retired — every family assertion runs strict again:

  • Goal corrected to (8,1) across the 10x10_dense_1 skeleton (S4, D3; M1–M6 confirmed already correct). Investigation showed (8,8) — the direction of the earlier 31a0549/319b902 fixes — is reachable via an always-open bottom corridor, making every mechanism optional; (8,1) is the designed, mechanism-mandatory goal that 208/214 of the corpus uses.
  • Metadata regressions restored (chain_pattern/description on five sg-family files, difficulty_tier on one D1 file — regressed in ogbench 271bf47).

⚠️ For the R1 analysis owner: S4/10x10_dense_1 and D3/10x10_dense_deadend_ky_dy_1 ran R1 at the wrong-direction (8,8) goal and may have been mechanism-bypassable — worth checking those episodes and their BFS baselines. Recorded episodes remain valid records of what was run; the corrected task specs invalidate caches for changed tasks by design (task-spec hash).

Merge blockers (human inputs — grep -n 'HUMAN:' README.md)

  • Logo asset at docs/figures/multinet_logo.png (or drop the img tag)
  • Release date + release-page link (Updates section)
  • Confirm headline counts (4/50, 1/50, 1/50) against final post-Kimi-rerun tables; add results page + arXiv links
  • v2 BibTeX (title/authors); full v1 author list
  • Paper/Discord badges when URLs exist

Notes for reviewers

  • Archived docs (and the two campaign launch scripts) live in the results repo under docs-archive/ — commit that repo separately. Per operator directive, all references to internal/unpublished documents were removed from tracked files (prose reworded to stand alone; manifest provenance reduced to bare set names; task rows / model blocks byte-identical — hash-verified).
  • The R1 manifests' task rows are byte-identical (verified by hash); only selection.source_set provenance strings were annotated.

Release-cut runbook (after approval — NOT part of this diff)

  1. Squash-merge this PR (keeps per-commit Claude-Session trailers out of the new history; verify the squash message carries none).
  2. git filter-repo message rewrite on main to strip the 62 pre-existing Claude-Session: trailer lines; force-push (coordinated).
  3. Delete stale remote branches (incl. postmortem/r1-teardown, fix/drop-key-bookkeeping, revert-*, cursor/*, pr*/) after checking each for unmerged work.
  4. Tag v2.0.0.
  5. Post-release: merge ogbench fix/d3-goal -> master and repin to master (corpus fixes already landed on fix/d3-goal); optionally ask GitHub support to gc unreferenced objects.

🤖 Generated with Claude Code

seanrivera and others added 30 commits August 1, 2026 18:24
@seanrivera @claude
feat(baselines): opt-in DROP edge for DROP-harness episodes
7b9f7e8 
The analysis planner deliberately had no DROP edge because R1 never exposed
DROP to the model, which is what makes a decoy-key pickup 'doomed'. The
2026-07-30 rerun did expose it, so scoring those episodes on a DROP-free graph
would call a recoverable state unwinnable.

drop_available defaults to False, so every existing caller is unchanged. The
dropped key stays in collected_keys (gone from the world, not re-acquirable):
conservative, and sufficient because no corpus episode ever emitted a DROP.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123tBVPUyVJA5NsocRCYTmU
@seanrivera @claude
fix(drop): over-reporting caveat on the doomed-detection planner + do…
a948247 
…c banner

I-1 from the final results-corpus review: gridworld/baselines.py's DROP
transition keeps a dropped key in collected_keys, so the planner can never
model re-acquiring it, while the runtime (custom_env.py:549) genuinely allows
re-pickup. That makes the model over-report doomed states in the
drop_available=True case (456 doomed states on the D2 spec, 304 false). No
current row is affected, but this must be revisited before scoring any
episode where the model actually drops and later retrieves a key. Comment
rewritten to say so plainly; behavior unchanged.

Also added a dated SUPERSEDED banner to docs/r1-kimi-rerun-package.md, a
pre-launch planning doc whose 3-arm launch has since completed — the
"PREPARED, NOT LAUNCHED" framing and the "excluded from R1 failure stats"
language are pre-launch state, not current.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123tBVPUyVJA5NsocRCYTmU
@seanrivera @claude
chore(release): drop editor swap + generated prompt dump; harden .git…
0ba31d9 
…ignore

The tracked .swp leaked username/hostname/home-path metadata; prompts.txt is
a regenerable preview dump with absolute local paths (regenerate via
multinet-preview-prompts). New ignore rules close every observed
accidental-commit vector (.claude/ and scratchpad/ were previously protected
only by machine-local excludes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
chore(release): remove non-runnable frontier-probe DRAFT fixtures
086b21a 
Self-declared NOT RUNNABLE; named speculative model ids and referenced an
analysis file that doesn't exist in this repo. Regenerable from selection
metadata if the probe is ever scheduled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
chore(release): move completed-campaign launch scripts to the results…
d2f5085 
… repo

launch_kimi_rerun.sh / run_qwen_backfill.sh hard-code a finished run's
worktree, run-id, and archive paths — ops residue, not reusable tooling.
Rerun fixtures stay for provenance. Copies live in
Multinet-v2-results/docs-archive/ops/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(mazes): single maze-corpus source of truth + corpus-wide goal inv…
16ad4d6 
…ariant

mazes/exp_maze_jsons/ was a stale byte-drifted duplicate of the ogbench
procgen corpus (195 identical / 9 drifted / 0 unique) and carried a live
maze.goal != goal.target violation in D3 that the D1-scoped assertion could
not catch. New corpus-wide invariant test proved the defect red before the
repoint; maze-family tests now read the corpus the R1 manifests actually
resolve, and the one-shot disjointness scan covers it too.

Known corpus defects found during the repoint are FROZEN, not fixed: the
submodule is the exact corpus the paid R1 runs resolved, so its files must
not change pre-release. tests/maze_test_utils.py's KNOWN_CORPUS_DEFECTS
registry pins each known-bad value (S4-vs-M goal desync from the 31a0549
half-fix; M2/D1/D2 metadata regressions from 271bf47) so any future change,
including the eventual fork-side fix, fails loudly. Runtime impact: none —
chain_pattern/difficulty_tier are unread labels and every file is
self-consistent on maze.goal == goal.target.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(submodule): track fix/d3-goal, the branch that contains the pin
b9329e3 
branch=master didn't contain the pinned commit, so 'git submodule update
--remote' silently reverted the d3/S4 goal-correctness fixes out of the live
R1 corpus. Merging fix/d3-goal into the fork's master and repinning is
post-release work (Appendix B).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(scripts): working console entry points, with a regression test
646f31e 
multinet-vlm-sanity pointed at a main() that didn't exist; three scripts
defaulted their task path under scripts/mazes/ (parent vs parents[1]). New
test loads every declared multinet-* entry point so this class of breakage
can't ship again.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
test: mark performance benchmarks slow and exclude by default
db700b5 
The hex step-throughput threshold went fail/fail/pass across three
same-machine runs — load-sensitive, so it can't sit inside the documented
install check. Opt back in with: pytest -m slow.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
build: honest packaging — v2.0.0, MIT metadata, no maze/fleet payload…
9bca8fb 
… in the wheel

pip show previously reported 0.1.0/UNKNOWN; the wheel shipped the eval-maze
package and GCP deploy tooling into site-packages; MANIFEST.in pruned a path
that never existed; the CLI still introduced itself as the 2026-Q1
"tests 1-3" milestone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(tests): repo-root conftest so unpackaged modules import in any py…
991d420 
…test run

Dropping deploy* from packaging (9bca8fb) removed `deploy` from the editable
finder map; eight test modules import it and full-suite collection survived
only because an alphabetically-early test inserts the repo root on sys.path.
A root conftest.py makes the path setup explicit and order-independent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
ci: run the suite on push/PR (submodules recursive, slow tests excluded)
f121cde 
1045 tests, zero automation until now — the broken entry point this branch
fixes is exactly the class of defect a 25-line workflow catches. Suite is
verified green without any API keys, so fork PRs are safe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs(release): triage the docs tree for public release
153ff25 
Archive 14 campaign/ops/stale docs to the results repo (cost projections,
runbooks, launch checklists, 261-test-era status snapshots, local-path
notes); rewrite docs/README.md as an honest index over the kept design
references; remove three image links to diagrams that never existed; reword
the analysis tombstone for a public audience; annotate the relocated
candidate-maze provenance paths in the R1 manifests.

Also drop test_future_directions_doc_flags_vllm, a hard file-existence
guard on the now-archived docs/future_directions.md (caught by the
full-suite run; the doc's relocation is intentional per this triage).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs(release): repair dangling pointers left by the docs archival
ffbabfe 
smoke_qwen's verdict message (and the test that pinned it) referenced the
archived future_directions doc; the kept two-tier design doc referenced two
archived siblings without saying where they went.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs: rewrite RUNME around the canonical run_pipeline stack
9e25266 
The old guide documented only the legacy run_eval/Ollama harness and
claimed 261 tests (actual: 1045). Canonical pipeline now leads: submodule
clone, green pytest, cheap smoke run, paid R1 reproduction with cost
warning, scoring, fleet rails. Legacy harness demoted to a labeled appendix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs: fix two legacy-appendix commands in RUNME (real CLI syntax)
0602115 
run_eval takes --benchmark tiers --tier N (not --tiers); play_task takes the
task file as a positional argument (not --task).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs(runme): smoke command passes the manifest guard; accurate R1 cap…
9b382c5 
…s; honest BYO-model steps

The documented smoke command omitted --manifest and died in
check_run_config_expectations; the R1 section claimed 64k caps for all three
models (Qwen tier is 8k phase-1); adding a model also requires a provider
branch in _build_agent_from_spec, not just a run-config entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
docs: public README — accurate quickstart, data-boundary note, acknow…
c3fb9c5 
…ledgments

Replaces the stub with the finalized draft: submodule-aware install, cheap
smoke quickstart with the paid R1 reproduction split out under a cost
warning, corrected structure table (analysis/ moved out; mazes/ vs ogbench
corpus), explicit eval-data note (panels public now, rotate at R3),
Limitations, OGBench/MiniGrid attribution, MIT license section. Remaining
HUMAN markers (logo, links, final headline numbers, BibTeX) gate the merge.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(release): fresh-clone test fix + honest pointers + CI wheel guard
5ce8c9e 
test_results_repo_is_gitignored only passed when the ignored results repo
existed on disk (dir-only gitignore pattern) — fresh clones and CI failed;
check-ignore now gets the trailing slash. Annotate the remaining comments
that pointed at relocated scripts, archived docs, and unpublished local
design notes. CI now asserts the wheel ships neither mazes/ nor deploy/.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
merge(main): join origin/main history — content already superseded
447304b 
origin/main's only commit not in this line is merge 325dc0b (PR #33,
text-summary-and-last3), whose branch tip 8423912 is already an ancestor
here; the feature line carried that work forward ~100 commits. -s ours
keeps this branch's tree wholesale and makes the release PR mergeable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
merge(main): integrate the playable-demo feature line
a9a5fd9 
Preserves PRs #39/#40 (demo/ package, enhanced play_task.py, watchdog and
checkpoint changes, expanded default manifest). Conflicts: RUNME.md kept
from the release rewrite (demo section re-added on top); the D3 maze kept
deleted — main's in-tree goal fix is already present in the canonical
ogbench submodule copy that replaced the duplicate corpus.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(demo): point the demo's manifest rows at the canonical maze corpu…
f351ae0 
…s + document the demo

The playable-demo rows added on main sourced the retired in-tree duplicate
corpus; the same files (canonical, goal-fixed) live in the ogbench
submodule the R1 manifests already resolve. RUNME gains the demo section
lost in the merge resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(demo): canonical corpus paths + launch without a results checkout
7d72a45 
The demo's default task and R1-folder hint referenced the retired duplicate
corpus; the R1-comparison table loader hard-required a sibling
Multinet-v2-results checkout and crashed play_task.py at startup without
one. It now also finds the nested layout and degrades to comparison-off
when no table exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(demo): non-R1 end card instead of a raise; ship demo/ in the whee…
0af8237 
…l; RUNME reconcile

Episode end on any task outside the R1 results table (the default on a
fresh clone) raised out of the render path and killed the window, losing
--record trajectories. play_task.py became a shim over demo/ but the wheel
didn't package demo/. RUNME's legacy appendix still advertised the
pre-demo play_task; SS3 gains a cost note about the expanded default
manifest catalog.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
merge(main): integrate PR #44 (demo_api)
169cbed 
demo/api subpackage + r1_config/r1_tasks modules from main. pyproject
resolved as the union of the release metadata/packaging trims and PR #44's
additions; demo/ui.py takes main's refactor with the release line's two
semantic fixes re-verified (no retired-corpus paths, no launch-blocking
raise without an R1 table).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
chore(release): remove internal-document references entirely
3c9085c 
Operator directive: internal docs (archived runbooks/campaign records,
local design notes, relocated analysis paths) must not be referenced from
public files at all — prior rounds only annotated them. Prose reworded to
stand alone; manifest selection provenance reduced to bare set names;
run-config notes keep only citations of docs that ship. Task rows, model
blocks, and experiment configs are byte-identical (episode-hash safety).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
chore(release): last archived-doc citation + backfill-launcher wording
98f1378 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
fix(mazes): re-pin ogbench with the corpus fixes; retire KNOWN_CORPUS…
d9da8e6 
…_DEFECTS

The fork now carries the (8,1) goal correction across the 10x10_dense_1
skeleton (mechanisms mandatory again) plus the restored sg-family
metadata, so the pinned-known-bad-value exemptions come out and every
family assertion runs strict. S4/D3 specs change vs what R1 ran (they ran
at the wrong-direction (8,8) goal, possibly mechanism-bypassable — an
analysis caveat, not a scoring error); cached episodes for changed tasks
invalidate by design via the task-spec hash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@seanrivera @claude
chore(release): make two private-results-repo references standalone
a077e4e 
- gridworld/baselines.py: drop the dangling RUN_NOTES_incidents_and_findings.md
  pointer from the DROP-model caveat comment (the caveat itself is already
  inlined; the cited file lives only in the private results repo)
- manifest.r1_kimi_rerun.json: reword selection.evidence from a results-repo
  CSV path to a standalone description (task rows untouched)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123tBVPUyVJA5NsocRCYTmU
@seanrivera @claude
chore(release): remove prototype-era orphans and retired campaign fix…
a6ad667 
…tures

Audit-verified zero-reference deletions (grep for importers, console scripts,
docs, dynamic filename construction in sweep_state.py):

- First-prototype-era code (2026-04/05, superseded by custom_env.py /
  run_pipeline.py / demo/): gridworld/demo.py, gridworld/envs/ (tier_envs),
  multigrid/demo.py, example_usage.py, interactive_demo.py,
  visualize_grid{,s_proper}.py, scripts/render_*_maze{,s}_jsons.py,
  scripts/render_ogbench_mazes.py, scripts/demo_baselines.py; pyproject
  py-modules pruned to match; backends doc tree updated
- Retired campaign fixtures (archived to the results repo docs-archive/
  fixtures-retired/ first): the pre-R1 ogbench_50 draft run-config, two
  superseded Qwen3.5-HF run-configs, the ctxfix backfill run-configs
  (context_window_{claude,kimi}_only) and their one-off kimi_missing
  manifests

Kept deliberately: all conditional_*_{claude_kimi,qwen}.json split configs
(sweep_state.py constructs these names per SWEEP_TOPO), the documented
legacy harness cluster (run_eval/evaluation_harness/model_interface/
adapters/gridworld runner+tasks), and frozen cross_domain/.

Suite: 1032 passed (was 1034; the two removed cases are
test_all_committed_manifest_sources_resolve params for the two deleted
diagnostic manifests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0123tBVPUyVJA5NsocRCYTmU
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@seanrivera @pranavguru