Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![e0-gh-vercel-connect[bot]](https://avatars.githubusercontent.com/b/10660?s=60&v=4){kind=small}
| "A root model routes a persistent capability request to self-mod, which creates a callable eve tool.", | ||
|
|
||
| async test(t) { | ||
| if (process.env.EVE_E2E_MODEL !== "openai/gpt-5.6-sol") { |
There was a problem hiding this comment.
[P2] Run this eval for the fixture’s default model
test:e2e invokes eve eval --strict without setting EVE_E2E_MODEL, while e2eModel() resolves an unset value to openai/gpt-5.6-sol. This raw comparison therefore skips the only eval that authors, rebuilds, and invokes the new tool during the normal fixture command, even though the agent is using the intended model. Normalize the unset value to the shared default, or reuse the shared model resolver before applying this guard.
| try { | ||
| await test(harness); | ||
| } finally { | ||
| await harness.reset(); |
There was a problem hiding this comment.
[P2] Stop background work before restoring source
The parent launches self-modification as a background task, but this finally block restores /source without resetting the parent session. If request() fails while the child is still running, the child can write after this reset and rebuild, contaminating the next serial eval. Normal eval errors do not call manager.cleanup(); the existing background-steering eval resets its parent session in finally because reset owns admitted tasks. Retain the parent session ID and reset it before restoring the source snapshot and rebuilding.
| @@ -0,0 +1,49 @@ | |||
| import assert from "node:assert/strict"; | |||
There was a problem hiding this comment.
[P3] Wire this test into CI
This node:test file is not reached by pnpm test:unit: that command runs package-level Turbo tasks, and .github is not a workspace package. The e2e workflows execute the discovery script, but none invokes this test file. These four world-selection assertions can therefore regress without failing CI. Add an explicit node --test .github/scripts/discover-e2e-fixtures.test.mjs step or include it in a checked-in test command.
Signed-off-by: benpankow <ben.pankow@vercel.com>
Signed-off-by: benpankow <ben.pankow@vercel.com>
9c92963 to
1ec4159
Compare
1 participant
Summary
Adds a dedicated self-modification eval fixture and shared harness for following the background child, rebuilding edited source, and restoring the fixture afterward. The self-mod child stays fixed on Claude Sonnet 5 while routing runs across the normal root-model matrix plus optional models from the authoring benchmarks; the slower tool-authoring flow runs once. Local-only fixtures can now opt out of world matrices so this does not add empty Vercel and Postgres legs.
Depends on #3278.
Validation
The routing and tool-authoring evals passed locally with
openai/gpt-5.6-sol; routing also passed withanthropic/claude-opus-5. Adds focused tests for fixture world selection.🤖