test(eve): consolidate overlapping coverage by chadhietala · Pull Request #3296 · vercel/eve

chadhietala · 2026-09-11T19:36:27Z

Summary

The integration, scenario, and e2e suites repeated identical setup and execution paths across separate cases, increasing runtime and maintenance without adding independent coverage. This consolidates those overlaps by carrying every prior assertion into shared workflows while preserving distinct inputs, negative checks, event ordering, platform gates, model tags, transport boundaries, and regression intent. The result removes 78 test cases: 41 integration cases, 25 scenario cases, and 12 fixture evals, with no product behavior changes.

Validation

fnm exec --using=24.14.0 pnpm --filter eve build
fnm exec --using=24.14.0 pnpm --filter eve exec vitest run --config vitest.unit.config.ts --maxWorkers=4 (796 files, 8,699 passed, 1 skipped)
fnm exec --using=24.14.0 pnpm --filter eve exec vitest run --config vitest.integration.config.ts --maxWorkers=4 (103 files, 744 passed)
fnm exec --using=24.14.0 pnpm typecheck (45 tasks passed)
fnm exec --using=24.14.0 pnpm guard:invariants
fnm exec --using=24.14.0 pnpm exec oxfmt --check <changed TypeScript files>
fnm exec --using=24.14.0 pnpm exec oxlint <changed TypeScript files>
Affected e2e fixture TypeScript checks for agent-basic-runtime, agent-tools, agent-tools-hitl, agent-tools-sandbox, agent-skills, agent-subagents, and agent-schedules
Consolidated scenario files: 10 files, 65 passed, 10 environment-gated skips
fnm exec --using=24.14.0 pnpm --filter eve test:scenario (92 files and 472 tests passed; 13 environment-gated skips; one untouched auth scenario timed out)
Focused rerun of src/setup/flows/chatgpt-auth.scenario.test.ts (3 passed; the same declined-sign-in test timed out)

An initial scenario invocation ran in parallel with the workspace typecheck and failed during global setup because that command rebuilt packages/eve/dist while the scenario tarball was being packed. The package was rebuilt and the scenario suite was rerun alone against a stable dist tree.

The rerun reached every scenario file. The only failure was the unchanged native ChatGPT browser login > rejects a declined sign-in without leaking the provider error, whose local callback fetch closed with ECONNRESET/UND_ERR_SOCKET and timed out at 120 seconds both in the full suite and alone. E2E eval execution remains CI-only.

Checklist

This change was requested or approved by a maintainer
I ran the relevant checks from CONTRIBUTING.md
I added tests and documentation where relevant
I added a changeset if this touches the published eve package
DCO sign-off passes for every commit (git commit --signoff)

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>

vercel · 2026-09-11T19:36:32Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
eve-docs	Ready	Preview, v0	Sep 11, 2026 7:38pm UTC
eve-docs-4759	Ready	Preview, v0	Sep 11, 2026 7:38pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
eve-pkg	Skipped	v0	Sep 11, 2026 7:38pm UTC

github-actions · 2026-09-11T19:40:23Z

Bundle + Package Summary: `apps/fixtures/weather-agent`

Key takeaways

No notable deltas vs main (109a699).

Delta vs `main (109a699)`

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	8.76 MB	8.76 MB	-358 B ✅
Package	Unpacked publish size	33.01 MB	33.01 MB	0 B ➖
Package	Installed footprint	78.55 MB	78.55 MB	0 B ➖
Package	Published files	3945	3945	0
Package	Installed files	7866	7866	0
Package	Installed package instances	33	33	0
Package	Distinct installed package names	32	32	0
Package	Installed dependency edges	51	51	0
Package	Installed optional peer edges	8	8	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	21.11 MB	21.11 MB	-80 B ✅
Runtime	Public routes	18	18	0

Changed function payloads vs main (109a699) (2)

Function	Status	Baseline	Current	Delta	Route changes
`functions/__server.func`	changed	10.56 MB	10.56 MB	-40 B ✅	none
`functions/.well-known/workflow/v1/flow.func`	changed	10.56 MB	10.56 MB	-40 B ✅	none

`eve init` install

Metric	Baseline	Current	Delta
Installed footprint	116.66 MB	116.66 MB	0 B ➖
Installed packages	113	113	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	49.24 MB	49.24 MB	0 B ➖
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

Preset: vercel
Nitro: nitro@3.0.260903-beta
Output directory: apps/fixtures/weather-agent/.vercel/output
Build metadata timestamp: 2026-09-11T19:39:25.429Z
Route aliases: 18 public, 1 internal (19 total aliases)
Vercel routes in config: 21
Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

Package: eve@0.54.2
Package directory: packages/eve
Tarball: 8.76 MB (eve-0.54.2.tgz)
Unpacked payload: 33.01 MB across 3945 published files
Installed footprint: 78.55 MB across 7866 installed files
Installed root package: 31.58 MB
Installed dependencies: 46.97 MB
Installed package instances: 33
Distinct installed package names: 32
Installed dependency edges: 51
Installed optional peer edges: 8
Runtime dependencies: 2
Peer dependencies: 6 (5 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.
Graph metrics read only package.json files in package directories directly beneath a node_modules boundary, including nested boundaries. Each directory is one package instance; distinct names come from those manifests. Dependency edges count each unique name in dependencies or optionalDependencies per instance; optional peer edges count peerDependencies marked optional.

Heavy installed dependencies

eve: 31.58 MB (40.2%)
@rolldown/binding-linux-x64-gnu: 19.31 MB (24.6%)
ai: 7.12 MB (9.1%)
zod: 6.74 MB (8.6%)
undici: 3.51 MB (4.5%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [#############...........] 9.76 MB 29.6%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [###.....................] 2.27 MB 6.9%
🟠 dist/src/compiled/@ai-sdk/code-mode/index.js     [#.......................] 1.03 MB 3.1%
🟡 dist/src/compiled/@vercel/blob/index.js          [#.......................] 901.4 kB 2.7%
🟡 dist/src/compiled/_chunks/workflow/signal-exi... [#.......................] 514.6 kB 1.6%
🔴 Other published files                            [########################] 18.54 MB 56.2%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 31.58 MB 40.2%
🔴 @rolldown/binding-linux-x64-gnu [###############.........] 19.31 MB 24.6%
🔴 ai                              [#####...................] 7.12 MB 9.1%
🔴 zod                             [#####...................] 6.74 MB 8.6%
🟠 undici                          [###.....................] 3.51 MB 4.5%
🟠 nitro                           [#.......................] 1.89 MB 2.4%
🔴 Other installed packages        [######..................] 8.39 MB 10.7%

Runtime dependencies (2)

Package	Range	Notes
`nitro`	`3.0.260903-beta`
`undici`	`8.9.0`

Peer dependencies (6)

Package	Range	Notes
`@opentelemetry/api`	`^1.0.0`	optional peer
`ai`	`catalog:`
`braintrust`	`^3.0.0`	optional peer
`dd-trace`	`^6.13.0`	optional peer
`just-bash`	`^3.1.0`	optional peer
`microsandbox`	`^0.5.0`	optional peer

eve init install drill-down

`eve init` install details

Command: eve init my-agent
Package manager: npm
Installed footprint: 116.66 MB across 9750 installed files
Installed packages: 113 total (107 transitive-only)
dependencies: 4 direct packages totaling 49.24 MB
devDependencies: 2 direct packages totaling 5.04 MB
Other transitive package files: 62.37 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

eve: 31.58 MB (27.1%)
@typescript/typescript-linux-x64: 27.95 MB (24.0%)
@rolldown/binding-linux-x64-gnu: 19.31 MB (16.6%)
zod: 10.37 MB (8.9%)
ai: 7.12 MB (6.1%)

Installed footprint breakdown

Installed package size
🔴 eve                              [########################] 31.58 MB 27.1%
🔴 @typescript/typescript-linux-x64 [#####################...] 27.95 MB 24.0%
🔴 @rolldown/binding-linux-x64-gnu  [###############.........] 19.31 MB 16.6%
🔴 zod                              [########................] 10.37 MB 8.9%
🔴 ai                               [#####...................] 7.12 MB 6.1%
🟠 undici                           [###.....................] 3.51 MB 3.0%
🔴 Other installed packages         [#############...........] 16.81 MB 14.4%

dependencies (4)

Package	Range	Installed size	Share
`@vercel/connect`	`1.0.0`	167.9 kB	0.1%
`ai`	`^7.0.93`	7.12 MB	6.1%
`eve`	`file:eve-0.54.2.tgz`	31.58 MB	27.1%
`zod`	`4.5.4`	10.37 MB	8.9%

devDependencies (2)

Package	Range	Installed size	Share
`@types/node`	`24.x`	2.54 MB	2.2%
`typescript`	`7.0.2`	2.50 MB	2.1%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 10.56 MB 50.0%
🔴 functions/__server.func                         [########################] 10.56 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 10.56 MB

Metric	Value
Public routes	`/.well-known/workflow/v1/flow`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	10.56 MB
Function files	10.56 MB across 114 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.38 MB (22.6%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.38 MB 22.6%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 699.5 kB 6.6%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.8%
🔴 Other bundled files              [########################] 5.16 MB 48.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 17 public routes, 1 internal alias • 10.56 MB

Metric	Value
Public routes	`/` `/.well-known/workflow/v1/webhook/[token]` `/eve/v1/activity/[token]` `/eve/v1/callback/[token]` `/eve/v1/connections/[name]/callback/[attemptId]/[token]` `/eve/v1/connections/[name]/callback/[token]` `/eve/v1/health` `/eve/v1/info` `/eve/v1/session` `/eve/v1/session/[parentSessionId]/subagents/[callId]/[childSessionId]/stream` `/eve/v1/session/[sessionId]` `/eve/v1/session/[sessionId]/cancel` `/eve/v1/session/[sessionId]/clear` `/eve/v1/session/[sessionId]/compact` `/eve/v1/session/[sessionId]/reset` `/eve/v1/session/[sessionId]/stream` `/eve/v1/task-input/[token]`
Internal aliases	`/__server`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	10.56 MB
Function files	10.56 MB across 114 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.38 MB (22.6%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                        [###########.............] 2.38 MB 22.6%
🟡 _libs/undici.mjs                 [#####...................] 980.8 kB 9.3%
🟡 _chunks/esm-Fqlolk7e.mjs         [###.....................] 723.3 kB 6.9%
🟡 _chunks/chatgpt-model.mjs        [###.....................] 699.5 kB 6.6%
🟡 _chunks/signal-exit-Dsy-TT0V.mjs [###.....................] 616.2 kB 5.8%
🔴 Other bundled files              [########################] 5.16 MB 48.8%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: `e2e/fixtures/agent-tools-sandbox`

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

Build pipeline: 6.12 s -> 6.09 s (-34.8 ms) vs main (109a699).
Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (109a699)`

Phase	Baseline	Current	Delta
`extension.check`	0.8 ms	0.8 ms	0.0 ms
`project.resolve`	0.3 ms	0.4 ms	+0.1 ms
`workspace.create`	0.5 ms	0.6 ms	+0.1 ms
`host.prepare`	529.3 ms	518.5 ms	-10.8 ms
`vercel.service-prefix.resolve`	2.2 ms	2.5 ms	+0.3 ms
`nitro.create`	612.0 ms	592.7 ms	-19.3 ms
`sandbox.prewarm`	397.3 ms	420.5 ms	+23.2 ms
`nitro.cache.prepare`	0.2 ms	0.3 ms	+0.1 ms
`nitro.prepare`	0.8 ms	0.9 ms	+0.1 ms
`nitro.public-assets`	0.7 ms	0.8 ms	+0.1 ms
`nitro.prerender`	0.6 ms	0.5 ms	-0.1 ms
`nitro.bundle`	4.52 s	4.49 s	-29.9 ms
`nitro.cache.write`	0.4 ms	0.4 ms	0.0 ms
`vercel.workflow-function.materialize`	50.2 ms	51.6 ms	+1.4 ms
`agent-summary.emit`	0.7 ms	0.7 ms	0.0 ms
`nitro.close`	0.2 ms	0.1 ms	-0.1 ms
`output.publish`	3.6 ms	3.6 ms	0.0 ms
`workspace.remove`	2.2 ms	2.5 ms	+0.3 ms

e0-gh-vercel-connect · 2026-09-11T19:49:10Z

This session has hit the input-token limit (40M) per session. This is a guardrail against defective long-running sessions. If session activity looks fine, just approve to keep going.

Approve - Grant a fresh token budget
Stop - Stop now

Answer by mentioning me in a reply, e.g. @e0 Approve.

test(eve): consolidate overlapping coverage

0749206

Signed-off-by: Chad Hietala <chad.hietala@vercel.com>

vercel Bot deployed to Preview – eve-docs-4759 September 11, 2026 19:37 View deployment

vercel Bot deployed to Preview – eve-docs September 11, 2026 19:38 View deployment

Navigation Menu

Uh oh!

test(eve): consolidate overlapping coverage - #3296

test(eve): consolidate overlapping coverage#3296
chadhietala wants to merge 1 commit into
mainfrom
chad/consolidate-test-coverage

chadhietala commented Sep 11, 2026

Uh oh!

vercel Bot commented Sep 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 11, 2026

Package Details

`eve init` install details

Payload Size Graph

Top Function Payloads

Uh oh!

e0-gh-vercel-connect Bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

chadhietala commented Sep 11, 2026

Summary

Validation

Checklist

Uh oh!

vercel Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 11, 2026

Bundle + Package Summary: apps/fixtures/weather-agent

Delta vs main (109a699)

eve init install

Package Details

eve init install details

Payload Size Graph

Top Function Payloads

Build Timing: e2e/fixtures/agent-tools-sandbox

Uh oh!

e0-gh-vercel-connect Bot commented Sep 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 11, 2026 •

edited

Loading

Bundle + Package Summary: `apps/fixtures/weather-agent`

Delta vs `main (109a699)`

`eve init` install

`eve init` install details

Build Timing: `e2e/fixtures/agent-tools-sandbox`