[feat] Unify eval loops by junaway · Pull Request #4341 · Agenta-AI/agenta

junaway · 2026-05-15T14:41:35Z

No description provided.

coderabbitai · 2026-05-15T14:41:42Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 617ceddf-9be5-4061-af93-0d4a05cffa06

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/unified-eval-loops

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

vercel · 2026-05-15T14:41:42Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	May 22, 2026 4:04pm

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

CLAassistant · 2026-05-20T13:22:23Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ jp-agenta
✅ junaway
❌ github-actions[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…ture (web/api/sdk/custom) Document who-executes-a-step semantics: origin answers only "who runs this step in a slice", not who created/started/dispatched it. Covers the shipped human/auto/custom model (the auto/custom asymmetry, the execute_custom flag, the web's custom->"SDK" classification) and proposes the future web/api/sdk/custom model with its schema/data-migration/back-compat implications and the unified-runtime problem that blocks adopting it now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

github-actions · 2026-05-22T12:39:59Z

Railway Preview Environment


Preview URL	https://gateway-production-68fb.up.railway.app/w
Project	`agenta-oss-pr-4341`
Image tag	`pr-4341-bd4b6ce`
Status	Deployed
Railway logs	Open logs
Workflow logs	View workflow run
Updated at 2026-05-22T16:15:37.370Z

[release] v0.100.1

[docs] Update API reference docs

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

jp-agenta added 9 commits April 24, 2026 11:47

initial design

a114ab3

initial implementation

747502d

initial review

2220e65

fixed basic evals

0c559d8

Add evaluation parallelization

3862dfe

Parallelization checks

b958574

quick engine fix

5b647ee

ongoing debug

c4d8368

intermediate design extensions

e049ef5

Copilot AI review requested due to automatic review settings May 15, 2026 14:41

junaway changed the title ~~[feat] Unified eval loops~~ [feat] Unify eval loops May 15, 2026

Copilot AI reviewed May 15, 2026

View reviewed changes

junaway changed the base branch from main to release/v0.99.9 May 15, 2026 14:41

Merge release/v0.99.9 and evals<>queues work

50d1d8e

vercel Bot deployed to Preview May 15, 2026 15:52 View deployment

evals<>queues implementation

603820f

Copilot AI review requested due to automatic review settings May 15, 2026 16:32

Copilot AI reviewed May 15, 2026

View reviewed changes

vercel Bot deployed to Preview May 15, 2026 16:33 View deployment

latest findings

3ebe721

vercel Bot deployed to Preview May 20, 2026 13:07 View deployment

Merge release/v0.100.1

f68e42a

Copilot AI review requested due to automatic review settings May 20, 2026 13:20

Copilot AI reviewed May 20, 2026

View reviewed changes

vercel Bot deployed to Preview May 20, 2026 13:22 View deployment

junaway changed the base branch from release/v0.99.9 to release/v0.100.1 May 20, 2026 13:22

Add missing sdk files

9b388bd

vercel Bot deployed to Preview May 22, 2026 10:21 View deployment

clean up logs and auto/custom/human

76a9833

vercel Bot deployed to Preview May 22, 2026 10:54 View deployment

jp-agenta and others added 2 commits May 22, 2026 12:57

bump py deps

e9ff86d

Copilot AI review requested due to automatic review settings May 22, 2026 10:57

Copilot AI reviewed May 22, 2026

View reviewed changes

vercel Bot deployed to Preview May 22, 2026 10:59 View deployment

fixing tensor

423800e

vercel Bot deployed to Preview May 22, 2026 11:30 View deployment

split refresh metrics from process slice

30f53c8

Copilot AI review requested due to automatic review settings May 22, 2026 12:26

Copilot AI reviewed May 22, 2026

View reviewed changes

vercel Bot deployed to Preview May 22, 2026 12:27 View deployment

junaway marked this pull request as ready for review May 22, 2026 12:28

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. Backend refactoring A code change that neither fixes a bug nor adds a feature labels May 22, 2026

add live evaluation tests

92d733e

vercel Bot deployed to Preview May 22, 2026 13:06 View deployment

junaway and others added 6 commits May 22, 2026 17:51

Merge pull request #4383 from Agenta-AI/release/v0.100.1

476f82a

[release] v0.100.1

docs(api): update API reference from production OpenAPI spec

d50a898

v0.100.2

06d3a42

Merge pull request #4402 from Agenta-AI/automated/update-api-docs

3c9b1db

[docs] Update API reference docs

Merge branch 'main' into release/v0.100.2

5a548d3

Merge release/v0.100.2

ae5fc5a

Copilot AI review requested due to automatic review settings May 22, 2026 16:02

Copilot AI reviewed May 22, 2026

View reviewed changes

vercel Bot deployed to Preview May 22, 2026 16:04 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Unify eval loops#4341

[feat] Unify eval loops#4341
junaway wants to merge 55 commits into
release/v0.100.1from
feat/unified-eval-loops

junaway commented May 15, 2026

Uh oh!

coderabbitai Bot commented May 15, 2026 •

edited

Loading

Review skipped

Uh oh!

vercel Bot commented May 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

CLAassistant commented May 20, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented May 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

junaway commented May 15, 2026

Uh oh!

coderabbitai Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

vercel Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

CLAassistant commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Railway Preview Environment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coderabbitai Bot commented May 15, 2026 •

edited

Loading

vercel Bot commented May 15, 2026 •

edited

Loading

CLAassistant commented May 20, 2026 •

edited

Loading

github-actions Bot commented May 22, 2026 •

edited

Loading