feat(harbor): upload standard verifier output - #716
Closed
David Leen (dleen) wants to merge 2 commits into
Closed
David Leen (dleen) wants to merge 2 commits into
David Leen (dleen) wants to merge 2 commits into
Conversation
David Leen (dleen)
force-pushed
the
codex/harbor-verifier-evidence
branch
from
August 28, 2026 01:45
02ccbe3 to
9e75b9f
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02ccbe3675
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
David Leen (dleen)
force-pushed
the
codex/harbor-verifier-evidence
branch
from
August 28, 2026 19:12
9e75b9f to
749f7e5
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Member
|
opened #724 as a follow up, will close this. Will credit you on that one David Leen (@dleen) - thanks for the PR! |
Abhijeet Prasad (AbhiPrasad)
added a commit
that referenced
this pull request
Sep 1, 2026
Supersedes #716 ## AI Summary Braintrust already received the reward numbers for each Harbor trial. Now it also receives the evidence the verifier wrote, so you can see *why* a trial failed without leaving the UI. ``` <trial-dir>/verifier/ ├── reward-details.json already uploaded ├── ctrf.json NEW standard test-report JSON ├── test-stdout.txt NEW raw verifier log └── test-stderr.txt NEW raw verifier log ``` Verifier evidence shows up in two places: a bounded inline preview and one complete normalized `verifier-output.json` attachment. ``` root (experiment row) ├── task │ └── verification ← verifier_output_summary (inline preview) │ verifier_output (full evidence, attached) └── scorer: reward ← same two fields ``` ## Attachment modes Harbor runs are evaluation environments, where complete failure evidence is generally more useful than omitting raw logs. The default therefore captures all standard verifier output. | `attachments=` | `ctrf.json` | `test-stdout/stderr.txt` | task artifacts | |---|:---:|:---:|:---:| | `none` | — | — | — | | `structured` | ✅ | — | — | | `all` *(default)* | ✅ | ✅ | ✅ (with `artifact_include`) | `structured` is the JSON-only tier for environments that should not upload raw verifier logs. Task artifacts remain opt-in even under `all`: they require an explicit `artifact_include` glob. Structured redaction matches sensitive **key names**, while plain text has no keys: ```python # ctrf.json — the key is named ANTHROPIC_API_KEY, so the value is caught {"env": {"ANTHROPIC_API_KEY": "[REDACTED]"}} # test-stdout.txt — plain text is covered only by configured patterns "+ export ANTHROPIC_API_KEY=sk-ant-REAL-SECRET" ``` If an eval environment contains sensitive values, configure `redact_patterns` or use `attachments="structured"`. `redact_patterns` is the only redaction applied to raw verifier logs and is empty by default. ## Size limits are gone There used to be a 5 MB per-file cap and a 20 MB per-trial cap. The per-trial cap was one shared pot, spent in source order: ``` artifacts ──► trajectory images ──► verifier output ──► reward details ^ spends first ^ often got nothing left ``` That meant a few screenshots could push out the score evidence, and the same trial with a slightly larger screenshot could upload different evidence. Attachments are intended for large payloads, so both attachment caps and their shared-budget plumbing are removed and every accepted file uploads whole. `trajectory.json` keeps a separate limit (`max_trajectory_bytes`, 20 MB) because it is parsed in-process rather than treated as an attachment. ## Configuration changes - `attachments="all"` is now the default. - `attachments="verifier-details"` is renamed to `attachments="structured"`. - `max_attachment_bytes` and `max_total_attachment_bytes` are removed and no longer impose an upload ceiling. --------- Co-authored-by: David Leen <dleen@netflix.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
|
released with https://github.com/braintrustdata/braintrust-sdk-python/releases/tag/py-sdk-v0.36.0. Thanks for the PR! Will update docs in a bit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complexity
Testing