Refresh Codex instructions on exec boot and record eval diagnostics - #526
Merged
TheGreatAxios merged 2 commits intoAug 22, 2026
Merged
Conversation
Exec now awaits the same refreshCodexInstructions() the TUI uses, best-effort before first Codex inference, so capability/public SWE evals stop running on stale cached or bundled instructions. eval-capability.ts also stamps each result with a Codex instructions hash, the advertised tool list, and the requested reasoning effort for easier failure triage. Fixes CL-6693 https://linear.app/abklabs/issue/CL-6693
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Exec now awaits refreshCodexInstructions() before first Codex inference, the same shared function the TUI already uses, so headless exec (and thus capability/public SWE evals) no longer runs on stale cached or bundled instructions. The refresh is best-effort: a network failure falls back to the disk cache or bundled copy without failing the run.
eval-capability.ts also stamps each per-cell result with diagnostics: a Codex instructions hash, the advertised built-in tool list, and the requested reasoning effort, to help triage eval failures.
Fixes CL-6693
https://linear.app/abklabs/issue/CL-6693