Skip to content

Add eval fixture validation CLI (CR-122)#96

Merged
coreytshaffer merged 1 commit into
mainfrom
codex/cr-122-eval-fixture-validation-cli
Jul 10, 2026
Merged

Add eval fixture validation CLI (CR-122)#96
coreytshaffer merged 1 commit into
mainfrom
codex/cr-122-eval-fixture-validation-cli

Conversation

@coreytshaffer

Copy link
Copy Markdown
Owner

Scope

Adds a tc eval validate-fixtures --input <path> command that wraps the CR-121 pure JSONL fixture validator.

Included

  • CLI success and fail-closed handling for eval fixture validation
  • Line-aware diagnostic output for malformed or contract-invalid JSONL
  • Missing-file/read-error handling without invoking eval execution
  • Synthetic CLI coverage and fixture-contract documentation updates

Excluded

No tc eval scoring changes, model/backend calls, routing/admission integration, ledger writes, probe execution, or adversarial/tampering expansion.

Validation

  • python -m pytest -q tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py -> 15 passed
  • python -m pytest -q tests/test_eval_fixture_cli.py tests/test_eval_fixture_validator.py tests/test_tc_cli.py tests/test_eval_review_cli.py -> 37 passed
  • python -m pytest -q -> 931 passed, 2 skipped
  • git diff --check clean
  • git show --check HEAD clean

@coreytshaffer coreytshaffer marked this pull request as ready for review July 10, 2026 08:15
@coreytshaffer coreytshaffer merged commit 4235124 into main Jul 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant