test: prune redundant integration tests and add unused-cassette checker - #802
Merged
Abhijeet Prasad (AbhiPrasad) merged 11 commits intoSep 24, 2026
Merged
Conversation
…stral, cohere, huggingface_hub, instructor) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd util tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…assettes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…llamaindex, dspy, strands) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records which cassette files tests actually open (via an audit hook enabled by BRAINTRUST_CASSETTE_USAGE_DIR) and reports files no session reads. Complements check-stale-cassettes.py, which only checks whole version dirs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions SQLAlchemy 2.1.0 ships only an sdist whose pyproject declares a duplicate normalized extra (mssql-pymssql), which uv refuses to build. Installing dspy (via optuna), google-adk and llama-index-core resolves it and fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nox always reads the per-version cassette directories; these base-dir copies are never loaded (found with check-unused-cassettes.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: eeaf047e8e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
- reuse find_spans_by_type / assert_metrics_are_valid / _find_agent_span instead of inline copies - share one _method_refs builder in mistral tests; trim unused helper params - drop the redundant install() in test_utils (braintrust.conftest covers it) - load the recorder by path in its tests (no full SDK import per subprocess) - checker: --reuse-venv flag, keep nox reports in the usage dir Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ttes run A relative --usage-dir was resolved differently by test processes that run from another cwd (e.g. pipecat's run_from_temp_dir), so nothing was recorded and every cassette looked unused. Reusing a usage dir also merged stale reads from earlier runs, hiding cassettes the current tests no longer open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 541291bb78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
…ettes A session can succeed while platform- or version-specific tests skip (e.g. cursor's sync-bridge tests on Windows), leaving their cassettes unread. Tests now log skips alongside cassette reads; unread files in a version directory whose session skipped anything are listed with the skip reasons for review instead of being reported as deletable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abhijeet Prasad (AbhiPrasad)
deleted the
chore/prune-integration-tests
branch
September 24, 2026 21:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR removes low-value and redundant integration tests, adds a checker that finds cassette files no test reads, and pins
sqlalchemy<2.1so three nox sessions install again.1. Prune redundant integration tests
integrations/, pluswrappers/test_langsmith_wrapper.py.auto_instrument()scripts.latestbut cover a code path specific to an older version. For example, anthropic's streaming tests are the only coverage of the cache-token fallback on 0.48.0.latestacross the affected sessions (older matrix versions drop the same tests).pydantic_ai/cassettes/{0.1.9,1.0.1}that no session loads. Nothing was re-recorded.2.
scripts/check-unused-cassettes.pycheck-stale-cassettes.pyonly catches whole version directories that fall out of the matrix. This catches individual files inside valid directories: cassettes left behind by deleted or renamed tests, and cassettes for tests that always skip at a given version.src/braintrust/_test_cassette_usage.pyrecords every cassette file tests open. It is active only whenBRAINTRUST_CASSETTE_USAGE_DIRis set.run <integrations|--all>runs every nox session that reads those cassettes, at every matrix version withCI=1, then reports files nothing opened.--cleandeletes them.make check-unused-cassettes INTEGRATIONS="…"and a section inAGENTS.md.mainfor agentscope, pydantic_ai and instructor, it found exactly the 89 dead cassettes found by hand in part 1.huggingface_hub/cassettes/*.yamlleftovers, deleted in this PR, and nothing else.3. Pin
sqlalchemy<2.1for the dspy, google-adk and llama-index sessionsmssql-pymssql), which uv refuses to build.test_dspy(2.6.0),test_google_adk(1.14.1)and bothtest_llamaindexversions failed to install.test-sqlalchemy-2-0constraint group, passed through the existing_install_matrix_dep(..., constraint_group)mechanism.uv.lockdiff adds the new group; the only other changes are regenerated environment markers, and no locked package versions changed.Test plan
CI=1(replay-only VCR). CrewAI, LiveKit and Pipecat were run under Python 3.13, since they skip on 3.14.src/braintrust/test_cassette_usage.py: tests for the recorder and checker logic. Deliberately breaking the recorder makes them fail.ruff check,ruff format --checkandpylintcheck-unused-cassettes.py run --all: every integration reports no unused cassettes.Notes
wrappers/test_langsmith_wrapper.pyisn't run by any nox session or CI job:test_coreignoresbraintrust/wrappers. This PR doesn't change that.🤖 Generated with Claude Code
Co-authored by StarfolkAI (@starfolkai)[bot]