feat: add configurable output validation retry for output_pydantic#6598
Open
h1jj wants to merge 3 commits into
Open
feat: add configurable output validation retry for output_pydantic#6598h1jj wants to merge 3 commits into
h1jj wants to merge 3 commits into
Conversation
Add `max_retries` parameter to `convert_to_model` and `output_validation_max_retries` field to Task, enabling LLM-based retry when Pydantic validation of agent output fails. Each retry feeds the validation error back to the LLM so it can self-correct. Defaults to 0 (backward compatible, no behavior change). - converter.py: add `_handle_and_retry`, `_convert_with_retry` helpers (sync + async) with error accumulation across retry attempts - task.py: add `output_validation_max_retries` configuration field - tests: 4 new tests covering retry success, exhaustion, no-LLM mode, and error context injection Closes: crewAIInc#4389 (related: converter robustness)
📝 WalkthroughWalkthroughThe PR adds ChangesOutput validation retries
Sequence Diagram(s)sequenceDiagram
participant Task
participant convert_to_model
participant handle_partial_json
participant convert_with_instructions
Task->>convert_to_model: output and max_retries
convert_to_model->>handle_partial_json: parse and validate output
handle_partial_json-->>convert_to_model: validation result
convert_to_model->>convert_with_instructions: retry with validation error
convert_with_instructions-->>convert_to_model: corrected output or ConverterError
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/crewai/src/crewai/utilities/converter.py`:
- Around line 224-232: The retry loops currently skip conversion when
max_retries is zero. In lib/crewai/src/crewai/utilities/converter.py lines
224-232 and 558-566, update _convert_with_retry and _async_convert_with_retry so
their attempt ranges always execute at least once, while preserving additional
retries when configured.
- Around line 663-666: Update the async output-validation log in the converter
helper to use the same attempt and retry totals as the sync helper and injected
prompt: format the message with attempt and max_retries directly, removing both
+1 offsets while preserving the existing error context.
In `@lib/crewai/tests/utilities/test_converter.py`:
- Around line 1015-1110: Add coverage for the converter_cls dispatch path in the
retry tests, using a configured converter class rather than converter_cls=None.
Add a max_retries=0 case that invokes the converter and asserts the expected
conversion behavior, ensuring the direct _convert_with_retry path does not
incorrectly skip the converter; keep assertions focused on observable behavior
rather than internal calls.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 338b1dc5-c9f8-4827-94ba-442181e4c388
📒 Files selected for processing (3)
lib/crewai/src/crewai/task.pylib/crewai/src/crewai/utilities/converter.pylib/crewai/tests/utilities/test_converter.py
Address code review feedback from CodeRabbit: 1. (CRITICAL) Fix _convert_with_retry loop to always run at least one iteration when converter_cls is set. Changed range(1, max_retries+1) to range(1, max(max_retries, 1)+1) in both sync and async versions. 2. (MINOR) Align async log attempt numbering with sync format: attempt/max_retries instead of attempt+1/max_retries+1. 3. (TEST) Add test_converter_cls_path_still_called_with_zero_retries to cover the converter_cls dispatch path when max_retries=0.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pr-body.md`:
- Line 10: Update the test coverage summary and repeated test-plan entry in
pr-body.md to match the actual TestConvertToModelWithRetry tests, including the
converter_cls path with max_retries=0. Correct the reported count and explicitly
list all covered scenarios, including success, exhaustion, zero-retry
converter_cls behavior, and error context injection.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d579f656-edea-4dea-a932-1b1ba46d2d21
📒 Files selected for processing (3)
lib/crewai/src/crewai/utilities/converter.pylib/crewai/tests/utilities/test_converter.pypr-body.md
🚧 Files skipped from review as they are similar to previous changes (2)
- lib/crewai/tests/utilities/test_converter.py
- lib/crewai/src/crewai/utilities/converter.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add configurable LLM-based retry mechanism for output_pydantic validation.
Changes
Test Plan