feat: add HauntExtractTool (web extraction with honest failure codes)#6610
feat: add HauntExtractTool (web extraction with honest failure codes)#6610Darko893 wants to merge 2 commits into
Conversation
📝 WalkthroughWalkthroughThe PR adds ChangesHaunt extraction tool
Sequence Diagram(s)sequenceDiagram
participant Client
participant HauntExtractTool
participant HauntAPI
Client->>HauntExtractTool: run(url, prompt)
HauntExtractTool->>HauntAPI: POST /v1/extract with URL and prompt
HauntAPI-->>HauntExtractTool: extraction payload or HTTP error
HauntExtractTool-->>Client: extracted content or ValueError
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py`:
- Around line 35-43: The HauntExtractTool.api_key field is currently serialized
by Pydantic. Remove it from the public model fields by storing the credential in
a PrivateAttr or SecretStr, update HauntExtractTool.__init__ to initialize that
private/secret value from the argument or HAUNT_API_KEY, and add a regression
test verifying model_dump()/JSON output never contains the API key.
- Around line 54-76: Update the response handling in the extraction method
around requests.post so every non-2xx status is rejected before normal success
parsing. Decode the response body defensively to tolerate plain-text or HTML
errors, use any available error details when raising the API error, and reserve
the existing success-false JSON result for 2xx responses only.
In `@lib/crewai-tools/tests/tools/haunt_extract_tool_test.py`:
- Around line 33-43: Update test_haunt_extract_markdown_unwrapped to capture the
mocked requests.post call as mock_post, then assert its request payload includes
response_format set to "markdown". Keep the existing response and result
assertion unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 58ddca15-92d3-4bfb-ab8f-7808c4c87d85
📒 Files selected for processing (6)
lib/crewai-tools/src/crewai_tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/__init__.pylib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/README.mdlib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/__init__.pylib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.pylib/crewai-tools/tests/tools/haunt_extract_tool_test.py
| response = requests.post( | ||
| f"{self.base_url}/v1/extract", | ||
| headers={"X-API-Key": self.api_key, "Content-Type": "application/json"}, | ||
| json=body, | ||
| timeout=self.timeout, | ||
| ) | ||
| data = response.json() | ||
| if response.status_code in (401, 402, 429): | ||
| raise ValueError( | ||
| data.get("message") | ||
| or data.get("error") | ||
| or f"Haunt API error {response.status_code}" | ||
| ) | ||
| if not data.get("success"): | ||
| # Honest failure: give the agent a reason code, never invented content. | ||
| return json.dumps( | ||
| { | ||
| "error_code": data.get("error_code") or "extraction_failed", | ||
| "message": data.get("message") | ||
| or data.get("error") | ||
| or "extraction failed", | ||
| } | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
path = Path('lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py')
print(f'FILE {path} lines={sum(1 for _ in path.open())}')
with path.open() as f:
for i, line in enumerate(f, 1):
if 1 <= i <= 180:
print(f"{i:4d}: {line.rstrip()}")
PYRepository: crewAIInc/crewAI
Length of output: 3727
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find related tests/usages that show how HTTP errors are expected to behave.
rg -n "haunt_extract_tool|success: false|error_code|JSONDecodeError|status_code in \(401, 402, 429\)|/v1/extract" lib/crewai-tools -SRepository: crewAIInc/crewAI
Length of output: 4490
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Pull the Haunt API docs page referenced in the review comment, if reachable.
python3 - <<'PY'
import urllib.request
url = "https://hauntapi.com/docs"
try:
with urllib.request.urlopen(url, timeout=15) as r:
print("STATUS", r.status)
print(r.read(4000).decode("utf-8", "replace"))
except Exception as e:
print(type(e).__name__, e)
PYRepository: crewAIInc/crewAI
Length of output: 297
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for path in [
Path('lib/crewai-tools/tests/tools/haunt_extract_tool_test.py'),
Path('lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/README.md'),
]:
print(f'\n### {path} ###')
with path.open() as f:
lines = f.readlines()
print(f'lines={len(lines)}')
for i, line in enumerate(lines, 1):
if path.name == 'haunt_extract_tool_test.py' and 1 <= i <= 220:
print(f"{i:4d}: {line.rstrip()}")
elif path.name == 'README.md' and 1 <= i <= 220:
print(f"{i:4d}: {line.rstrip()}")
PYRepository: crewAIInc/crewAI
Length of output: 5173
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
path = Path('lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py')
# Simple behavioral probe of the current branching logic using representative payload shapes.
def classify(status_code, payload):
if status_code in (401, 402, 429):
return "raise"
if not payload.get("success"):
return "extraction_failed"
return "ok"
cases = [
(400, {"message": "bad request"}),
(500, {"error": "oops"}),
(500, "<non-json>"),
]
for status, payload in cases:
if payload == "<non-json>":
print(status, "would JSONDecodeError before branching")
else:
print(status, json.dumps(payload), "=>", classify(status, payload))
PYRepository: crewAIInc/crewAI
Length of output: 292
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
targets = [
'lib/crewai-tools/tests/tools/haunt_extract_tool_test.py',
'lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/README.md',
]
for rel in targets:
path = Path(rel)
print(f'\n### {rel} ###')
with path.open() as f:
for i, line in enumerate(f, 1):
if i <= 220:
print(f"{i:4d}: {line.rstrip()}")
PYRepository: crewAIInc/crewAI
Length of output: 5155
Handle non-2xx responses before parsing JSON.
response.json() runs first, so a plain-text/HTML 4xx/5xx can raise JSONDecodeError, and JSON error payloads from non-2xx responses can be misreported as extraction_failed. Decode defensively, reject every non-2xx status up front, and only treat 2xx success: false responses as page-level extraction failures.
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 68-75: use jsonify instead of json.dumps for JSON output
Context: json.dumps(
{
"error_code": data.get("error_code") or "extraction_failed",
"message": data.get("message")
or data.get("error")
or "extraction failed",
}
)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py`
around lines 54 - 76, Update the response handling in the extraction method
around requests.post so every non-2xx status is rejected before normal success
parsing. Decode the response body defensively to tolerate plain-text or HTML
errors, use any available error details when raising the API error, and reserve
the existing success-false JSON result for 2xx responses only.
…g, stricter tests
|
All three review points addressed in the latest commit: api_key moved to a PrivateAttr so it can never appear in model_dump or serialized snapshots (regression test added), non-2xx and non-JSON responses now raise a clean ValueError with the status code instead of crashing in json parsing (test added), and the markdown test now asserts response_format is actually sent in the request body. 7 tests green. |
There was a problem hiding this comment.
♻️ Duplicate comments (1)
lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py (1)
55-70: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winDo not follow redirects with the API key, and enforce a strict 2xx/object response contract.
requests.postfollows redirects by default. Requests documents special off-host removal forAuthorization, while custom headers are otherwise passed through; therefore,X-API-Keycan be forwarded to a redirected origin (inference). (requests.readthedocs.io)Additionally,
response.okis true for every status below 400, and.json()may return a list or scalar, so 3xx responses can be accepted and unexpected JSON shapes can crash at.get(). Always include the HTTP status when raising an API error, even when a message is available. (requests.readthedocs.io)Use
allow_redirects=False, check200 <= status_code < 300, require a dictionary payload, and add regression tests for 3xx and non-object JSON responses.Based on the PR objective, non-2xx responses must fail cleanly rather than being treated as successful extraction.
Proposed fix
response = requests.post( f"{self.base_url}/v1/extract", headers={"X-API-Key": self._api_key, "Content-Type": "application/json"}, json=body, timeout=self.timeout, + allow_redirects=False, ) try: data = response.json() except ValueError: - data = {} - if not response.ok: + data = None + if not 200 <= response.status_code < 300: + detail = ( + data.get("message") or data.get("error") + if isinstance(data, dict) + else None + ) + message = f"Haunt API error {response.status_code}" + if detail: + message += f": {detail}" raise ValueError( - data.get("message") - or data.get("error") - or f"Haunt API error {response.status_code}" + message ) + if not isinstance(data, dict): + raise ValueError( + f"Haunt API returned invalid JSON for status {response.status_code}" + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py` around lines 55 - 70, Update the extraction request in the relevant tool method to set allow_redirects=False, require a 200–299 response status, and validate that response.json() returns a dictionary before accessing message or error fields. Raise a clean ValueError for redirects, other non-2xx statuses, and non-object JSON, always including the HTTP status; add regression tests covering 3xx responses and list/scalar JSON payloads.
🧹 Nitpick comments (2)
lib/crewai-tools/tests/tools/haunt_extract_tool_test.py (1)
19-31: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winCover the request-boundary behavior.
The success test verifies URL and prompt, but not the
X-API-Keyheader, and no test exercises invalid URL handling. Add the header assertion plus a case that confirms an invalid URL raises beforerequests.postis called.As per coding guidelines, new functionality should have behavior-focused unit tests.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/tests/tools/haunt_extract_tool_test.py` around lines 19 - 31, Add the expected X-API-Key header assertion to test_haunt_extract_success_returns_json, and add a behavior-focused test for invalid URLs that verifies HauntExtractTool.run raises the appropriate validation error before requests.post is called.Source: Coding guidelines
lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py (1)
25-38: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd Python docstrings for the public tool API.
The agent-facing
descriptionis useful, butHauntExtractTooland its complex_runpath lack Python docstrings. Add concise class and method documentation for IDE/help and generated API docs.As per coding guidelines, public APIs and complex logic in Python code should be documented.
Also applies to: 45-46
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py` around lines 25 - 38, Add concise Python docstrings to the public HauntExtractTool class and its complex _run method, documenting the tool’s purpose, inputs, and returned structured data or error codes. Keep the existing agent-facing description unchanged and limit documentation changes to these public API symbols.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py`:
- Around line 55-70: Update the extraction request in the relevant tool method
to set allow_redirects=False, require a 200–299 response status, and validate
that response.json() returns a dictionary before accessing message or error
fields. Raise a clean ValueError for redirects, other non-2xx statuses, and
non-object JSON, always including the HTTP status; add regression tests covering
3xx responses and list/scalar JSON payloads.
---
Nitpick comments:
In
`@lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.py`:
- Around line 25-38: Add concise Python docstrings to the public
HauntExtractTool class and its complex _run method, documenting the tool’s
purpose, inputs, and returned structured data or error codes. Keep the existing
agent-facing description unchanged and limit documentation changes to these
public API symbols.
In `@lib/crewai-tools/tests/tools/haunt_extract_tool_test.py`:
- Around line 19-31: Add the expected X-API-Key header assertion to
test_haunt_extract_success_returns_json, and add a behavior-focused test for
invalid URLs that verifies HauntExtractTool.run raises the appropriate
validation error before requests.post is called.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e884ad62-bdea-4312-845c-c71aecf86d93
📒 Files selected for processing (2)
lib/crewai-tools/src/crewai_tools/tools/haunt_extract_tool/haunt_extract_tool.pylib/crewai-tools/tests/tools/haunt_extract_tool_test.py
1 participant
Adds
HauntExtractTool, a web extraction tool backed by the Haunt API.What it does: the agent supplies a URL and a plain-language prompt ("the product name, price and stock status") and gets structured JSON back, no selectors.
Why it earns a place on a shelf that already has scraping tools: honest failure. When a page cannot be read (bot wall, login wall, missing page), the tool returns a structured
error_code(access_denied,login_required,not_found) instead of invented content, so agents can branch on the failure instead of acting on made-up data. Failed reads are not charged.validate_url, env-var key (HAUNT_API_KEY), pydantic input schema.__init__.pyfiles next to the Jina tool.🤖 Generated with Claude Code
https://claude.ai/code/session_01K8HLNV61qpmJXhKtGA23a5