From 774339dfcb6a5f273fb44f2454f63ee17e08c793 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Wed, 8 Jul 2026 20:52:56 +0200 Subject: [PATCH 1/9] docs: literal-text repo + AGENTS.md breadcrumb in README intro (di-bench run 20260708-165318-6406) --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index ed527d8..bca33a6 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ SourceryKit is the Python SDK for [Provably](https://provably.ai). It provides verifiable guardrails for AI agents by automatically recording outbound HTTP calls, enforcing endpoint policies, and checking your agent's claims against a source of truth—all before any request leaves your process. +Source and docs live at https://github.com/ProvablyAI/sourcerykit. AI agents: start at https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md — the verification flow plus runnable cookbooks to mirror. + > ⚠️ **IMPORTANT:** Upgrading the SDK from v0.2 to v1.0? See the [v1.0 migration guide](https://github.com/ProvablyAI/sourcerykit/blob/main/docs/migrations/v1_0/v1_0.md). From 15a3b6e29bddd2a284a1f82da5a017de69edc1f5 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Wed, 8 Jul 2026 21:29:46 +0200 Subject: [PATCH 2/9] docs: note AGENTS.md is canonical (repo) + mirrored as provably.ai machine view --- AGENTS.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 7de13d2..ba0cf52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,10 @@ # AGENTS.md +> This file is the canonical agent entry point. It lives in the repository at +> https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md and is also mirrored as the +> machine view of https://provably.ai. If you reached it through the landing page, the repo +> copy is the source of truth — read it there when in doubt. + SourceryKit is the Python SDK for [Provably](https://provably.ai): verifiable guardrails for AI agents — it records outbound HTTP calls, enforces endpoint policies, and checks an agent's claims against what those calls actually returned, so a hallucinated value is From 070031db89f0913bf1465e77653e0365e50694d6 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Wed, 8 Jul 2026 22:00:45 +0200 Subject: [PATCH 3/9] docs: show claimed_value shape + steer to SourceryKitAgentResponse binding in handoff.md (di-bench run 20260708-192958-e188) --- docs/handoff.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/docs/handoff.md b/docs/handoff.md index 8260c63..aac087d 100644 --- a/docs/handoff.md +++ b/docs/handoff.md @@ -81,11 +81,32 @@ The `build_handoff_payload` function accepts a structured `payload_data` diction | Field | Type | Description | |---|---|---| | `action_name` | `str` | Logical identifier for the agent action producing the claim. | -| `claimed_value` | `Any` | The specific data value or object subset the agent claims to be true. | +| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — a **flat list** of `{path, value, sourcerykit_ref}` objects (see below). Not an arbitrary dict of your own field names. | | `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). | | `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. | | `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. | +### Claim value shape (`claimed_value`) + +`claimed_value` is the agent's `claimed_values`: a **flat list of `ClaimedValue` objects**, each +with three string fields — nothing else. + +| Field | Type | Description | +|---|---|---| +| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. | +| `value` | `str` | The extracted value, as a string. | +| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. | + +> [!IMPORTANT] +> Do not build `claimed_values` by hand. Bind `SourceryKitAgentResponse` as your agent's +> structured output (`output_type=` / `response_format=` — see the [cookbooks](../cookbooks)), +> and let the **agent** produce the list as part of its answer. Pass +> `final_output.claimed_values` straight into `claimed_value`, exactly as shown in the example +> above. Assembling claims yourself from the fetched data (a dict of your own keys like +> `{"hint_weight": 90, ...}`) both defeats the point — the *agent* must make the claim — and +> resolves **zero** claims: `evaluate_handoff` returns `outcome: "ERROR"` with an empty +> `per_claim`. The shape above is for understanding what the agent returns, not a hand-build recipe. + ## Verification Modes The evaluation engine processes claims using one of four specific strategies: From 03cd7ca59ebf4d22461d40e7a6b1faa0c620ddce Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 12:28:58 +0200 Subject: [PATCH 4/9] =?UTF-8?q?docs:=20make=20the=20SourceryKitAgentRespon?= =?UTF-8?q?se=20binding=20copyable=20=E2=80=94=20accept=20OPENROUTER=5FAPI?= =?UTF-8?q?=5FKEY,=20steer=20off=20bare=20HTTP=20(di-bench=20binding=20gap?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 8 ++++++++ cookbooks/openai_agents/agent_run.py | 4 +++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index ba0cf52..eba21eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,6 +43,14 @@ so the same tool can be called many times, and separate agents can each own a st binding is framework-specific. Copy the closest cookbook and change just that binding; each cookbook's README covers its own wiring. +**The claim must come from the agent, not your code.** Run a real agent with +`output_type=SourceryKitAgentResponse` (or your framework's equivalent) and pass its +`final_output.claimed_values` straight into the handoff. Do NOT hand-roll raw chat-completion +calls or assemble claims yourself from the fetched data — that skips the agent the whole +system exists to verify. Using any OpenAI-compatible endpoint (e.g. OpenRouter)? Point the +SDK's client at your `MODEL_URL` with `MODEL_API_KEY` or `OPENROUTER_API_KEY`, exactly as +[cookbooks/openai_agents](cookbooks/openai_agents) does — don't drop to bare HTTP. + **Single-agent** — one agent fetches, claims, and verifies (weather): | Cookbook | Framework | What you'll find | diff --git a/cookbooks/openai_agents/agent_run.py b/cookbooks/openai_agents/agent_run.py index 6dfe711..c6c0a02 100644 --- a/cookbooks/openai_agents/agent_run.py +++ b/cookbooks/openai_agents/agent_run.py @@ -38,7 +38,9 @@ _OPEN_METEO_BASE_URL = "https://api.open-meteo.com/v1/forecast" _DEFAULT_MODEL_URL = os.getenv("MODEL_URL", "http://127.0.0.1:1234/v1") -_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY", "") +# Accept either key name: MODEL_API_KEY, or OPENROUTER_API_KEY if you point MODEL_URL at +# OpenRouter (a common OpenAI-compatible gateway). +_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY") or os.getenv("OPENROUTER_API_KEY", "") _DEFAULT_MODEL = os.getenv("MODEL_NAME", "Qwen3.5-0.8B-MLX-4bit") From 633224ac5d712845f4bdb7129f00afb342400479 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 14:16:04 +0200 Subject: [PATCH 5/9] =?UTF-8?q?fix(evaluator):=20explain=20the=20empty-per?= =?UTF-8?q?=5Fclaim=20ERROR=20=E2=80=94=20evaluate=5Fhandoff=20was=20retur?= =?UTF-8?q?ning=20ERROR=20with=20an=20empty=20errors=20list=20when=20zero?= =?UTF-8?q?=20claims=20resolved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- src/sourcerykit/evaluator/evaluator.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/sourcerykit/evaluator/evaluator.py b/src/sourcerykit/evaluator/evaluator.py index 057d79b..6a17b88 100644 --- a/src/sourcerykit/evaluator/evaluator.py +++ b/src/sourcerykit/evaluator/evaluator.py @@ -73,6 +73,15 @@ async def evaluate_handoff(*, payload: HandoffPayload) -> dict[str, Any]: outcome = _resolve_outcome(per_claim, errors) + if outcome == Outcome.ERROR and not per_claim and not errors: + # Zero claims reached the verifier. Say so — an ERROR with an empty errors list gives + # the caller nothing to act on. + errors = [ + f"no claims were verified: the handoff payload contained {len(payload.claims)} " + "claim(s), and none resolved to a recorded intercept. Check that each claimed " + "value carries a sourcerykit_ref matching a recorded tool call." + ] + return {"outcome": outcome, "per_claim": per_claim, "errors": errors} From 66dd7c1489a1952096183ec7e694c39244cd6e61 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 14:23:34 +0200 Subject: [PATCH 6/9] =?UTF-8?q?docs:=20reorder=20handoff.md=20=E2=80=94=20?= =?UTF-8?q?introduce=20SourceryKitAgentResponse=20before=20claimed=5Fvalue?= =?UTF-8?q?s=20shape?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/handoff.md | 55 +++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/docs/handoff.md b/docs/handoff.md index aac087d..bb443a6 100644 --- a/docs/handoff.md +++ b/docs/handoff.md @@ -1,15 +1,41 @@ # Handoff The handoff mechanism structures agent claims—such as the result of an API call—and submits them to an evaluation service. These claims are evaluated deterministically against authoritative records to provide verifiable runtime guardrails. -Agent frameworks (OpenAI Agents SDK, LangChain) should use `SourceryKitAgentResponse` as the structured output type. This enforces a typed contract where the LLM returns a `reasoning` string and a `claimed_values` list—a flat collection of `ClaimedValue` objects, each with a JSONPath-style `path` and an extracted string `value`. These `claimed_values` feed directly into the handoff payload. - - ## Core Flow - **Collect Claims**: The agent records statements about its actions alongside supporting request/response payloads. - **Build Payload**: The claims, trusted endpoint snapshots, and execution metadata are compiled into a structured `HandoffPayload`. - **Evaluate**: The payload is sent to the evaluation service via the SDK, which matches claims against the backend source of truth and returns a verdict. +## The agent's structured output: `SourceryKitAgentResponse` +Claims originate from the agent itself, not from your code. Agent frameworks (OpenAI Agents SDK, LangChain) bind `SourceryKitAgentResponse` as the structured output type — `output_type=` for the OpenAI Agents SDK, `response_format=` for LangChain — so the LLM is forced to return this typed contract instead of free-form text. + +`SourceryKitAgentResponse` has two fields: + +| Field | Type | Description | +|---|---|---| +| `reasoning` | `str` | The agent's explanation of its actions for this execution slice. | +| `claimed_values` | `list[ClaimedValue]` | A **flat list** of the values the agent claims it produced (see below). | + +Each `ClaimedValue` in that list has three string fields — nothing else: + +| Field | Type | Description | +|---|---|---| +| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. | +| `value` | `str` | The extracted value, as a string. | +| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. | + +`final_output.claimed_values` feeds directly into the handoff payload — you pass the list straight through, as shown in the [Example](#example) and payload tables below. + +> [!IMPORTANT] +> Do not build `claimed_values` by hand. Bind `SourceryKitAgentResponse` as your agent's +> structured output (see the [cookbooks](../cookbooks)) and let the **agent** produce the list +> as part of its answer. Assembling claims yourself from the fetched data (a dict of your own +> keys like `{"hint_weight": 90, ...}`) both defeats the point — the *agent* must make the claim +> — and resolves **zero** claims: `evaluate_handoff` returns `outcome: "ERROR"` with an empty +> `per_claim`. + + ## Example The following example demonstrates how to construct a claim, bundle it into a HandoffPayload, and submit it for evaluation: @@ -81,32 +107,11 @@ The `build_handoff_payload` function accepts a structured `payload_data` diction | Field | Type | Description | |---|---|---| | `action_name` | `str` | Logical identifier for the agent action producing the claim. | -| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — a **flat list** of `{path, value, sourcerykit_ref}` objects (see below). Not an arbitrary dict of your own field names. | +| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — pass `final_output.claimed_values` straight through. See [`SourceryKitAgentResponse`](#the-agents-structured-output-sourcerykitagentresponse) for the shape. Not an arbitrary dict of your own field names. | | `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). | | `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. | | `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. | -### Claim value shape (`claimed_value`) - -`claimed_value` is the agent's `claimed_values`: a **flat list of `ClaimedValue` objects**, each -with three string fields — nothing else. - -| Field | Type | Description | -|---|---|---| -| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. | -| `value` | `str` | The extracted value, as a string. | -| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. | - -> [!IMPORTANT] -> Do not build `claimed_values` by hand. Bind `SourceryKitAgentResponse` as your agent's -> structured output (`output_type=` / `response_format=` — see the [cookbooks](../cookbooks)), -> and let the **agent** produce the list as part of its answer. Pass -> `final_output.claimed_values` straight into `claimed_value`, exactly as shown in the example -> above. Assembling claims yourself from the fetched data (a dict of your own keys like -> `{"hint_weight": 90, ...}`) both defeats the point — the *agent* must make the claim — and -> resolves **zero** claims: `evaluate_handoff` returns `outcome: "ERROR"` with an empty -> `per_claim`. The shape above is for understanding what the agent returns, not a hand-build recipe. - ## Verification Modes The evaluation engine processes claims using one of four specific strategies: From c161bbe4947ecf01da0b09d4c718e5391a7d7814 Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 14:30:24 +0200 Subject: [PATCH 7/9] fix(handoff): surface per-claim drop reasons through build_handoff_payload to evaluate_handoff instead of only logging them --- src/sourcerykit/evaluator/evaluator.py | 7 +++++-- src/sourcerykit/handoff/payload_builder.py | 24 +++++++++++++++------- src/sourcerykit/schemas/handoff.py | 5 +++++ 3 files changed, 27 insertions(+), 9 deletions(-) diff --git a/src/sourcerykit/evaluator/evaluator.py b/src/sourcerykit/evaluator/evaluator.py index 6a17b88..006b7d4 100644 --- a/src/sourcerykit/evaluator/evaluator.py +++ b/src/sourcerykit/evaluator/evaluator.py @@ -71,11 +71,14 @@ async def evaluate_handoff(*, payload: HandoffPayload) -> dict[str, Any]: } per_claim.append(verdict) + # Claims dropped while building the payload never reach the verifier above. Surface those + # reasons — otherwise a payload that arrives with zero claims is an ERROR with no + # explanation of what went wrong. + errors = list(payload.build_errors) + errors + outcome = _resolve_outcome(per_claim, errors) if outcome == Outcome.ERROR and not per_claim and not errors: - # Zero claims reached the verifier. Say so — an ERROR with an empty errors list gives - # the caller nothing to act on. errors = [ f"no claims were verified: the handoff payload contained {len(payload.claims)} " "claim(s), and none resolved to a recorded intercept. Check that each claimed " diff --git a/src/sourcerykit/handoff/payload_builder.py b/src/sourcerykit/handoff/payload_builder.py index 1596f82..1427f7e 100644 --- a/src/sourcerykit/handoff/payload_builder.py +++ b/src/sourcerykit/handoff/payload_builder.py @@ -61,12 +61,13 @@ async def build_handoff_payload( instr = instructions if instructions is not None else default_instructions() # Run claim resolution and trusted-endpoint fetch concurrently - (claims, query_urls, query_ids), trusted_endpoint_registry = await asyncio.gather( + (claims, query_urls, query_ids, build_errors), trusted_endpoint_registry = await asyncio.gather( _build_claims(trace_id, blob, intercept_agent_id), list_all_trusted_endpoints(), ) - _log.info("build_handoff_payload_completed", claim_count=len(claims)) + _log.info("build_handoff_payload_completed", claim_count=len(claims), + dropped=len(build_errors)) return HandoffPayload( provably_mcp_url=settings.provably_mcp, @@ -83,6 +84,7 @@ async def build_handoff_payload( query_urls=query_urls, task=task, reasoning=reasoning, + build_errors=build_errors, ) @@ -123,15 +125,20 @@ async def _build_claims( trace_id: uuid.UUID, fetch_and_claim_json: Any, intercept_agent_id: str, -) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID]]: - """Aggregates, resolves intercept IDs, and emits tracking metadata.""" +) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID], list[str]]: + """Aggregates, resolves intercept IDs, and emits tracking metadata. + + The fourth return value is the drop reasons: one message per claim that could not be + resolved. These travel to the caller (on the payload) instead of being only logged, so a + payload that ends up with zero claims can explain why. + """ if not isinstance(fetch_and_claim_json, dict): - return [], [], [] + return [], [], [], ["fetch_and_claim is not a dict; no claims could be read"] raw_claims = fetch_and_claim_json.get("claims") if not isinstance(raw_claims, list): - return [], [], [] + return [], [], [], ["fetch_and_claim['claims'] is missing or not a list"] # Filter to valid claim dicts up-front valid_raws = [raw for raw in raw_claims if isinstance(raw, dict) and str(raw.get("action_name") or "").strip()] @@ -150,19 +157,22 @@ async def _build_claims( claims = [] urls = [] ids = [] + drop_errors: list[str] = [] for raw, r in zip(expanded, results): if isinstance(r, BaseException): + reason = f"claim '{raw.get('action_name')}' dropped: {r}" _log.warning( "claim_resolution_failed", action_name=raw.get("action_name"), error=str(r), ) + drop_errors.append(reason) continue claims.append(r[0]) urls.append(r[1]) ids.append(r[2]) - return claims, urls, ids + return claims, urls, ids, drop_errors async def _resolve_claim( diff --git a/src/sourcerykit/schemas/handoff.py b/src/sourcerykit/schemas/handoff.py index fbfc6a7..0a5990c 100644 --- a/src/sourcerykit/schemas/handoff.py +++ b/src/sourcerykit/schemas/handoff.py @@ -107,6 +107,11 @@ class HandoffPayload(BaseModel): ) task: str = Field(default="", description="Short task title.") reasoning: str = Field(default="", description="Agent's natural-language reasoning trace.") + build_errors: list[str] = Field( + default_factory=list, + description="Reasons any claims were dropped while building this payload (e.g. a claim " + "whose sourcerykit_ref matched no recorded intercept). Surfaced by evaluate_handoff.", + ) sdk_precheck: dict[str, Any] | None = Field( default=None, description="Optional SDK-side health/precheck output captured before handoff.", From 64466379cbe56f36d49f07a518cba4937dce980e Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 17:34:15 +0200 Subject: [PATCH 8/9] =?UTF-8?q?docs:=20give=20the=20true=20reason=20not=20?= =?UTF-8?q?to=20hand-build=20claims=20=E2=80=94=20a=20shaped-but-hand-buil?= =?UTF-8?q?t=20claim=20PASSes=20yet=20proves=20nothing=20(di-bench=20run?= =?UTF-8?q?=2020260709-151308-f17d)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/handoff.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/handoff.md b/docs/handoff.md index bb443a6..633e7ce 100644 --- a/docs/handoff.md +++ b/docs/handoff.md @@ -28,12 +28,17 @@ Each `ClaimedValue` in that list has three string fields — nothing else: `final_output.claimed_values` feeds directly into the handoff payload — you pass the list straight through, as shown in the [Example](#example) and payload tables below. > [!IMPORTANT] -> Do not build `claimed_values` by hand. Bind `SourceryKitAgentResponse` as your agent's -> structured output (see the [cookbooks](../cookbooks)) and let the **agent** produce the list -> as part of its answer. Assembling claims yourself from the fetched data (a dict of your own -> keys like `{"hint_weight": 90, ...}`) both defeats the point — the *agent* must make the claim -> — and resolves **zero** claims: `evaluate_handoff` returns `outcome: "ERROR"` with an empty -> `per_claim`. +> `claimed_values` must be the agent's own structured output. Bind `SourceryKitAgentResponse` +> (see the [cookbooks](../cookbooks)) and pass `final_output.claimed_values` straight through — +> do not assemble the list yourself from the fetched data. +> +> A *wrong-shaped* hand-built claim (a dict of your own keys like `{"hint_weight": 90, ...}`) +> resolves **zero** claims — `evaluate_handoff` returns `ERROR` with an empty `per_claim`. But a +> *correctly-shaped* hand-built claim is the more dangerous case: it can return `PASS` and still +> prove nothing, because you have only compared the fetched data to itself. SourceryKit exists to +> catch a value **the agent got wrong**; if you build the claim from the data by hand there is no +> independent agent claim to check. The verification is only meaningful when the values are what +> the agent actually declared. ## Example From 0a25f215066fb4103e364985513bcb869da4bd6b Mon Sep 17 00:00:00 2001 From: rimkusaurimas Date: Thu, 9 Jul 2026 17:50:07 +0200 Subject: [PATCH 9/9] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20drop?= =?UTF-8?q?=20README=20repo=20breadcrumb=20(repetition)=20and=20the=20inst?= =?UTF-8?q?ructional=20block=20from=20handoff.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 2 -- docs/handoff.md | 13 ------------- 2 files changed, 15 deletions(-) diff --git a/README.md b/README.md index bca33a6..ed527d8 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,6 @@ SourceryKit is the Python SDK for [Provably](https://provably.ai). It provides verifiable guardrails for AI agents by automatically recording outbound HTTP calls, enforcing endpoint policies, and checking your agent's claims against a source of truth—all before any request leaves your process. -Source and docs live at https://github.com/ProvablyAI/sourcerykit. AI agents: start at https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md — the verification flow plus runnable cookbooks to mirror. - > ⚠️ **IMPORTANT:** Upgrading the SDK from v0.2 to v1.0? See the [v1.0 migration guide](https://github.com/ProvablyAI/sourcerykit/blob/main/docs/migrations/v1_0/v1_0.md). diff --git a/docs/handoff.md b/docs/handoff.md index 633e7ce..1cfb013 100644 --- a/docs/handoff.md +++ b/docs/handoff.md @@ -27,19 +27,6 @@ Each `ClaimedValue` in that list has three string fields — nothing else: `final_output.claimed_values` feeds directly into the handoff payload — you pass the list straight through, as shown in the [Example](#example) and payload tables below. -> [!IMPORTANT] -> `claimed_values` must be the agent's own structured output. Bind `SourceryKitAgentResponse` -> (see the [cookbooks](../cookbooks)) and pass `final_output.claimed_values` straight through — -> do not assemble the list yourself from the fetched data. -> -> A *wrong-shaped* hand-built claim (a dict of your own keys like `{"hint_weight": 90, ...}`) -> resolves **zero** claims — `evaluate_handoff` returns `ERROR` with an empty `per_claim`. But a -> *correctly-shaped* hand-built claim is the more dangerous case: it can return `PASS` and still -> prove nothing, because you have only compared the fetched data to itself. SourceryKit exists to -> catch a value **the agent got wrong**; if you build the claim from the data by hand there is no -> independent agent claim to check. The verification is only meaningful when the values are what -> the agent actually declared. - ## Example The following example demonstrates how to construct a claim, bundle it into a HandoffPayload, and submit it for evaluation: