docs: di-bench improvement loop

AGENTS.md

@@ -1,5 +1,10 @@
 # AGENTS.md
 
+> This file is the canonical agent entry point. It lives in the repository at
+> https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md and is also mirrored as the
+> machine view of https://provably.ai. If you reached it through the landing page, the repo
+> copy is the source of truth — read it there when in doubt.
+
 SourceryKit is the Python SDK for [Provably](https://provably.ai): verifiable guardrails
 for AI agents — it records outbound HTTP calls, enforces endpoint policies, and checks an
 agent's claims against what those calls actually returned, so a hallucinated value is
@@ -38,6 +43,14 @@ so the same tool can be called many times, and separate agents can each own a st
 binding is framework-specific. Copy the closest cookbook and change just that binding; each
 cookbook's README covers its own wiring.
 
+**The claim must come from the agent, not your code.** Run a real agent with
+`output_type=SourceryKitAgentResponse` (or your framework's equivalent) and pass its
+`final_output.claimed_values` straight into the handoff. Do NOT hand-roll raw chat-completion
+calls or assemble claims yourself from the fetched data — that skips the agent the whole
+system exists to verify. Using any OpenAI-compatible endpoint (e.g. OpenRouter)? Point the
+SDK's client at your `MODEL_URL` with `MODEL_API_KEY` or `OPENROUTER_API_KEY`, exactly as
+[cookbooks/openai_agents](cookbooks/openai_agents) does — don't drop to bare HTTP.
+
 **Single-agent** — one agent fetches, claims, and verifies (weather):
 
 | Cookbook | Framework | What you'll find |

cookbooks/openai_agents/agent_run.py

@@ -38,7 +38,9 @@
 
 _OPEN_METEO_BASE_URL = "https://api.open-meteo.com/v1/forecast"
 _DEFAULT_MODEL_URL = os.getenv("MODEL_URL", "http://127.0.0.1:1234/v1")
-_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY", "")
+# Accept either key name: MODEL_API_KEY, or OPENROUTER_API_KEY if you point MODEL_URL at
+# OpenRouter (a common OpenAI-compatible gateway).
+_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY") or os.getenv("OPENROUTER_API_KEY", "")
 _DEFAULT_MODEL = os.getenv("MODEL_NAME", "Qwen3.5-0.8B-MLX-4bit")
 
 

docs/handoff.md

@@ -1,15 +1,33 @@
 # Handoff
 The handoff mechanism structures agent claims—such as the result of an API call—and submits them to an evaluation service. These claims are evaluated deterministically against authoritative records to provide verifiable runtime guardrails.
 
-Agent frameworks (OpenAI Agents SDK, LangChain) should use `SourceryKitAgentResponse` as the structured output type. This enforces a typed contract where the LLM returns a `reasoning` string and a `claimed_values` list—a flat collection of `ClaimedValue` objects, each with a JSONPath-style `path` and an extracted string `value`. These `claimed_values` feed directly into the handoff payload.
-
-
 ## Core Flow
 - **Collect Claims**: The agent records statements about its actions alongside supporting request/response payloads.
 - **Build Payload**: The claims, trusted endpoint snapshots, and execution metadata are compiled into a structured `HandoffPayload`.
 - **Evaluate**: The payload is sent to the evaluation service via the SDK, which matches claims against the backend source of truth and returns a verdict.
 
 
+## The agent's structured output: `SourceryKitAgentResponse`
+Claims originate from the agent itself, not from your code. Agent frameworks (OpenAI Agents SDK, LangChain) bind `SourceryKitAgentResponse` as the structured output type — `output_type=` for the OpenAI Agents SDK, `response_format=` for LangChain — so the LLM is forced to return this typed contract instead of free-form text.
+
+`SourceryKitAgentResponse` has two fields:
+
+| Field | Type | Description |
+|---|---|---|
+| `reasoning` | `str` | The agent's explanation of its actions for this execution slice. |
+| `claimed_values` | `list[ClaimedValue]` | A **flat list** of the values the agent claims it produced (see below). |
+
+Each `ClaimedValue` in that list has three string fields — nothing else:
+
+| Field | Type | Description |
+|---|---|---|
+| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. |
+| `value` | `str` | The extracted value, as a string. |
+| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. |
+
+`final_output.claimed_values` feeds directly into the handoff payload — you pass the list straight through, as shown in the [Example](#example) and payload tables below.
+
+
 ## Example
 The following example demonstrates how to construct a claim, bundle it into a HandoffPayload, and submit it for evaluation:
 
@@ -81,7 +99,7 @@ The `build_handoff_payload` function accepts a structured `payload_data` diction
 | Field | Type | Description |
 |---|---|---|
 | `action_name` | `str` | Logical identifier for the agent action producing the claim. |
-| `claimed_value` | `Any` | The specific data value or object subset the agent claims to be true. |
+| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — pass `final_output.claimed_values` straight through. See [`SourceryKitAgentResponse`](#the-agents-structured-output-sourcerykitagentresponse) for the shape. Not an arbitrary dict of your own field names. |
 | `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). |
 | `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. |
 | `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. |

src/sourcerykit/evaluator/evaluator.py

@@ -71,8 +71,20 @@ async def evaluate_handoff(*, payload: HandoffPayload) -> dict[str, Any]:
         }
         per_claim.append(verdict)
 
+    # Claims dropped while building the payload never reach the verifier above. Surface those
+    # reasons — otherwise a payload that arrives with zero claims is an ERROR with no
+    # explanation of what went wrong.
+    errors = list(payload.build_errors) + errors
+
     outcome = _resolve_outcome(per_claim, errors)
 
+    if outcome == Outcome.ERROR and not per_claim and not errors:
+        errors = [
+            f"no claims were verified: the handoff payload contained {len(payload.claims)} "
+            "claim(s), and none resolved to a recorded intercept. Check that each claimed "
+            "value carries a sourcerykit_ref matching a recorded tool call."
+        ]
+
     return {"outcome": outcome, "per_claim": per_claim, "errors": errors}
 
 

src/sourcerykit/handoff/payload_builder.py

@@ -61,12 +61,13 @@ async def build_handoff_payload(
     instr = instructions if instructions is not None else default_instructions()
 
     # Run claim resolution and trusted-endpoint fetch concurrently
-    (claims, query_urls, query_ids), trusted_endpoint_registry = await asyncio.gather(
+    (claims, query_urls, query_ids, build_errors), trusted_endpoint_registry = await asyncio.gather(
         _build_claims(trace_id, blob, intercept_agent_id),
         list_all_trusted_endpoints(),
     )
 
-    _log.info("build_handoff_payload_completed", claim_count=len(claims))
+    _log.info("build_handoff_payload_completed", claim_count=len(claims),
+              dropped=len(build_errors))
 
     return HandoffPayload(
         provably_mcp_url=settings.provably_mcp,
@@ -83,6 +84,7 @@ async def build_handoff_payload(
         query_urls=query_urls,
         task=task,
         reasoning=reasoning,
+        build_errors=build_errors,
     )
 
 
@@ -123,15 +125,20 @@ async def _build_claims(
     trace_id: uuid.UUID,
     fetch_and_claim_json: Any,
     intercept_agent_id: str,
-) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID]]:
-    """Aggregates, resolves intercept IDs, and emits tracking metadata."""
+) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID], list[str]]:
+    """Aggregates, resolves intercept IDs, and emits tracking metadata.
+
+    The fourth return value is the drop reasons: one message per claim that could not be
+    resolved. These travel to the caller (on the payload) instead of being only logged, so a
+    payload that ends up with zero claims can explain why.
+    """
 
     if not isinstance(fetch_and_claim_json, dict):
-        return [], [], []
+        return [], [], [], ["fetch_and_claim is not a dict; no claims could be read"]
 
     raw_claims = fetch_and_claim_json.get("claims")
     if not isinstance(raw_claims, list):
-        return [], [], []
+        return [], [], [], ["fetch_and_claim['claims'] is missing or not a list"]
 
     # Filter to valid claim dicts up-front
     valid_raws = [raw for raw in raw_claims if isinstance(raw, dict) and str(raw.get("action_name") or "").strip()]
@@ -150,19 +157,22 @@ async def _build_claims(
     claims = []
     urls = []
     ids = []
+    drop_errors: list[str] = []
     for raw, r in zip(expanded, results):
         if isinstance(r, BaseException):
+            reason = f"claim '{raw.get('action_name')}' dropped: {r}"
             _log.warning(
                 "claim_resolution_failed",
                 action_name=raw.get("action_name"),
                 error=str(r),
             )
+            drop_errors.append(reason)
             continue
         claims.append(r[0])
         urls.append(r[1])
         ids.append(r[2])
 
-    return claims, urls, ids
+    return claims, urls, ids, drop_errors
 
 
 async def _resolve_claim(

src/sourcerykit/schemas/handoff.py

@@ -107,6 +107,11 @@ class HandoffPayload(BaseModel):
     )
     task: str = Field(default="", description="Short task title.")
     reasoning: str = Field(default="", description="Agent's natural-language reasoning trace.")
+    build_errors: list[str] = Field(
+        default_factory=list,
+        description="Reasons any claims were dropped while building this payload (e.g. a claim "
+        "whose sourcerykit_ref matched no recorded intercept). Surfaced by evaluate_handoff.",
+    )
     sdk_precheck: dict[str, Any] | None = Field(
         default=None,
         description="Optional SDK-side health/precheck output captured before handoff.",