docs: di-bench improvement loop by aural-psynapse · Pull Request #70 · ProvablyAI/sourcerykit

aural-psynapse · 2026-07-08T18:53:12Z

Changes

Docs

AGENTS.md — note it is the canonical agent entry point (mirrored as the provably.ai machine view), and make explicit that claims must come from the agent's own SourceryKitAgentResponse output, not be hand-assembled.
docs/handoff.md — add a SourceryKitAgentResponse section describing the typed output and its ClaimedValue fields (path, value, sourcerykit_ref), ordered type → component. Correct the claimed_value payload row from Any to list[ClaimedValue].
cookbooks/openai_agents — accept OPENROUTER_API_KEY as well as MODEL_API_KEY, so a verbatim copy authenticates against an OpenAI-compatible gateway (OpenRouter).

Code

evaluate_handoff / build_handoff_payload — when a claim can't be resolved it was dropped with only a log line, so a payload that ended up with zero claims surfaced as ERROR with an empty errors list. The drop reason now travels on the payload (build_errors) and evaluate_handoff returns it in errors, so a zero-claim result explains itself.

Why

These changes come out of a discoverability loop that runs a fresh agent from provably.ai to a working SourceryKit integration and turns each observed failure into one fix. Runs kept failing at the same points: agents couldn't find the entry doc, hand-assembled claims instead of binding SourceryKitAgentResponse, couldn't authenticate the cookbook with the key they had, and hit an ERROR with no reason when claims failed to resolve. Each item above addresses one of those. (The separate preprocess bug that blocked verification end-to-end is fixed in #71.)

…nch run 20260708-165318-6406)

…chine view

…nding in handoff.md (di-bench run 20260708-192958-e188)

SimoneBottoni · 2026-07-09T08:01:52Z

+Source and docs live at https://github.com/ProvablyAI/sourcerykit. AI agents: start at https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md — the verification flow plus runnable cookbooks to mirror.
+


I don't like the location, right after the intro.
I don't see any point in saying "Source and docs live at https://github.com/ProvablyAI/sourcerykit."

Since the README is also a front-page for PyPi, this becomes fragmented and needs attribution. If we can change the PyPi README individually, we can remove it from the Github README. In other way - I'd keep it

It's still a repetition. The necessary links is already present in the PyPi sidebar.

SimoneBottoni · 2026-07-09T08:10:26Z

+> [!IMPORTANT]
+> Do not build `claimed_values` by hand. Bind `SourceryKitAgentResponse` as your agent's
+> structured output (`output_type=` / `response_format=` — see the [cookbooks](../cookbooks)),
+> and let the **agent** produce the list as part of its answer. Pass
+> `final_output.claimed_values` straight into `claimed_value`, exactly as shown in the example
+> above. Assembling claims yourself from the fetched data (a dict of your own keys like
+> `{"hint_weight": 90, ...}`) both defeats the point — the *agent* must make the claim — and
+> resolves **zero** claims: `evaluate_handoff` returns `outcome: "ERROR"` with an empty
+> `per_claim`. The shape above is for understanding what the agent returns, not a hand-build recipe.
+


The rest of the document isn't strictly for instructions, but for component descriptions. Don't mix things up.

Docs are docs, from my PoV. If we need to write a disclaimer we shouldn't be limited from doing it.. Especially if it improves the agent flow

No, we prefer quality over quantity, for the code as well as for the documentation, which is equally important.

SimoneBottoni · 2026-07-09T08:10:52Z

+| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — a **flat list** of `{path, value, sourcerykit_ref}` objects (see below). Not an arbitrary dict of your own field names. |
 | `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). |
 | `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. |
 | `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. |

+### Claim value shape (`claimed_value`)
+
+`claimed_value` is the agent's `claimed_values`: a **flat list of `ClaimedValue` objects**, each
+with three string fields — nothing else.
+
+| Field | Type | Description |
+|---|---|---|
+| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. |
+| `value` | `str` | The extracted value, as a string. |
+| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. |
+


There's no point in describing the structure this way if it's not exposed. If you want to add a description, that's fine, but you need to follow a flow. Suggestion: First describe the SourceryKitAgentResponse, how it's used, etc., then its component, and here you can describe the claimed_values.

Agreed, will fix

…NROUTER_API_KEY, steer off bare HTTP (di-bench binding gap)

…was returning ERROR with an empty errors list when zero claims resolved

…claimed_values shape

…yload to evaluate_handoff instead of only logging them

…nd-built claim PASSes yet proves nothing (di-bench run 20260709-151308-f17d)

…he instructional block from handoff.md

SimoneBottoni · 2026-07-09T16:29:35Z

-_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY", "")
+# Accept either key name: MODEL_API_KEY, or OPENROUTER_API_KEY if you point MODEL_URL at
+# OpenRouter (a common OpenAI-compatible gateway).
+_DEFAULT_MODEL_API_KEY = os.getenv("MODEL_API_KEY") or os.getenv("OPENROUTER_API_KEY", "")


This is not general. MODEL_API_KEY should contain OPENROUTER_API_KEY if OPENROUTER is used as provided.

SimoneBottoni · 2026-07-09T17:04:58Z

+    drop_errors: list[str] = []
    for raw, r in zip(expanded, results):
        if isinstance(r, BaseException):
+            reason = f"claim '{raw.get('action_name')}' dropped: {r}"


r represents the result set composed of claim, qurl, and qid. Too much content and no reason for the error.

SimoneBottoni · 2026-07-09T17:07:03Z

+    The fourth return value is the drop reasons: one message per claim that could not be
+    resolved. These travel to the caller (on the payload) instead of being only logged, so a
+    payload that ends up with zero claims can explain why.


Remove this part of the method description. It makes no sense to describe only 1 return value of 4.

aural-psynapse added 3 commits July 8, 2026 20:52

docs: literal-text repo + AGENTS.md breadcrumb in README intro (di-be…

774339d

…nch run 20260708-165318-6406)

docs: note AGENTS.md is canonical (repo) + mirrored as provably.ai ma…

15a3b6e

…chine view

docs: show claimed_value shape + steer to SourceryKitAgentResponse bi…

070031d

…nding in handoff.md (di-bench run 20260708-192958-e188)

aural-psynapse force-pushed the docs/di-bench-loop branch from 5d7847c to 070031d Compare July 8, 2026 20:03

SimoneBottoni requested changes Jul 9, 2026

View reviewed changes

aural-psynapse added 6 commits July 9, 2026 12:28

docs: make the SourceryKitAgentResponse binding copyable — accept OPE…

03cd7ca

…NROUTER_API_KEY, steer off bare HTTP (di-bench binding gap)

fix(evaluator): explain the empty-per_claim ERROR — evaluate_handoff …

633224a

…was returning ERROR with an empty errors list when zero claims resolved

docs: reorder handoff.md — introduce SourceryKitAgentResponse before …

66dd7c1

…claimed_values shape

fix(handoff): surface per-claim drop reasons through build_handoff_pa…

c161bbe

…yload to evaluate_handoff instead of only logging them

docs: give the true reason not to hand-build claims — a shaped-but-ha…

6446637

…nd-built claim PASSes yet proves nothing (di-bench run 20260709-151308-f17d)

docs: address review — drop README repo breadcrumb (repetition) and t…

0a25f21

…he instructional block from handoff.md

aural-psynapse requested a review from SimoneBottoni July 9, 2026 15:51

Merge remote-tracking branch 'origin/main' into docs/di-bench-loop

24aecff

SimoneBottoni requested changes Jul 9, 2026

View reviewed changes

		Source and docs live at https://github.com/ProvablyAI/sourcerykit. AI agents: start at https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md — the verification flow plus runnable cookbooks to mirror.

Uh oh!

Conversation

aural-psynapse commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Why

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SimoneBottoni Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aural-psynapse Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aural-psynapse Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aural-psynapse commented Jul 8, 2026 •

edited

Loading

SimoneBottoni Jul 9, 2026 •

edited

Loading

aural-psynapse Jul 9, 2026 •

edited

Loading

aural-psynapse Jul 9, 2026 •

edited

Loading