fix(tests): stop the LLM judge mistaking context for the response

[nicoloboschi]

The bug

test_reflect_split_synthesis::test_facts_from_distinct_chunks_reach_the_answer failed four times across four CI runs, on three different PRs — including #3543, which changed nothing but PL/pgSQL maintenance routines. Every failure carried byte-identical judge reasoning, so it was never a flake.

Criteria: The answer mentions BOTH hobbies: Zara's beekeeping AND Marco's vintage synthesizers.
Response: * Zara keeps bees on her rooftop.
          * Marco collects vintage synthesizers.
Judge:    "The response does not mention either of the hobbies. It only states that
           the memory data contained two hobby facts."

The response plainly meets the criteria. The judge's sentence is a near-verbatim paraphrase of the test's context, not its response — and that is the whole bug.

tests/llm_judge.py built the prompt as:

f"## Response to evaluate\n{response}\n"
f"{context_block}\n"          # -> "\n\nContext provided to the system:\n{context}"
f"## Criteria\n{criteria}\n\n"

The response and criteria get ## headers; the context gets a bare line. So a multi-line response runs straight into the context with nothing marking where one ends and the other begins. This test's response is a two-bullet markdown list immediately followed by a context that opens "The memory data contained two hobby facts…" — and the judge read that opening sentence as the answer.

Any judge call passing a prose context is exposed. This one hit it reliably because its context restates the facts.

The fix

Tag each section rather than merely heading it. Headers alone cannot delimit, because a response may itself be markdown — a response containing ## Criteria would forge a section. Tags survive that. The system prompt now states outright that only <response> is judged and that <context> is background, never the subject.

Prompt assembly moves into build_judge_messages() so it is covered by fast unit tests instead of only by the LLM tests whose outcome it silently decides.

Verification

A/B against the live judge, using the exact response/criteria/context from the failing test, bypassing the reflect pipeline to isolate the prompt:

	verdict
old prompt	0/4 criteria met — every run reproducing the CI reasoning verbatim
new prompt	4/4 criteria met

test_facts_from_distinct_chunks_reach_the_answer passes again. Five new unit tests cover the assembly: sections stay separated, an absent context leaves no empty block, the system prompt scopes judgement, and a response containing its own ## headers cannot forge a section. The regression test uses the exact bullet-list-plus-prose shape that broke.

Risk

Scoping the judge strictly to <response> could in principle flip a verdict in some other test that was passing because context leaked into the judgement. Core LLM tests and the six LLM-acceptance jobs exercise the whole judge surface, so CI is the check on that rather than my assertion.

🤖 Generated with Claude Code