perf(postgres): preserve hash join in observation scoring

[stepanov1975]

Summary

• replace the planner-sensitive correlated seed-source anti-join in PostgreSQL observation expansion with an equivalent EXCEPT set difference
• preserve the existing per-entity fanout limit, seed-source exclusion, retrieval breadth, and scoring behavior
• add a regression test for the SQL shape and strengthen seed-source scoring coverage

Fixes #3510.

Problem

On a sufficiently connected PostgreSQL-backed observation graph, connected_sources can be underestimated as one row. PostgreSQL then chooses it as the outer side of a nested-loop join and performs millions of rejected comparisons while scoring otherwise set-wise candidates.

A representative warm-cache plan performed about 8.56 million rejected comparisons and took about 4.4 seconds in this path.

Approach

The previous form used SELECT DISTINCT ... WHERE NOT EXISTS (...). This change expresses the same distinct seed-source exclusion as EXCEPT:

SELECT DISTINCT t.unit_id AS source_id
...
EXCEPT
SELECT source_id FROM seed_sources

The left-side DISTINCT is intentionally retained to keep the source change minimal. The per-entity lateral ORDER BY ... LIMIT remains in the same position, before seed-source removal.

Results

Across three additional realistic seed sets, the original and rewritten SQL returned identical complete top-300 id -> score mappings:

Case	Original	EXCEPT	Speedup
1	1.753s	0.026s	67.1x
2	1.299s	0.021s	62.8x
3	2.069s	0.030s	69.8x

Median: 1.753s -> 0.026s (67.4x).

A packaged PostgreSQL full-stack A/B on the same synthetic graph-linked database also returned identical ordered result IDs, entities, chunks, source facts, and non-final score inputs. Tiny final-score drift was reproduced on repeated requests against the same image and came from request-time freshness, not the SQL rewrite. Restart persistence and rollback to the unmodified 0.9.1 image were also exercised successfully.

Testing

• uv run pytest tests/test_observation_expansion_scoring.py -q -n0 — 3 passed
• selected non-LLM observation/link-expansion suite — 8 passed
• repository lint hook: ./scripts/hooks/lint.sh
• Ruff formatting check
• uv run ty check hindsight_api
• packaged Hindsight 0.9.1 + PostgreSQL retain/recall smoke test
• real client/plugin retain and recall through the packaged image
• clean restart and rollback/restore smoke tests

The complete external-LLM test suite was not run locally; the focused and selected suites above exclude tests requiring an external LLM provider.