Skip to content

fix(consolidation): keep source dates when observations merge - #3500

Open
nicoloboschi wants to merge 1 commit into
mainfrom
fix/3477-consolidation-temporal-merge
Open

fix(consolidation): keep source dates when observations merge#3500
nicoloboschi wants to merge 1 commit into
mainfrom
fix/3477-consolidation-temporal-merge

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Fixes #3477: an observation could cite dated source facts while reporting occurred_start and occurred_end as null.

Reproduced end-to-end against a real database. The cause is the semantic-dedup folds, not the ordinary write paths:

  • CREATE fold (_dedup_reconcile_create) — when a CREATE is merged into a near-duplicate twin instead of being inserted, the fold UPDATE rewrote only text, source_memory_ids, proof_count and updated_at. The dates the skipped CREATE was going to be stamped with were dropped, so the twin gained a dated source fact and kept its own (often empty) interval. Dedup is on by default (consolidation_dedup_threshold 0.97), so this is the default path.
  • UPDATE fold (_dedup_reconcile_update) — same fold, and here the folded-from row is then deleted, so anything only it knew about was lost outright.
  • Ordinary UPDATE (_execute_update_action) — this one already merged occurred_start/occurred_end/mentioned_at correctly (LEAST/GREATEST ignore NULLs in Postgres), but never touched event_date, leaving it at whatever the observation was stamped with when it was created. Now it carries event_date too, so all four fields follow the same rule everywhere.

The rule is the one _aggregate_source_fields already documents and CREATE already applies: event_date/occurred_start keep the earliest known value, occurred_end/mentioned_at the latest, and a missing value on either side is ignored. A merge can only ever widen an observation's interval. _TemporalBounds names that contract in one place; the SQL paths express it as LEAST(col, COALESCE(x, col)) / GREATEST(col, COALESCE(x, col)) — the idiom already in this file, which is also NULL-safe on Oracle (a bare LEAST(col, $n) would wipe the column there when the parameter is NULL).

Verification

tests/test_consolidation_temporal_merge.py covers each merge path. The three SQL-executing tests run against a real database on purpose — the merge lives in SQL, and a mocked connection cannot tell a working statement from one PostgreSQL refuses to plan. Each was confirmed to fail when its merge clause is removed:

test path executes
test_dedup_create_fold_widens_bounds_of_the_twin full run_consolidation_job, dedup fold real SQL
test_update_widens_bounds_from_its_source_facts ordinary observation UPDATE real SQL
test_dedup_update_fold_unions_the_bounds_of_both_rows UPDATE-time fold + delete real SQL
test_temporal_bounds_merge_keeps_the_widest_known_interval the min/max/NULL contract unit
test_temporal_bounds_of_reads_a_source_aggregation source aggregation → bounds unit
test_store_owned_fold_merges_bounds_like_the_sql_path external-store fold (no SQL) unit

The first one is the issue's exact scenario: an undated observation, a dated source fact whose CREATE folds into it, and the survivor must come out with the source's interval.

Also ran the whole hindsight-api-slim suite locally (non-LLM markers): the consolidation/observation/temporal modules are green, and the handful of unrelated failures left reproduce identically on unmodified main. Plus ./scripts/hooks/lint.sh, check-unused.sh and ty.

Note on #3482

#3482 diagnosed this first and proposed the same min/max contract — credited as co-author. It is superseded rather than merged: its SQL uses CASE WHEN $n IS NULL … ELSE LEAST(col, $n), which gives PostgreSQL nothing to infer the parameter type from, so two of its three statements fail to prepare (asyncpg.exceptions.AmbiguousParameterError) — including the ordinary observation UPDATE, which would break every consolidation update. Its tests assert on SQL substrings against a mocked connection, and test-api is skipped on fork PRs, so nothing caught it. That is the reason the tests here execute the statements for real.

Fixes #3477

Both semantic-dedup folds rewrote only text/sources/proof_count, so an
observation could end up citing dated source facts while reporting no
event interval at all (#3477): the CREATE fold dropped the dates of the
CREATE it skipped, and the UPDATE fold dropped everything the row it
deletes knew. Both now widen the survivor's bounds, and the ordinary
UPDATE carries event_date through like the other three fields.

Diagnosis and the min/max contract come from #3482.

Co-authored-by: Sanderhoff-alt <264975235+Sanderhoff-alt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(consolidation): dated source facts can produce observations with null occurred dates

1 participant