fix(consolidation): keep source dates when observations merge - #3500
Open
nicoloboschi wants to merge 1 commit into
Open
fix(consolidation): keep source dates when observations merge#3500nicoloboschi wants to merge 1 commit into
nicoloboschi wants to merge 1 commit into
Conversation
Both semantic-dedup folds rewrote only text/sources/proof_count, so an observation could end up citing dated source facts while reporting no event interval at all (#3477): the CREATE fold dropped the dates of the CREATE it skipped, and the UPDATE fold dropped everything the row it deletes knew. Both now widen the survivor's bounds, and the ordinary UPDATE carries event_date through like the other three fields. Diagnosis and the min/max contract come from #3482. Co-authored-by: Sanderhoff-alt <264975235+Sanderhoff-alt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3477: an observation could cite dated source facts while reporting
occurred_startandoccurred_endasnull.Reproduced end-to-end against a real database. The cause is the semantic-dedup folds, not the ordinary write paths:
_dedup_reconcile_create) — when a CREATE is merged into a near-duplicate twin instead of being inserted, the foldUPDATErewrote onlytext,source_memory_ids,proof_countandupdated_at. The dates the skipped CREATE was going to be stamped with were dropped, so the twin gained a dated source fact and kept its own (often empty) interval. Dedup is on by default (consolidation_dedup_threshold0.97), so this is the default path._dedup_reconcile_update) — same fold, and here the folded-from row is then deleted, so anything only it knew about was lost outright._execute_update_action) — this one already mergedoccurred_start/occurred_end/mentioned_atcorrectly (LEAST/GREATESTignore NULLs in Postgres), but never touchedevent_date, leaving it at whatever the observation was stamped with when it was created. Now it carriesevent_datetoo, so all four fields follow the same rule everywhere.The rule is the one
_aggregate_source_fieldsalready documents and CREATE already applies:event_date/occurred_startkeep the earliest known value,occurred_end/mentioned_atthe latest, and a missing value on either side is ignored. A merge can only ever widen an observation's interval._TemporalBoundsnames that contract in one place; the SQL paths express it asLEAST(col, COALESCE(x, col))/GREATEST(col, COALESCE(x, col))— the idiom already in this file, which is also NULL-safe on Oracle (a bareLEAST(col, $n)would wipe the column there when the parameter is NULL).Verification
tests/test_consolidation_temporal_merge.pycovers each merge path. The three SQL-executing tests run against a real database on purpose — the merge lives in SQL, and a mocked connection cannot tell a working statement from one PostgreSQL refuses to plan. Each was confirmed to fail when its merge clause is removed:test_dedup_create_fold_widens_bounds_of_the_twinrun_consolidation_job, dedup foldtest_update_widens_bounds_from_its_source_factstest_dedup_update_fold_unions_the_bounds_of_both_rowstest_temporal_bounds_merge_keeps_the_widest_known_intervaltest_temporal_bounds_of_reads_a_source_aggregationtest_store_owned_fold_merges_bounds_like_the_sql_pathThe first one is the issue's exact scenario: an undated observation, a dated source fact whose CREATE folds into it, and the survivor must come out with the source's interval.
Also ran the whole
hindsight-api-slimsuite locally (non-LLM markers): the consolidation/observation/temporal modules are green, and the handful of unrelated failures left reproduce identically on unmodifiedmain. Plus./scripts/hooks/lint.sh,check-unused.shandty.Note on #3482
#3482 diagnosed this first and proposed the same min/max contract — credited as co-author. It is superseded rather than merged: its SQL uses
CASE WHEN $n IS NULL … ELSE LEAST(col, $n), which gives PostgreSQL nothing to infer the parameter type from, so two of its three statements fail to prepare (asyncpg.exceptions.AmbiguousParameterError) — including the ordinary observation UPDATE, which would break every consolidation update. Its tests assert on SQL substrings against a mocked connection, andtest-apiis skipped on fork PRs, so nothing caught it. That is the reason the tests here execute the statements for real.Fixes #3477