docs: deduplicate=True merges distinct-but-similar records — ids are upsert keys, not merge guards

[silversurfer562]

Summary

create_long_term_memory(..., deduplicate=True) (the default) performs semantic deduplication that can MERGE distinct-but-similar records — and supplying distinct ids does not prevent the merge. The id acts as an upsert key (identical re-write → same id → no duplicate), not a merge guard; the merge decision is content-driven.

How it bit us

An accumulate-style store (session findings, several per day, often similar phrasing) lost records to merges despite unique ids. The only reliable guard was deduplicate=False + caller-supplied stable ids (Smart-AI-Memory/attune-ai#666).

A diagnosis caveat worth passing on: eventual indexing + the relevance cutoff in semantic search make this easy to misread in probes — counting via empty-text search until the count stabilizes was the only reliable verification method we found.

Ask

Document the dedup semantics explicitly: (a) semantic, content-driven; (b) ids don't opt a record out; (c) accumulate-style consumers should pass deduplicate=False. A per-record opt-out (or id-respecting mode) would be a nice enhancement but the doc note alone prevents the data-loss surprise.

Server + client 0.14.0, redis-stack, Ollama embeddings. Happy to PR the docs.

[hiSandog]

The docs should explicitly separate two independent behaviors: id selects the upsert target, while deduplicate performs semantic merge across records regardless of ID. A short accumulate-style example with two similar records and distinct IDs, showing the result with the flag on and off, would make the data-loss risk concrete. If a per-record override is added later, keeping the default visible in the response/trace would also aid diagnosis.

[hegu-1]

The key ambiguity is that deduplicate currently covers two different contracts: identity idempotency and semantic consolidation. I would separate them in both naming and response shape.

For example:

• upsert_by_id: same stable id updates/reuses one record;
• semantic_merge: similar content may create a consolidation candidate;
• append: always create an independent observation.

For accumulate-style stores, append should be explicit and safe. For semantic merge, the server should return a receipt even when the API reports success: created, updated, or merged_into, plus target id, similarity/evidence, policy version, and source ids. That makes silent record loss observable.

I would also avoid auto-merging mutable facts with similar wording unless their subject/predicate/scope match. "Alice prefers blue" and "Bob prefers blue" are semantically close but must never merge across owner scope.

Tests should cover unique ids with near-identical text, same text across users/agents, repeated writes, changed embeddings, and enumeration after eventual indexing settles. A dry-run/candidate mode would make semantic consolidation reviewable without weakening the fast append path.

This is consistent with the Personal AI OS distinction between append-only trace and replaceable snapshot: similar content does not imply the same persistence semantics: https://github.com/hegu-1/personal-ai-os/blob/main/docs/continuity-snapshot-trace.md