fix(engine): drop null metadata values at retain and recall (#3209) - #3531
fix(engine): drop null metadata values at retain and recall (#3209)#3531NovaLux12 wants to merge 1 commit into
Conversation
Retain accepts arbitrary JSON metadata; a null value (e.g. {"ocr_engine":
null}) stored verbatim made every read path that returns MemoryFact fail
dict[str, str] validation — recall, consolidation, and mental-model refresh
errored for the affected bank. Normalize on both ends: RetainContent drops
null-valued keys at construction (canonical storage) and
MemoryFact.parse_metadata drops them for legacy rows, preserving the
existing string coercion for other non-string values.
Closes vectorize-io#3209
|
One write-side gap remains: delta updates bypass RetainContent normalization.
In both cases, documents.retain_params.metadata should preserve the null value, while memory_units.metadata should omit the null-valued key. |
Closes #3209
Problem
MemoryFact.metadatais typeddict[str, str] | None, but retain acceptsarbitrary user-supplied JSON metadata. A null value (e.g.
{"ocr_engine": null}) was stored verbatim, and every read path that returnsMemoryFact— recall, consolidation, mental-model refresh — then failedPydantic validation (
Input should be a valid string), permanently wedgingthe affected bank.
Change
Normalize on both ends, per the plan in the issue:
RetainContent.__post_init__drops null-valued metadatakeys at construction, so facts extracted from it (
metadata=content.metadatain
fact_extraction) are stored canonically. Single chokepoint: covers themain retain path, delta retain, recovery, and transfer import. Document
metadata in
retain_paramsis untouched (built from the raw request dicts).MemoryFact.parse_metadatadrops null-valued keys insteadof coercing them to the string
"None", so already-stored legacy rowsbecome readable again with no data migration. Existing string coercion for
other non-string values is preserved.
Verification
tests/test_response_models.py(read path): null metadata value omitted,null inside a JSONB-string is omitted, all-null becomes
{}.tests/test_retain_metadata_normalization.py(write path):RetainContentdrops nulls, keeps non-string values as-is, and
_build_contents(the APIingestion funnel) normalizes a null-metadata request.
uv run pytest tests/test_response_models.py tests/test_retain_metadata_normalization.py tests/test_retain_orchestrator_mapping.py -q→ 27 passed.test_large_document_replacement.py,test_retain_entity_prune_race.py,test_validation_result_enrichment.py,test_response_exclude_none.py): 16 passed; the 9 errors are all thesame pre-existing environmental
ImportError: pg0-embedded is required for embedded PostgreSQL(optionalembedded-dbextra not installed in thisvenv), unrelated to this change.
ruff check,ruff format --check, andty checkpass on all changed files.