feat(decay): native usage-aware memory decay engine + run_decay_scan MCP tool

[bertheto]

Summary

Adds a native, usage-aware memory decay engine so memory confidence and obsolescence are managed by the server instead of relying on agent-side discipline. Closes the gap surfaced by the 2026 open-source memory MCP landscape review: of 73 memory systems surveyed, only 2 combine decay with scheduled execution, and none track usage as a decay weight.

What changed

Schema (additive, forward-compatible)

• memories gains access_count (int, default 0, NOT NULL) and last_accessed_at (datetime, nullable) + ix_memories_last_accessed_at.
• Alembic migration 20260704_add_memory_usage_tracking (upgrade + downgrade both clean; validated on a DB copy before live application).
• Dual-repo: SQLite + Postgres MemoryTable ORM extended identically; MemoryRepository Protocol gains record_memory_access and get_decay_candidates.
• Memory Pydantic model exposes the new fields read-only; MemoryUpdate does not — access counters stay internal, updated_at is never distorted by read-side tracking.

Service (app/services/memory_service.py)

• record_memory_access increments access_count + sets last_accessed_at via a dedicated UPDATE that never mutates updated_at.
• Read paths query_memory (post-truncation) and get_memory call it best-effort (try/except, never breaks the read).
• run_decay_scan returns a DecayScanResponse with per-memory DecayCandidateAction (skip / decay / obsolete, delta, proposed_confidence, reason).

Decay formula (_propose_action, classmethod for pure unit-testability):

• effective_access = last_accessed_at ?? updated_at ?? created_at
• SKIP if importance >= 8 OR tags in {decision, architecture, critical}
• SKIP if confidence is None (no safe target without a default-confidence policy)
• SKIP if age <= 90 days
• OBSOLETE if age > 180 days AND confidence <= 0.3
• DECAY: delta = clamp(0.1 - min(access_count, 8) * 0.01, 0.02, 0.1); proposed = max(0.3, confidence - delta); if clamped at floor → skip

MCP tool

• run_decay_scan(ctx, memory_ids=None, project_id=None, dry_run=True) registered in the memory category, following the existing rebuild_embeddings triad pattern (memory_tools.py + tool_adapters.py + tool_metadata_registry.py).
• dry_run=True previews; dry_run=False applies via existing update_memory / mark_memory_obsolete (never raw SQL).
• mutates: True; tags: memory, decay, admin, lifecycle, gc.

Tests (23 new, all green)

• tests/integration/test_decay_engine.py (16): pure-formula unit tests (protected tags/importance, null confidence, boundary ages 89/90/180+, usage-weight shrink, clamped floor, obsolete path) + integration via InMemoryMemoryRepository (updated_at preservation, get_memory access tracking, dry-run no-write, live confidence decay, live obsolete).
• tests/integration/test_decay_tool_adapters.py (4): adapter registered, registry metadata bound, dry-run returns DecayScanResponse.
• tests/e2e_sqlite/test_decay_scan_sqlite.py (3): discover_forgetful_tools lists it, execute_forgetful_tool dry-run + live round-trip.
• tests/integration/conftest.py: InMemoryMemoryRepository stubs the two new repo methods.

============================= 23 passed in 4.42s ==============================

Backward compatibility

• New columns have safe defaults (access_count = 0, last_accessed_at = NULL), so existing memories decay as if never accessed — the effective_access fallback to updated_at/created_at keeps the age signal intact.
• No existing tool signature changes; run_decay_scan is purely additive to the tool registry.
• Migration downgrade drops the columns + index cleanly.

Test plan

• pytest tests/integration/test_decay_engine.py tests/integration/test_decay_tool_adapters.py tests/e2e_sqlite/test_decay_scan_sqlite.py → 23 passed
• Postgres E2E (not run locally; SQLite + Postgres ORM are extended identically)
• Maintainer review of decay constants (90d fresh-window, 180d obsolete-threshold, 0.3 floor, 0.02–0.1 delta band) — these are centralized in memory_service.py as module constants and easy to tune.

[ScottRBK]

Thanks for the PR, there's clearly a lot of work in here, and memory lifecycle is a real topic
worth tackling.

I'm still thinking about how I want to handle this in Forgetful, so I'm not going to move on the
PR just yet. Where my thinking currently sits: I'm not in favour of time-based decay. Access
recency/frequency isn't a reliable proxy for truth or value in a semantic store, a memory
doesn't become less true because it hasn't been queried for 90 days. I'm more drawn to something
purposeful and agent-driven: monitoring for contradictions, explicit supersession/invalidation of
facts that have actually been superseded, that kind of mechanism, rather than a clock.

I haven't landed on an approach yet though, so I'll leave this open while I think it through, and
I'll follow up here once I've settled on a direction.

[bertheto]

Follow-up: naive SQLite datetime fix

Live run_decay_scan failed every candidate with TypeError: can't subtract offset-naive and offset-aware datetimes because SQLite returns naive timestamps while the scan uses datetime.now(UTC).

This commit adds MemoryService._as_utc / UTC-aware _effective_access_date and mirrors the guard in SQLite + Postgres get_decay_candidates. New unit test covers naive model_construct rows. 17 decay-engine tests green.

[bertheto]

Thanks for the clear product signal. I agree access recency is a weak proxy for truth in a semantic store.

I'll leave this PR open with green CI as a reference implementation. Next I'll open a smaller follow-up that keeps only usage tracking (access_count / last_accessed_at on read paths) and drops time-based decay entirely.

Happy to help design contradiction detection / explicit supersession when you settle on a direction. No rush on my side.

[bertheto]

Follow-up opened: #53

Telemetry-only slice (access_count / last_accessed_at on read paths). No time-based decay or run_decay_scan. Leaving this PR open as the reference implementation.

[bertheto]

Hey Scott, quick check-in when you have a moment.

Your July note on contradiction and explicit supersession still matches how I think about this on my side. No rush on #45 from me, it stays parked as reference.

If it helps, I can draft a short supersession RFC or open a design-only issue. Happy either way.