Skip to content

feat: Add health monitoring and fallback system#232

Open
esafwan wants to merge 2 commits into
developfrom
kimi/vector-store-health-fallback
Open

feat: Add health monitoring and fallback system#232
esafwan wants to merge 2 commits into
developfrom
kimi/vector-store-health-fallback

Conversation

@esafwan

@esafwan esafwan commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements health monitoring and automatic fallback system for vector backends.

Features

  • BackendHealthMonitor - Health check caching (5-min TTL), status tracking
  • FallbackRetriever - Automatic failover to SQLite on backend failure
  • Scheduled job - Background health checks every 5 minutes

Safety

  • Graceful degradation to SQLite
  • Health status visible in DocType
  • Configurable fallback per knowledge source

Files

  • huf/ai/knowledge/health_monitor.py
  • huf/ai/knowledge/fallback_retriever.py
  • huf/huf/scheduled_job_type/health_check_backends/

Related

Ensures reliability when using external vector databases.

esafwan added 2 commits March 28, 2026 11:04
- PRD: Programmable Memory & Learning Layer
- DocType designs: Memory Record, Policy, Profile
- Tech specs: Capture/Retrieval, Storage Architecture
- Profiles: Travel Planning, Programming, Documentation
- Implementation plan and project tracking
…r-database support

- Add BackendHealthMonitor with 5-minute TTL caching
- Add FallbackRetriever with automatic backend failover
- Add scheduled job definition for periodic health checks (every 5 minutes)
- Add health_check, supports_filters, supports_hybrid_search methods to sqlite_fts backend
- Update knowledge search tool to use FallbackRetriever

Files added/modified:
- huf/ai/knowledge/health_monitor.py
- huf/ai/knowledge/fallback_retriever.py
- huf/huf/scheduled_job_type/health_check_backends/health_check_backends.json
- huf/ai/knowledge/tool.py
- huf/ai/knowledge/backends/sqlite_fts.py
@esafwan

esafwan commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

TODO before this PR is taken up: rebuild on the PR #280 contract

Since this PR was opened, HUF established, in PR #280 (in review), the KnowledgeBackend contract and get_advanced_config_schema(). Health monitoring and fallback are still valid future features, but they should be expressed as capabilities on the backend contract (audit P2 item 9), not as a parallel BackendHealthMonitor + FallbackRetriever system that duplicates backend initialization and bypasses the standard retriever.

This PR also imports BackendFactory from factory.py in the new pgvector_backend.py, which is not the chosen registry design.

  • Remove huf/ai/knowledge/backends/factory.py usage from pgvector_backend.py; register pgvector via get_backend() in backends/__init__.py instead.
  • Refactor BackendHealthMonitor so it delegates to a health_check() method on KnowledgeBackend rather than maintaining its own backend lifecycle; cache results via Frappe cache if needed.
  • Express supports_filters / supports_hybrid_search as contract capability flags and make fallback_retriever.py consult them before applying filters or hybrid logic.
  • Wire fallback into huf/ai/knowledge/retriever.py or keep it as an opt-in wrapper in huf/ai/knowledge/tool.py; avoid forcing fallback on every knowledge search.
  • Remove or make optional the scheduled health-check job in huf/huf/scheduled_job_type/health_check_backends/ until the feature is proven; ensure it does not spam logs on SQLite-only sites.
  • Rebase onto post-feat: add PGVector knowledge source #280 code and pass the live e2e test bar (UI save → index → query with primary and simulated fallback).

See the cross-backend audit (P2 item 9) and pgvector_backend.py / chroma_backend.py (PR #280) as reference implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant