fix(embed): make daemon health-check timeout configurable (default 30s, was 2s) - #3402
Open
handnewb wants to merge 2 commits into
Open
fix(embed): make daemon health-check timeout configurable (default 30s, was 2s)#3402handnewb wants to merge 2 commits into
handnewb wants to merge 2 commits into
Conversation
The embed daemon liveness check used a hardcoded 2-second HTTP timeout on /health. When the daemon's asyncio event loop was blocked by a slow LLM call (measured 17.4s for consolidation against custom providers), /health could not respond within 2s. The embed manager then misclassified the daemon as dead and force-killed + restarted it. Changes: - New HINDSIGHT_EMBED_DAEMON_HEALTH_TIMEOUT env var (default 30.0s). - is_running() and _port_health_ok() both use the configurable timeout instead of the hardcoded 2s. The default of 30s is set above typical LLM latency (15-20s) to avoid false positives while still detecting truly dead daemons. Operators whose provider is known to be faster can lower this to detect outages sooner. Closes vectorize-io#3099.
handnewb
force-pushed
the
fix/daemon-health-timeout-configurable
branch
from
August 11, 2026 21:28
3f8d089 to
43c9ca8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3099: The embed daemon liveness check used a hardcoded 2-second HTTP timeout on
/health. When the daemon's asyncio event loop was blocked by a slow LLM call (measured 17.4s for consolidation against custom providers),/healthcould not respond within 2s. The embed manager then misclassified the daemon as dead and force-killed + restarted it — even though it was merely busy.Root Cause
Three hardcoded
timeout=2(and onetimeout=1) values in the health-check path:is_running()line 198timeout=2_port_health_ok()line 382timeout=2profile_manager._check_daemon_running()line 578timeout=1Any LLM call that blocked the event loop for > 2s triggered a false kill.
Changes
HINDSIGHT_EMBED_DAEMON_HEALTH_TIMEOUT(default 30.0s) — configurable, above typical LLM latency.is_running()and_port_health_ok()useDAEMON_HEALTH_TIMEOUTinstead of hardcoded 2s.profile_manager._check_daemon_running()bumped from 1s to 2s for consistency (less critical path).Impact
HINDSIGHT_EMBED_DAEMON_HEALTH_TIMEOUT=5to detect dead daemons sooner._wait_for_port_healthis separate.