fix: close storage when application creation fails (startup hang) - #212
Open
com3run wants to merge 1 commit into
Open
fix: close storage when application creation fails (startup hang)#212com3run wants to merge 1 commit into
com3run wants to merge 1 commit into
Conversation
A ConfigurationError raised inside create_application() left the aiosqlite connection pool running. Its connections are non-daemon threads, so the subsequent sys.exit(1) blocked in wait_for_thread_shutdown() and the process hung at 0% CPU instead of exiting. Supervisors read that as a healthy service: launchd reports "running" with no exit code, so KeepAlive never fires, and systemd Restart= behaves the same way. Guard component creation so storage is closed before the exception propagates. This covers every failure between pool startup and run_application(), which owns the only other close() call. Add regression tests asserting storage is closed and that no non-daemon threads survive a failed startup. Fixes RichardAtCT#211 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #211.
Problem
A
ConfigurationErrorraised insidecreate_application()never terminates the process.sys.exit(1)runs, but interpreter finalization blocks forever on the aiosqlite connection pool's non-daemon threads, which nothing closes on that path.The failure mode is worse than a crash: supervisors see a live PID and treat the service as healthy.
launchdreportsstate = runningwithlast exit code = (never exited), soKeepAlivenever fires. systemdRestart=behaves the same way, which also affects the setup inSYSTEMD_SETUP.md.Fix
create_application()callsstorage.initialize()early, but the onlyawait storage.close()lives inrun_application()'s shutdown block. Any failure in between leaves the pool running.This wraps component creation so storage is closed before the exception propagates. It covers every failure between pool startup and
run_application(), not just the auth one, so the twoENABLE_PROJECT_THREADSerrors are handled as well.I used
except BaseExceptiondeliberately, soCancelledErrorduring startup cleans up too.The diff is mostly re-indentation.
git diff -wshows the real change is 11 lines:storage = Storage(config.database_url) await storage.initialize() + try: # Create security components providers = [] ... "security_validator": security_validator, } + except BaseException: + # Storage has already started aiosqlite's connection pool, whose + # threads are non-daemon. run_application() owns the only other + # close() call, and it never runs if we fail here, so those threads + # would keep the interpreter alive: sys.exit() then blocks forever + # in wait_for_thread_shutdown() and the process hangs instead of + # exiting. Supervisors read that as a healthy service. + logger.debug("Closing storage after failed application creation") + await storage.close() + raiseOne unrelated formatting change came along: the extra indent pushed
audit_storage = InMemoryAuditStorage() # TODO: ...past 88 chars, and black wrapped it into an awkward parenthesised form. I moved theTODOcomment onto its own line above instead, which keeps black happy and reads better.Tests
New
tests/unit/test_main.pywith two tests:storage.close()is awaited whencreate_application()fails.Both fail on
mainand pass with the fix. Worth noting: when run againstmain, the pytest process itself hangs at interpreter shutdown, which is the bug reproducing inside the test runner.End-to-end check with a config that has no auth providers:
Verification
poetry run pytest tests/: 532 passedblack --check,isort --check-only,flake8: cleanmypy src: 512 errors before and after, so nothing new. All pre-existing and not gated in CI.I left
CHANGELOG.mdalone, since entries there look like they get written at release time with PR numbers. Happy to add one under[Unreleased]if you would rather have it in the PR.