Skip to content

fix: close storage when application creation fails (startup hang) - #212

Open
com3run wants to merge 1 commit into
RichardAtCT:mainfrom
com3run:fix/startup-config-error-hang
Open

fix: close storage when application creation fails (startup hang)#212
com3run wants to merge 1 commit into
RichardAtCT:mainfrom
com3run:fix/startup-config-error-hang

Conversation

@com3run

@com3run com3run commented Aug 11, 2026

Copy link
Copy Markdown

Fixes #211.

Problem

A ConfigurationError raised inside create_application() never terminates the process. sys.exit(1) runs, but interpreter finalization blocks forever on the aiosqlite connection pool's non-daemon threads, which nothing closes on that path.

The failure mode is worse than a crash: supervisors see a live PID and treat the service as healthy. launchd reports state = running with last exit code = (never exited), so KeepAlive never fires. systemd Restart= behaves the same way, which also affects the setup in SYSTEMD_SETUP.md.

Fix

create_application() calls storage.initialize() early, but the only await storage.close() lives in run_application()'s shutdown block. Any failure in between leaves the pool running.

This wraps component creation so storage is closed before the exception propagates. It covers every failure between pool startup and run_application(), not just the auth one, so the two ENABLE_PROJECT_THREADS errors are handled as well.

I used except BaseException deliberately, so CancelledError during startup cleans up too.

The diff is mostly re-indentation. git diff -w shows the real change is 11 lines:

     storage = Storage(config.database_url)
     await storage.initialize()

+    try:
         # Create security components
         providers = []
...
             "security_validator": security_validator,
         }
+    except BaseException:
+        # Storage has already started aiosqlite's connection pool, whose
+        # threads are non-daemon. run_application() owns the only other
+        # close() call, and it never runs if we fail here, so those threads
+        # would keep the interpreter alive: sys.exit() then blocks forever
+        # in wait_for_thread_shutdown() and the process hangs instead of
+        # exiting. Supervisors read that as a healthy service.
+        logger.debug("Closing storage after failed application creation")
+        await storage.close()
+        raise

One unrelated formatting change came along: the extra indent pushed audit_storage = InMemoryAuditStorage() # TODO: ... past 88 chars, and black wrapped it into an awkward parenthesised form. I moved the TODO comment onto its own line above instead, which keeps black happy and reads better.

Tests

New tests/unit/test_main.py with two tests:

  1. storage.close() is awaited when create_application() fails.
  2. No non-daemon threads survive a failed startup. This is the real regression guard, since surviving threads are what cause the hang.

Both fail on main and pass with the fix. Worth noting: when run against main, the pytest process itself hangs at interpreter shutdown, which is the bug reproducing inside the test runner.

End-to-end check with a config that has no auth providers:

### WITHOUT fix:  RESULT: STILL RUNNING after 20s (HANG)
### WITH fix:     RESULT: exited cleanly, code 1

Verification

  • poetry run pytest tests/: 532 passed
  • black --check, isort --check-only, flake8: clean
  • mypy src: 512 errors before and after, so nothing new. All pre-existing and not gated in CI.

I left CHANGELOG.md alone, since entries there look like they get written at release time with PR numbers. Happy to add one under [Unreleased] if you would rather have it in the PR.

A ConfigurationError raised inside create_application() left the aiosqlite
connection pool running. Its connections are non-daemon threads, so the
subsequent sys.exit(1) blocked in wait_for_thread_shutdown() and the process
hung at 0% CPU instead of exiting. Supervisors read that as a healthy service:
launchd reports "running" with no exit code, so KeepAlive never fires, and
systemd Restart= behaves the same way.

Guard component creation so storage is closed before the exception propagates.
This covers every failure between pool startup and run_application(), which
owns the only other close() call.

Add regression tests asserting storage is closed and that no non-daemon
threads survive a failed startup.

Fixes RichardAtCT#211

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Startup ConfigurationError never terminates the process: sys.exit(1) blocks forever in wait_for_thread_shutdown()

1 participant