Fix SIGSEGV in Python stub on non-graceful (parent death) termination#446
Open
mrpre wants to merge 1 commit into
Open
Fix SIGSEGV in Python stub on non-graceful (parent death) termination#446mrpre wants to merge 1 commit into
mrpre wants to merge 1 commit into
Conversation
When the parent (tritonserver) process dies abruptly -- SIGKILL after the Kubernetes termination grace period, an OOM-kill, or a crash -- it cannot send a Finalize command, so the stub's health-monitoring background thread takes the "Non-graceful termination detected" path. That path called Stub::DestroyInstance() and exit(1). Both unmap the shared-memory region (DestroyInstance via ~SharedMemoryManager -> ~mapped_region -> munmap, and exit() via static destructors) while the RunCommand loop and the model instance threads are still operating on that region and its process-shared mutexes. Unmapping the region out from under those still-running threads is a data race: a thread that touches a shm mutex after the region is unmapped faults with SIGSEGV inside libpthread (SEGV_MAPERR on the lock word). Because a whole pod's stubs share one parent, they all take this path at once and a batch of them crash simultaneously at the same instruction. The background thread cannot safely tear the region down: it cannot join the main thread (which in turn joins it), and the main thread is almost always either blocked on, or actively locking, a shm object. Since the process is exiting anyway and the region is owned and reclaimed by the parent (the kernel reclaims all mappings on process exit), terminate immediately with _exit() so that no destructors run. Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Python backend stub processes segfault when the parent
tritonserverprocess goes away abruptly (SIGKILL after the K8s grace period, OOM-kill, or
a crash). In production this appears as bursts of identical crashes across
many stubs of the same model:
error 6= user-mode write to an unmapped page; the fault address is thein-shm
bi::interprocess_mutex.Root cause
The stub's health-monitoring background thread (
pb_stub.cc,main()) pollsParentProcessActive(parent_pid). When the parent is gone it took the"Non-graceful termination detected" branch, which called
Stub::DestroyInstance()andexit(1). Both unmap the shared-memory region(
~SharedMemoryManager -> ~mapped_region -> munmap, and again via the staticdestructors run by
exit()), while theRunCommandloop and the modelinstance threads are still using that region and its process-shared
mutexes/semaphores. Unmapping it under them is a data race that faults them
with
SIGSEGV(SEGV_MAPERR) insidepthread_mutex_*.It is intermittent because it only crashes when another thread happens to be
in the userspace lock/unlock of the shm mutex during the window between the
munmapand the process actually exiting — which matches the batchy,intermittent nature seen in production.
Fix
On non-graceful termination the background thread cannot safely destroy the
region (it cannot join the main thread, which is almost always blocked on or
holding a shm lock). Since the process is exiting and the parent owns and
recreates the region, terminate immediately with
_exit(1)and run nodestructors.