Make `py_zstd_compress_mt2` thread-safe by KowalskiThomas · Pull Request #321 · sergey-dryabzhinsky/python-zstd

KowalskiThomas · 2026-06-19T12:54:10Z

What is this PR?

This PR is a fix for a crash we witnessed in a service of ours. py_zstd_compress_mt2 used a single global compression context (at the module level), m_cctx, which is thus shared across all calls:

python-zstd/src/python-zstd.h

Line 130 in fec54dd

static ZSTD_CCtx* m_cctx;

python-zstd/src/python-zstd.c

Lines 279 to 281 in fec54dd

    
               return NULL; 
        
           } 
        
           ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, level);

Since ZSTD_CCtx is not thread-safe, and because compression releases the GIL, two Python threads calling these functions at the same time use the same context concurrently. This can corrupt the state and lead to out-of-bounds access inside zstd.

The crash was detected in this form:

ZSTD_compressBlock_doubleFast_noDict_generic   zstd/lib/compress/zstd_double_fast.c:185
ZSTD_compressBlock_doubleFast_noDict_4         zstd/lib/compress/zstd_double_fast.c:557
ZSTD_compressBlock_doubleFast                  zstd/lib/compress/zstd_double_fast.c:577
ZSTD_buildSeqStore                             zstd/lib/compress/zstd_compress.c:3420
ZSTD_compressBlock_internal                    zstd/lib/compress/zstd_compress.c:4399
ZSTD_compress_frameChunk                       zstd/lib/compress/zstd_compress.c:4646
ZSTD_compressContinue_internal                 zstd/lib/compress/zstd_compress.c:4834
ZSTD_compressEnd_public                        zstd/lib/compress/zstd_compress.c:5412
ZSTD_compressStream_generic                    zstd/lib/compress/zstd_compress.c:6151
ZSTD_compressStream2                           zstd/lib/compress/zstd_compress.c:6540
ZSTD_compressStream2_simpleArgs                zstd/lib/compress/zstd_compress.c:6561
ZSTD_compress2                                 zstd/lib/compress/zstd_compress.c:6581
py_zstd_compress_mt2                           src/python-zstd.c:276
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault

Looking further into it, we found another race as well. When called with different level values, reset_cContext frees and recreates m_cctx. If one thread does this while another is still inside the GIL-released ZSTD_compress2, the latter compresses into a freed context.

Proposed fix

The fix proposed in this PR is to make py_zstd_compress_mt2 create and free its own ZSTD_CCtx on each call, instead of sharing the module-global one. This is the same pattern already used by py_zstd_compress_mt:

python-zstd/src/python-zstd.c

Lines 141 to 149 in fec54dd

    
           cctx = ZSTD_createCCtx(); 
        
           ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, level); 
        
           ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, threads); 
        
           Py_BEGIN_ALLOW_THREADS 
        
           cSize = ZSTD_compress2(cctx, dest, (size_t)dest_size, source, (size_t)source_size); 
        
           Py_END_ALLOW_THREADS 
        
           ZSTD_freeCCtx(cctx);

sergey-dryabzhinsky · 2026-06-20T10:22:51Z

I intentionally created context only one time, hoped it speed things up.
Is there easy way to create test for it?

KowalskiThomas · 2026-06-20T14:41:41Z

I intentionally created context only one time, hoped it speed things up.

That's what I assumed 😅

I think if you want to keep caching contexts, I could change the code to have a kind of pool for contexts and re-use them across calls while guaranteeing a single context is never used for more than one call at a time.

Is there easy way to create test for it?

If you want a reproducer, I don't know if it's possible to create one easily, not one that will consistently fail anyway (the crash is rare). Maybe under free-threaded Python, it would show up more. I can try to give it a look.

sergey-dryabzhinsky · 2026-06-20T18:37:18Z

a kind of pool for contexts
Then I'll look for it.

I can try to give it a look.
Shure, I'll wait.

KowalskiThomas · 2026-06-21T07:34:13Z

I have a reproducer now, which consistently triggers crashes even with the GIL (and triggers TSan, on a free-threaded TSan build). I think this could be added as a regression test after the fix is added.

import threading

import zstd

# Large payload so each call takes long enough for threads to overlap.
DATA: bytes = b"hello world this is a test " * 200_000  # 5 MB

THREADS: int = 64
ITERS: int = 500
LEVELS: list[int] = [1, 3, 5, 9, 3, 1]  # mix of levels triggers reset_cContext

errors: list[Exception] = []
lock = threading.Lock()

def compress_loop() -> None:
    for i in range(ITERS):
        level = LEVELS[i % len(LEVELS)]
        try:
            zstd.compress2(DATA, level)
        except Exception as exc:
            with lock:
                errors.append(exc)


threads = [threading.Thread(target=compress_loop) for _ in range(THREADS)]

for t in threads:
    t.start()
for t in threads:
    t.join()

Output

% python3 repro.py
<frozen importlib._bootstrap>:549: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'zstd', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.
==================
WARNING: ThreadSanitizer: data race (pid=11580)
  Read of size 8 at 0x0001139026b0 by thread T2:
    #0 ZSTD_freeCCtx <null> (zstd.cpython-316t-darwin.so:arm64+0x10610)
    #1 py_zstd_compress_mt2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1014)
    #2 cfunction_call methodobject.c:575 (python.exe:arm64+0x1001387b0)
    #3 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #4 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #5 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #6 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #8 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #9 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #10 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #11 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #12 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #13 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #14 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1002dce38)
    #15 context_run context.c:728 (python.exe:arm64+0x1002dce38)
    #16 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x1000a8644)
    #17 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100090e80)
    #18 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090e80)
    #19 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #20 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #21 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #22 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #23 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #24 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #25 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #26 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #27 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x10009112c)
    #28 _PyObject_Call call.c:348 (python.exe:arm64+0x10009112c)
    #29 PyObject_Call call.c:373 (python.exe:arm64+0x1000911a4)
    #30 thread_run _threadmodule.c:388 (python.exe:arm64+0x10044df04)
    #31 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10038a12c)

  Previous write of size 8 at 0x0001139026b0 by thread T1:
    #0 ZSTD_CCtx_init_compressStream2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1c1d4)
    #1 ZSTD_compressStream2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1bc24)
    #2 ZSTD_compress2 <null> (zstd.cpython-316t-darwin.so:arm64+0x14824)
    #3 py_zstd_compress_mt2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1090)
    #4 cfunction_call methodobject.c:575 (python.exe:arm64+0x1001387b0)
    #5 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #6 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #7 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #8 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #9 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #10 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #11 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #12 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #13 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #14 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #15 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #16 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1002dce38)
    #17 context_run context.c:728 (python.exe:arm64+0x1002dce38)
    #18 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x1000a8644)
    #19 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100090e80)
    #20 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090e80)
    #21 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #22 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #23 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #24 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #25 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #26 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #27 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #28 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #29 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x10009112c)
    #30 _PyObject_Call call.c:348 (python.exe:arm64+0x10009112c)
    #31 PyObject_Call call.c:373 (python.exe:arm64+0x1000911a4)
    #32 thread_run _threadmodule.c:388 (python.exe:arm64+0x10044df04)
    #33 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10038a12c)

  Location is heap block of size 5280 at 0x000113901800 allocated by thread T1:
    #0 calloc <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x64ea0)
    #1 ZSTD_createCCtx <null> (zstd.cpython-316t-darwin.so:arm64+0x1018c)
    #2 py_zstd_compress_mt2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1018)
    #3 cfunction_call methodobject.c:575 (python.exe:arm64+0x1001387b0)
    #4 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #5 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #6 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #7 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #8 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #9 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #10 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #11 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #12 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #13 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #14 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #15 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1002dce38)
    #16 context_run context.c:728 (python.exe:arm64+0x1002dce38)
    #17 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x1000a8644)
    #18 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100090e80)
    #19 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090e80)
    #20 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #21 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #22 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #23 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #24 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #25 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #26 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #27 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #28 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x10009112c)
    #29 _PyObject_Call call.c:348 (python.exe:arm64+0x10009112c)
    #30 PyObject_Call call.c:373 (python.exe:arm64+0x1000911a4)
    #31 thread_run _threadmodule.c:388 (python.exe:arm64+0x10044df04)
    #32 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10038a12c)

  Thread T2 (tid=21203051, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x31d84)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100389300)
    #2 PyThread_start_joinable_thread thread_pthread.h:323 (python.exe:arm64+0x100389138)
    #3 ThreadHandle_start _threadmodule.c:475 (python.exe:arm64+0x10044dd10)
    #4 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x10044d7dc)
    #5 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x10044c844)
    #6 cfunction_call methodobject.c:564 (python.exe:arm64+0x100138744)
    #7 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #8 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #9 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #10 _Py_VectorCall_StackRefSteal ceval.c:724 (python.exe:arm64+0x100290228)
    #11 _PyEval_EvalFrameDefault generated_cases.c.h:3528 (python.exe:arm64+0x100299d1c)
    #12 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fa34)
    #13 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fa34)
    #14 PyEval_EvalCode ceval.c:677 (python.exe:arm64+0x10028fa34)
    #15 run_eval_code_obj pythonrun.c:1369 (python.exe:arm64+0x100366a70)
    #16 run_mod pythonrun.c:1472 (python.exe:arm64+0x1003667cc)
    #17 pyrun_file pythonrun.c:1296 (python.exe:arm64+0x100361d38)
    #18 _PyRun_SimpleFileObject pythonrun.c:518 (python.exe:arm64+0x100361d38)
    #19 _PyRun_AnyFileObject pythonrun.c:81 (python.exe:arm64+0x1003614c8)
    #20 pymain_run_file_obj main.c:411 (python.exe:arm64+0x1003a3328)
    #21 pymain_run_file main.c:430 (python.exe:arm64+0x1003a3328)
    #22 pymain_run_python main.c:715 (python.exe:arm64+0x1003a2658)
    #23 Py_RunMain main.c:796 (python.exe:arm64+0x1003a2658)
    #24 pymain_main main.c:826 (python.exe:arm64+0x1003a2bd0)
    #25 Py_BytesMain main.c:850 (python.exe:arm64+0x1003a2cd0)
    #26 main python.c:15 (python.exe:arm64+0x100000a78)

  Thread T1 (tid=21203050, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x31d84)
    #1 do_start_joinable_thread thread_pthread.h:281 (python.exe:arm64+0x100389300)
    #2 PyThread_start_joinable_thread thread_pthread.h:323 (python.exe:arm64+0x100389138)
    #3 ThreadHandle_start _threadmodule.c:475 (python.exe:arm64+0x10044dd10)
    #4 do_start_new_thread _threadmodule.c:1919 (python.exe:arm64+0x10044d7dc)
    #5 thread_PyThread_start_joinable_thread _threadmodule.c:2042 (python.exe:arm64+0x10044c844)
    #6 cfunction_call methodobject.c:564 (python.exe:arm64+0x100138744)
    #7 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #8 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #9 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #10 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #11 _PyEval_EvalFrameDefault generated_cases.c.h:3325 (python.exe:arm64+0x10029964c)
    #12 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fa34)
    #13 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fa34)
    #14 PyEval_EvalCode ceval.c:677 (python.exe:arm64+0x10028fa34)
    #15 run_eval_code_obj pythonrun.c:1369 (python.exe:arm64+0x100366a70)
    #16 run_mod pythonrun.c:1472 (python.exe:arm64+0x1003667cc)
    #17 pyrun_file pythonrun.c:1296 (python.exe:arm64+0x100361d38)
    #18 _PyRun_SimpleFileObject pythonrun.c:518 (python.exe:arm64+0x100361d38)
    #19 _PyRun_AnyFileObject pythonrun.c:81 (python.exe:arm64+0x1003614c8)
    #20 pymain_run_file_obj main.c:411 (python.exe:arm64+0x1003a3328)
    #21 pymain_run_file main.c:430 (python.exe:arm64+0x1003a3328)
    #22 pymain_run_python main.c:715 (python.exe:arm64+0x1003a2658)
    #23 Py_RunMain main.c:796 (python.exe:arm64+0x1003a2658)
    #24 pymain_main main.c:826 (python.exe:arm64+0x1003a2bd0)
    #25 Py_BytesMain main.c:850 (python.exe:arm64+0x1003a2cd0)
    #26 main python.c:15 (python.exe:arm64+0x100000a78)

SUMMARY: ThreadSanitizer: data race (zstd.cpython-316t-darwin.so:arm64+0x10610) in ZSTD_freeCCtx+0x7c
==================

(more reports)

==================
==11580==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000118 (pc 0x0001129893f4 bp 0x000381005e30 sp 0x000381005d90 T21203175)
==11580==The signal is caused by a READ memory access.
==11580==Hint: address points to the zero page.
    #0 ZSTDMT_compressStream_generic <null> (zstd.cpython-316t-darwin.so:arm64+0x9d3f4)
    #1 ZSTD_compressStream2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1b974)
    #2 ZSTD_compress2 <null> (zstd.cpython-316t-darwin.so:arm64+0x14824)
    #3 py_zstd_compress_mt2 <null> (zstd.cpython-316t-darwin.so:arm64+0x1090)
    #4 cfunction_call methodobject.c:575 (python.exe:arm64+0x1001387b0)
    #5 _PyObject_MakeTpCall call.c:242 (python.exe:arm64+0x10009030c)
    #6 _PyObject_VectorcallTstate pycore_call.h:142 (python.exe:arm64+0x100090f14)
    #7 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090f14)
    #8 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #9 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #10 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #11 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #12 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #13 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #14 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #15 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #16 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x1002dce38)
    #17 context_run context.c:728 (python.exe:arm64+0x1002dce38)
    #18 method_vectorcall_FASTCALL_KEYWORDS descrobject.c:421 (python.exe:arm64+0x1000a8644)
    #19 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100090e80)
    #20 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x100090e80)
    #21 _Py_VectorCallInstrumentation_StackRefSteal ceval.c:766 (python.exe:arm64+0x100290968)
    #22 _PyEval_EvalFrameDefault generated_cases.c.h:1846 (python.exe:arm64+0x100295ff8)
    #23 _PyEval_EvalFrame pycore_ceval.h:122 (python.exe:arm64+0x10028fe5c)
    #24 _PyEval_Vector ceval.c:2134 (python.exe:arm64+0x10028fe5c)
    #25 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000914bc)
    #26 _PyObject_VectorcallTstate pycore_call.h:144 (python.exe:arm64+0x100092c3c)
    #27 _PyObject_VectorcallPrepend call.c:855 (python.exe:arm64+0x100092c3c)
    #28 method_vectorcall classobject.c:55 (python.exe:arm64+0x100095fdc)
    #29 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x10009112c)
    #30 _PyObject_Call call.c:348 (python.exe:arm64+0x10009112c)
    #31 PyObject_Call call.c:373 (python.exe:arm64+0x1000911a4)
    #32 thread_run _threadmodule.c:388 (python.exe:arm64+0x10044df04)
    #33 pythread_wrapper thread_pthread.h:234 (python.exe:arm64+0x10038a12c)
    #34 __tsan_thread_start_func <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x31cf4)
    #35 _pthread_start <null> (libsystem_pthread.dylib:arm64e+0x6c54)
    #36 thread_start <null> (libsystem_pthread.dylib:arm64e+0x1c18)

==11580==Register values:
 x[0] = 0x0000000115b5c000   x[1] = 0x0000100000000230   x[2] = 0x000000004d8e3f0f   x[3] = 0x0000000000000002  
 x[4] = 0x0000000000000001   x[5] = 0x0000000000000001   x[6] = 0x0000000010000000   x[7] = 0x0000000000000000  
 x[8] = 0x000000000d8e3f00   x[9] = 0x000000000d8e3f0f  x[10] = 0x0000000116ca8e28  x[11] = 0x0000000116ca8e20  
x[12] = 0x0000100000000230  x[13] = 0x0000000000000000  x[14] = 0x00000000000000ff  x[15] = 0x000000000000003f  
x[16] = 0x000000018bec6c8c  x[17] = 0x0000000000000008  x[18] = 0x0000000000000000  x[19] = 0x0000000113901800  
x[20] = 0x0000000000000000  x[21] = 0x0000000381005ea0  x[22] = 0x0000000000000002  x[23] = 0x0000000000000001  
x[24] = 0x0000000000000c0c  x[25] = 0x00000000000000e4  x[26] = 0x0000000381005ea0  x[27] = 0x00000000000000b4  
x[28] = 0x0000000381005eb8     fp = 0x0000000381005e30     lr = 0x00000001129893f4     sp = 0x0000000381005d90  
ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV (zstd.cpython-316t-darwin.so:arm64+0x9d3f4) in ZSTDMT_compressStream_generic+0x64
==11580==ABORTING
zsh: abort      python3 repro.py

sergey-dryabzhinsky · 2026-06-28T16:23:42Z

Ok, maybe there is no simple way. And it's safer to do a new context every time.
How work on context thread pool going?

sergey-dryabzhinsky · 2026-06-28T16:25:10Z

Marked 15.7.2 as last safe release.

KowalskiThomas · 2026-06-29T18:20:27Z

How work on context thread pool going?

Ah, I thought you were looking into it! I'll have a look this week :)

KowalskiThomas marked this pull request as ready for review June 19, 2026 12:56

Fix: make py_zstd_compress_mt2 thread-safe

693cd72

KowalskiThomas force-pushed the kowalski/fix-thread-safety branch from 645526c to 693cd72 Compare June 19, 2026 12:57

sergey-dryabzhinsky mentioned this pull request Jun 29, 2026

Version 1.5.7.3 are not thread safe #323

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make `py_zstd_compress_mt2` thread-safe#321

Make `py_zstd_compress_mt2` thread-safe#321
KowalskiThomas wants to merge 1 commit into
sergey-dryabzhinsky:masterfrom
KowalskiThomas:kowalski/fix-thread-safety

KowalskiThomas commented Jun 19, 2026

Uh oh!

sergey-dryabzhinsky commented Jun 20, 2026

Uh oh!

KowalskiThomas commented Jun 20, 2026

Uh oh!

sergey-dryabzhinsky commented Jun 20, 2026

Uh oh!

KowalskiThomas commented Jun 21, 2026 •

edited

Loading

Uh oh!

sergey-dryabzhinsky commented Jun 28, 2026

Uh oh!

sergey-dryabzhinsky commented Jun 28, 2026

Uh oh!

KowalskiThomas commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	return NULL;
	}
	ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, level);

	cctx = ZSTD_createCCtx();

	ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, level);
	ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, threads);
	Py_BEGIN_ALLOW_THREADS
	cSize = ZSTD_compress2(cctx, dest, (size_t)dest_size, source, (size_t)source_size);
	Py_END_ALLOW_THREADS

	ZSTD_freeCCtx(cctx);

Conversation

KowalskiThomas commented Jun 19, 2026

What is this PR?

Proposed fix

Uh oh!

sergey-dryabzhinsky commented Jun 20, 2026

Uh oh!

KowalskiThomas commented Jun 20, 2026

Uh oh!

sergey-dryabzhinsky commented Jun 20, 2026

Uh oh!

KowalskiThomas commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sergey-dryabzhinsky commented Jun 28, 2026

Uh oh!

sergey-dryabzhinsky commented Jun 28, 2026

Uh oh!

KowalskiThomas commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

KowalskiThomas commented Jun 21, 2026 •

edited

Loading