Benchmarking with Pytest Parallelized Cython Code results in Fatal Python Error

Question

I have the following test:

import array
def test_parallel_cython_clip(benchmark: Any) -> None:
    benchmark.pedantic(
        math_cython.parallel_cython_clip_vector,
        args=(array.array("f", [0.0]),-1.0,1.0,array.array("f", [0.0])),
        rounds=1,
        iterations=1
    )

that I execute via:

pytest --benchmark-columns=min,max,mean,stddev --benchmark-sort=mean benchmark.py -vv -k test_parallel_cython_clip

which results in the error: benchmark.py::test_parallel_cython_clip Fatal Python error: Aborted

Furthermore the following information is provided:

Extension modules: mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, math_cython.cython_computations (total: 16)

The package folder math_cython, which is editable-installed into the conda environment, contains a .py module with the following entry:

def parallel_cython_clip_vector(
    vector_in: array.array,
    min_value: float,
    max_value: float,
    vector_out: array.array,
) -> None:
    _parallel_cython_clip_vector(vector_in, min_value, max_value, vector_out)

and a .pyx file with the _parallel_cython_clip_vector function defined as follows:

@cython.boundscheck(False)  # Deactivate bounds checking (which is possible in Python, but not in C)
@cython.wraparound(False)  # Deactivate negative indexing (which is possible in Python, but not in C)
def _parallel_cython_clip_vector(
    float[:] vector_in,
    float min_value,
    float max_value,
    float[:] vector_out,
):
    cdef signed int idx = 0
    <this part I commented for the moment>

Operating System Info:

macOS 14.3.1 
Darwin 23.3.0

Installed Python libraries:

cython 3.0.6
pytest 8.0.0
pytest-benchmark 4.0.0
python 3.11.7
numpy 1.26.3

I saw similar questions here, but I do not think they address my issue:

• Enabling Parallelism with Cython
• Cython prange fails with Fatal Python error: PyThreadState_Get: no current thread
• Cython crashed using memoryviews

Any help to resolve the Fatal Python error would be very much appreciated.

EDIT: Running debugger in VSCode I got more information on the error:

OMP: Error #15: Initializing libomp.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
Fatal Python error: Aborted

Answer 1

The solution to my problem is:

conda install nomkl

The reasons why this is needed on MacOS is nicely explained here: https://stackoverflow.com/a/58869103/9698518

I short there is a clash between Conda's Intel MKL Optimizations and macOS's own Accelerate Framework that also comes with OpenMP.