Calling Pardiso 6 in Python

Question

I'm trying to use Pardiso 6 sparse solver library in Python. The problem is that I can't seem to load the Pardiso shared object (SO). Here's the error that I get when calling

import ctypes
pardiso = ctypes.CDLL(pardiso_so_address)

Traceback (most recent call last):
  File "test.py", line 27, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/idp/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: sgetrf_

I'd really appreciate it if someone could shed some light on this.

---

PS. I already contacted Pardiso developers and they told me that I need to link against optimized BLAS, but I already have MKL installed via conda.

---

Update 1: I installed mkl via conda, but it didn't help. Strangely, I added import scipy to the header and the error went away. The same thing happens if I add import mkl. So, for some reason, unless scipy or mkl are manually imported, the .so doesn't know that a lapack installation exists. Anyway, now another error is thrown, which I think might be related the libgfortran library. Here's the error

Traceback (most recent call last):
  File "test.py", line 34, in <module>
    pardiso = ctypes.CDLL(lib720)
  File "/home/amin/anaconda3/envs/test/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: ./libpardiso600-GNU720-X86-64.so: undefined symbol: _gfortran_st_close

I double-checked to see if libgfortran is installed, and indeed it is:

(test) PyPardisoProject$ ldconfig -p | grep libgfortran
    libgfortran.so.5 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.5
    libgfortran.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libgfortran.so.4

I think something similar might be at play, i.e. the library is there but it needs to be triggered (similar to what import scipy seems to have done for liblapack, but I have no idea how I can trigger it.

Note: I found an example in C on Pardiso website and tested the .so file against it via

$ gcc pardiso_sym.c -o pardiso_sym -L . -lpardiso600-GNU720-X86-64 -llapack -fopenmp -lgfortran
$ OMP_NUM_THREADS=1 ./pardiso_sym 

and it worked with no problem (with the existing libraries on my machine). So, the .so works, it's just that I don't know how to inform it of its dependencies in Python.

Update 2: Here's the output of ldd pardiso_sym:

Scripts$ ldd pardiso_sym
    linux-vdso.so.1 (0x00007ffe7e982000)
    libpardiso600-GNU720-X86-64.so (0x00007f326802d000)
    liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 (0x00007f3267976000)
    libgfortran.so.4 => /lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f3267795000)
    libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f326775b000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3267568000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3267545000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f32673f6000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f32685df000)
    libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 (0x00007f3267389000)
    libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007f32670e9000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f32670cf000)
    libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f3267083000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f326707d000)

So, I added the common path, i.e. /lib/x86_64-linux-gnu and /lib64 to PATH and ran the Python script again via:

PATH=$PATH:/lib/x86_64-linux-gnu:/lib64 python padiso_script.py

but the same error is thrown. I also tried adding to LD_LIBRARY_PATH as well, but didn't work either.

Answer 1

The trick is, rather than adding the location of dependencies to system PATHs, you need to explicitly load the dependencies, i.e. lapack, blas, and gfortran in the Python script prior to loading the Pardiso library. Also, it's essential that you explicitly pass the optional mode=ctypes.RLTD_GLOBAL argument to ctypes.CDLL method in order to make the dependencies globally accessible and hence, Pardiso can access them.

import ctypes
import ctypes.util

shared_libs = ["lapack", "blas", "omp", "gfortran"]
for lib in shared_libs:
    # Fetch the proper name of the dependency
    libname = ctypes.util.find_library(lib)
    # Load the dependency and make it globally accessible
    ctypes.CDLL(libname, mode=ctypes.RTLD_GLOBAL)
# Finally, load the Pardiso library
pardiso = ctypes.CDLL(pardiso_so_address)

In my experience, if you are inside a conda environment with mkl installed, you only need to list gfortran as dependency and the rest are automatically loaded and accessible, in which case set shared_libs = ["gfortran"].

Answer 2

Pardiso 6 sparse solver depends on Lapack functions at least sgetrf, that computes an LU factorization of a general M-by-N matrix A using partial pivoting with row interchanges.

From what we read, libpardiso600-GNU720-X86-64.so is linked dynamically against a shared Lapack library. You need to provide a PATH containing one implementation.

Before launching Python, I would recommend you to play with the LD_LIBRARY_PATH and include the path to the BLAS/Lapack library you are using. It can be the netlib implementation, the ATLAS implementation or the MKL implementation.

LD_LIRARY_PATH=$LD_LIRARY_PATH:/my_path_to_lapack \
python -c"import ctypes; pardiso = ctypes.CDLL(pardiso_so_address)"

If you use conda, you can install with the command

conda install -c anaconda mkl 

In this case, the installation may directly solves the problem.

Answer 3

Pardiso 6 and Intel MKL Pardiso are not compatible as they have different API. You may try to remove MKL from your systems paths, add OpenBLAS, and try to link your example once again.