How to use a compiled shared library from one Cython extension in another extension?

Question

I am building a mathematical library. Most of the code is implemented in Python using numpy and scipy, but some computational heavy parts like simulation models or random processes sampling are written in C++ and integrated into Python using Cython.

All the C++ code is organized in multiple libraries. Most common parts are in a single core library, while custom models reside in separate libraries. The C++ code is built in such a way, that users may use it without Python if they only need the simulation models or the core functions. If users need, say, model A, then they need to link their executable against libmodel_A and libcore.

The question is, is it possible to reflect separate C++ libraries with Cython, without the need to include the core library into each separate extension?

---

To make things clear, here is a minimal example (the example code is also available at github: https://github.com/larioandr/dummy-cython-exts-deps).

.
├── cpp_src
│   ├── app
│   │   ├── app.cpp
│   │   └── app.h
│   ├── core
│   │   ├── core.cpp
│   │   └── core.h
│   ├── main.cpp
│   └── Makefile
├── dummy
│   ├── app
│   │   ├── __init__.pxd
│   │   ├── py_app.pxd
│   │   └── py_app.pyx
│   ├── core
│   │   ├── __init__.pxd
│   │   ├── py_core.pxd
│   │   └── py_core.pyx
│   └── __init__.pxd
└── setup.py

The C++ code is inside cpp_src, the Python package is in dummy. core is a library that can be used by itself and is also required by the app. C++ library:

• cpp_src/core/core.h:

#ifndef LIB_CORE_H
#define LIB_CORE_H

void greet();  // this function can be called by the user
int someServiceFun();  // this function is used by libapp

#endif // LIB_CORE_H

• cpp_src/core/core.cpp:

#include "core/core.h"
#include <cstdio>

void greet() { printf("Hello from core\n"); }
int someServiceFun() { return 42; }

• cpp_src/app/app.h:

#ifndef APP_APP_H
#define APP_APP_H

int getTheAnswer();

#endif

• cpp_src/app/app.cpp

#include "app/app.h"
#include "core/core.h"  // <-- here we reference libcore!

int getTheAnswer() { return someServiceFun(); }

• cpp_src/Makefile (real code uses CMake, skipping it for simplicity):

all: main core/libcore.a app/libapp.a

main: main.cpp core/libcore.a app/libapp.a
    g++ -o main main.cpp core/libcore.a app/libapp.a

core/libcore.a: core/core.cpp core/core.h
    g++ -c core/core.cpp -o core/core.o -I.
    ar rvs core/libcore.a core/core.o

app/libapp.a: app/app.cpp app/app.h core/core.h core/core.cpp
    g++ -c app/app.cpp -o app/app.o -I.
    ar rvs app/libapp.a app/app.o
    
clean:
    rm -f core/core.o app/app.o core/libcore.a app/libapp.a main

Python package dummy has the same layout as the C++ library. All __init__.pxd are empty. The library files are:

• dummy/core/py_core.pxd:

cdef extern from "core/core.h":
    cdef void greet()

• dummy/core/py_core.pyx:

from .py_core cimport greet as cpp_greet

def call_greet():
    cpp_greet()

• dummy/app/py_app.pxd:

cdef extern from "app/app.h":
    cdef int getTheAnswer()

• dummy/app/py_app.pyx:

from .py_app cimport getTheAnswer

def get_the_answer():
    return getTheAnswer()

• setup.py:

from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize

extensions = [
    Extension(
        "dummy.core.py_core", [
            "dummy/core/py_core.pyx", 
            "cpp_src/core/core.cpp"
        ], include_dirs=["cpp_src"], language="c++"),
    Extension(
        "dummy.app.py_app", [
            "dummy/app/py_app.pyx",
            "cpp_src/app/app.cpp",
            # "cpp_src/core/core.cpp"  # <-- if uncomment this, everything will work well
        ], include_dirs=["cpp_src"], language="c++")
]

setup(name="dummy", version="1.0", packages=find_packages(), zip_safe=False,
      ext_modules=cythonize(extensions))

Now, if we call the py_core extension, everything is ok:

$ python -c "from dummy.core.py_core import call_greet; call_greet()"
Hello from core

However, if we call the py_app extension, we fail since someServiceFun() is defined in libcore, called by getTheAnswer() and not found in libapp:

$ python -c "from dummy.app.py_app import get_the_answer; get_the_answer()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: ***/dummy/app/py_app.cpython-310-x86_64-linux-gnu.so: undefined symbol: _Z14someServiceFunv

---

I understand, that Cython builds shared libs, and the extension py_app know nothing about the dependencies inside cpp files (not pyx, but cpp those are wrapped by). Is it possible to provide this information to Cython in some way?

Right now I see four ways:

1. Compile libcore inside each extension (that is like adding cpp_src/core/core.cpp to the second extension in the example above). However, this means object files growth, code duplicating and other smelling stuff.

2. Compile a single monolith extension for the whole library. This way is even worse then the first one, since in real library there is a bunch of namespaces. Moreover, different models may b developed by different people, so naming collisions, as well as the monolith Cython extension, will become a mess very soon.

3. Compile cppsrc manually and link explicitly libcore and libapp into the extensions appropriately by calling Extension(..., libraries=["libcore", "libapp"]). The key downside of this approach is that then I need to take into account compilation under various platforms (Windows, Linux, OS X) and difficulties with distribution. Since most users know only Python, I want to minimize the installation complexity to something like pip install my_library.

4. Manually referencing the shared lib. Not a good hack, since the library names are subject to change between different platforms and architectures. And actually I failed doing this (maybe doing something wrong, but including libraries=["py_core.cpython-310-x86_64-linux-gnu"], library_dirs=["dummy/core"] caused linking error due to the library not found).

What I actually want, is to say "Hey, Cython, if you need something from py_app extension, please, load also a shared lib from the py_core extension and find all the missing symbols in it". Is it possible?