PyBind11 : share opaque pointers between independently-built C++ modules through Python

Question

I have two PyBind11-linked Python modules which must compile separately (no common headers etc) and I would like to share pointers to a common custom C++ class defined the same way in both modules. I would like the pointers to be passed by a module to another using Python. Python does not need to understand the type passed.

For example

from ModuleA import ClassA
from ModuleB import ClassB

A= ClassA()
B= ClassB()
B.setSharedClassPointer(A.getSharedClassPointer())

How can I do that ?

I tried three things :

1/ Define the same SharedClass in both C++ modules, expose it to Python through PyBind11 in the "giver" module only, and expose the setSharedClassPointer (taking *SharedClass) in ModuleB and getSharedClassPointer (returning *SharedClass) in ModuleA.

//in  ModuleA/srcpy/bindings.cpp

class SharedClass
{
//....//
};

class ClassA{
{

ClassA(){
//...//
SharedClassPointer = new SharedClass;};
getSharedClassPointer(){return SharedClassPointer;};


SharedClass *SharedClassPointer;

};

pybind11::class_<SharedClass,std::shared_ptr<SharedClass, pybind11::nodelete>>>(m, "SharedClass")
    .def(pybind11::init<>());

pybind11::class_<ClassA>(m, "ClassA")
    .def(pybind11::init<>())
    .def("getSharedClassPointer",&ClassA::getSharedClassPointer);



//in  ModuleB/srcpy/bindings.cpp

class SharedClass
{
//....//
};

class ClassB{
{

ClassB(){//...//};
setSharedClassPointer(SharedClass* p){SharedClassPointer = p;}


SharedClass *SharedClassPointer;

};

pybind11::class_<ClassB>(m, "ClassB")
    .def(pybind11::init<>())
    .def("setSharedClassPointer",&ClassB::setSharedClassPointer);


//Python File

A = ClassA()
B = ClassB()
B.setSharedClassPointer(A.getSharedClassPointer())

Python returns a type error because it considers SharedClass defined in ModuleA to be a different type from SharedClass defined in ModuleB (although both have the same definitions repeated in each module) :

ModuleA.SharedClass is incompatible with function taking as argument ModuleB.SharedClass

2/ Use a PyCapsule :

//in  ModuleA/srcpy/bindings.cpp
...
pybind11::capsule getSharedClassPointer(ClassA& vt)
{
    return pybind11::capsule(vt.PointerToSharedClass, "SharedClass",nullptr);
}
pybind11::class_<ClassA>(m, "ClassA")
    .def(pybind11::init<>())
    .def("getSharedClassPointer",&getSharedClassPointer);

//in  ModuleB/srcpy/bindings.cpp
 ... 

void setSharedClassPointer(Class B&vt,pybind11::capsule capsule)
{
        SharedClass* data = static_cast<SharedClass*>(PyCapsule_GetPointer(capsule, "SharedClass"));
        vt.PointerToSharedClass=data;
}

pybind11::class_<ClassB>(m, "ClassB")
    .def(pybind11::init<>())
    .def("setSharedClassPointer",&setSharedClassPointer);

//in Python File

B.setSharedClassPointer(A.getSharedClassPointer())

But the call to A.getSharedClassPointer() returns a TypeError too.

TypeError: Unable to convert function return value to a Python type!

3/ Use Shared Data

//in  ModuleA/srcpy/bindings.cpp
void getSharedClassPointer(ClassA& vt)
{
    pybind11::set_shared_data("SharedClass",vt.PointerToSharedClass);
    printf("%p\n",vt.PointerToSharedClass);
}


//in  ModuleB/srcpy/bindings.cpp
void setSharedClassPointer(ClassB& vt)
{
    vt.PointerToSharedClass = pybind11::get_shared_data("SharedClass");
    printf("%p\n",vt.PointerToSharedClass);
}


// In Python File : 

A = ClassA()
B = ClassB()

A.getSharedClassPointer()
B.setSharedCLassPointer()

But this does not work either. The printf in getSharedClassPointer prints a non zero value but the one of setSharedClassPointer prints (nil).

I think I don't really understand what I am doing but hope there is a very simple way to transmit an OPAQUE pointer from one C++ module to another using pybind11.

Answer 1

This can happen because the ABI that each module is compiled with is is different. In my case, one module was a PyTorch extension, and the other the platform default ABI.

A work-around (not great IMO) is to sneak it into the module where the type is defined though a void pointer. In my case, I have a known ABI-safe type that is laid-out the same for both modules.

import mod2
import home_module

arg = mod2.SomeArg()
abi = mod2.get_abi_type(arg)
home_module.takes_abi_type(abi) # worked for me!

#include "abi_type.hpp"

PYBIND11_MODULE(home_module, m) {
    pybind11::class_<AbiType>(m, "AbiType");
    m.def("_abi_type_from_void_ptr", [](void const* vp) {
        return *static_cast<AbiType const*>(vp); // (returns a copy)
    };
    m.def("takes_abi_type", [](AbiType const&) { /**/ });
}

#include "abi_type.hpp"
struct SomeArg { AbiType impl; }

PYBIND11_MODULE(mod2, m) {
    pybind11::class_<SomeArg>(m, "SomeArg").def(pybind11::init<>());

    auto home_module = pybind11::module_::import("home_module");
    auto abi_type_from_void_ptr = home_module.attr("_abi_type_from_void_ptr");
    m.def("get_abi_type", [abi_type_from_void_ptr](SomeArg const& arg) {
        return abi_type_from_void_ptr(static_cast<void const*>(arg.impl));
    });
}