Sharing "cdef extern" functions

[Marcel Martin]

Hi,

I’d like to get some advice about how to share "pure" C functions
(defined in a .c file) between Cython modules. The way I tried to do it,
I get an "ImportError: ... undefined symbol:" when trying to import one
module from the other.

A minimal working example is at
https://github.com/marcelm/cython-shared-cfunc-mwe . I show only the
relevant parts below.

The actual project wraps C++, but I hope that this does not matter.

The first extension module "foo.one" wraps a C function named "xyz"
defined in "cfunc.c" (with a declaration in cfunc.h). Both the .pyx and
the .c source are linked into the .so. In setup.py, it looks like this:

setup(ext_modules=[
Extension("foo.one", sources=["foo/one.pyx", "foo/cfunc.c"]),
Extension("foo.two", sources=["foo/two.pyx"]),
])

one.pyx is empty. A one.pxd file contains just this declaration:

cdef extern from "cfunc.h":
int xyz()

In two.pyx, I want to use the xyz() function. I tried to do that by
cimporting:

cimport foo.one
foo.one.xyz()

However, when I run python -c "import foo.two", I get

ImportError: .../foo/two.cpython-310-x86_64-linux-gnu.so: undefined
symbol: xyz

I understand that this would work if the xyz() function were instead
written in Cython (with a definition in one.pyx and declaration in
one.pxd). In that case, Cython generates code to dynamically load the
function via __Pyx_ImportFunction from the imported module. Could I
somehow make this work with a pure C function?

I thought creating a shared library for all the C++ classes would be the
way to go, but I haven’t found instructions for how to get this to work
with setuptools and also how to properly load the shared library at
runtime. And then I already *have* a shared library (the .so for
foo.one) and there already is working machinery to load it at runtime,
so it feels unnecessary to duplicate it.

At the moment, our workaround is to just not have a second extension
module, that is, to put nearly everything into a single module, but this
is starting to get too unwieldy.

Regards,
Marcel

[Marcel Martin]

Hello,

On 22/08/2022 13.07, Marcel Martin wrote:
> I’d like to get some advice about how to share "pure" C functions
> (defined in a .c file) between Cython modules. The way I tried to do it,
> I get an "ImportError: ... undefined symbol:" when trying to import one
> module from the other.

After learning a bit more about how dynamic loading of shared libraries
works, I added a call to sys.setdlopenflags() to our __init__.py, which
appears to be the solution:

def _load_cpp_classes():
import sys
import os

flags = sys.getdlopenflags()
sys.setdlopenflags(os.RTLD_GLOBAL | os.RTLD_NOW)
import foo.one # noqa
sys.setdlopenflags(flags)

_load_cpp_classes()


dlopen(3) describes RTLD_GLOBAL:
> The symbols defined by this library will be made available for symbol
resolution of subsequently loaded libraries.

That is, when I subsequently do an `import foo.two`, the symbols are
available. The default when loading Python extension modules appears to
be RTLD_LOCAL.

Thanks for listening – this appears to have been another case of
"writing down the question helps answering it" ...

Regards,
Marcel