How to re-use a CFFI module in another CFFI module?

[Matthias Geier]

Dear list.

I would like to create a re-usable CFFI module, which slightly
simplified could look something like this:

one_build.py:
#########################################
from cffi import FFI
ffibuilder = FFI()
ffibuilder.cdef('int one_function(void);')
ffibuilder.set_source(
'_one',
'int one_function(void) { return 42; }'
)
if __name__ == '__main__':
ffibuilder.compile(verbose=True)
#########################################

I now want to create a second module that re-uses the "one" module.
I tried this:

two_build.py:
#########################################
from cffi import FFI
ffibuilder = FFI()
ffibuilder.cdef('int two_function(void);')
ffibuilder.set_source(
'_two',
'int one_function(void);'
'int two_function(void) { return one_function(); }'
)
if __name__ == '__main__':
ffibuilder.compile(verbose=True)
#########################################

Trying to import the second module throws an error:

>>> import _two
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: [...]/_two.cpython-35m-x86_64-linux-gnu.so: undefined
symbol: one_function

I tried the "sources", "libraries" and "extra_objects" arguments of
ffibuilder.set_source(), but I couldn't get it to work.

Are there any other options that I could try?

How can I make my module "_two" load the module "_one"?

And with "load" I probably mean dynamically (or even statically?) link to it ...

cheers,
Matthias

[Armin Rigo]

Hi Matthias,

On 25 February 2017 at 13:45, Matthias Geier <matthia...@gmail.com> wrote:
> I would like to create a re-usable CFFI module, which slightly
> simplified could look something like this:

It's not directly supported. If you write one_function() in the first
module, then you get a compiled library named something like "_one.so"
(python 2.7) or "_one.cpython-35.so" (python 3.5); similarly for the
second module. The problem is that you want to call code from
"_one.*so" from inside "_two.*so". In order to do so, you need to
know exactly how "_one.*so" is called and pass the correct option to
the linker: for example, ``libraries=['_one']`` or (untested)
``libraries=['_one.cpython-35']``. You probably have to convince gcc
to look in the current directory: ``library_dirs=['.']``. Finally,
you might have to add more horrible tweaks for "_two.*so" to locate
"_one.*so" at runtime, though that's not needed on all operating
systems if you import "_one" before "_two". (The tweaks are very
OS-dependent and noted in the page
http://cffi.readthedocs.io/en/latest/embedding.html#issues-about-using-the-so.)

A better approach is probably to move the logic you want to reuse
inside some general C library, which you compile the normal way for a
C library; then use that from both "_one" and "_two".

Of course, if the two parts of the code are closely tied together,
then it's much simpler to just generate a single CFFI module. You
shouldn't think "it's extra code that is not always necessary", as
that argument doesn't really apply on modern OSes which will load only
the parts of a library that are actually used anyway.

Also, note that cffi has support for ffi.include(), which builds a FFI
object that extends another. It mostly means sharing the base's types
and constants. That's essentially a way to make an interface over two
existing C libraries, one of which depends on the other. In C, the
two libraries would come with C headers "first.h" and "second.h", and
the file "second.h" typically starts with ``#include "first.h"`` and
then adds its own functions. But your question seems to be about
reusing code compiled *inside the CFFI first interface,* which is
different.


A bientôt,

Armin.

[Matthias Geier]

Hi Armin.

Thanks very much for the thorough (but still very quick!) answer!

Based on your suggestions I found a solution that works on Debian
Linux/GCC, I don't know if it will work on other OSs/Compilers ...

I thought that the _one.lib object must know where "it came from", and
indeed the absolute path of the shared library can be obtained with
_one.__file__.

I put this into "extra_objects" and the compiler finds it, that's great!
It also seems to work with "extra_link_args", which makes more sense?

In case you are interested, my module "one" is actually called
"pa_ringbuffer": https://github.com/mgeier/python-pa-ringbuffer

This was previously part of the "rtmixer" module:
https://github.com/mgeier/python-rtmixer.

I've just created a PR that removes the RingBuffer functionality from
rtmixer (making it the module "two") and instead uses the new
pa_ringbuffer module: https://github.com/mgeier/python-rtmixer/pull/3.

Does that make sense?

cheers,
Matthias

On Sat, Feb 25, 2017 at 3:02 PM, Armin Rigo wrote:
> Hi Matthias,
>

[Matthias Geier]

Dear Armin, dear list.

As I wrote a few days ago, I'm trying to link one CFFI module to
another by using the library file name in the "extra_objects" argument
of the set_source() function.

I've done this successfully on Linux (Debian) and I've heard it also
works on Windows (Visual C++).

Sadly, it doesn't work (yet) on macOS, where I'm getting this error message:

$ python3 setup.py develop
...
creating build/temp.macosx-10.11-x86_64-3.6/build/temp.macosx-10.11-x86_64-3.6
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common
-dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -UNDEBUG
-Isrc -I/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/include/python3.6m
-c build/temp.macosx-10.11-x86_64-3.6/_rtmixer.c -o
build/temp.macosx-10.11-x86_64-3.6/build/temp.macosx-10.11-x86_64-3.6/_rtmixer.o
creating build/lib.macosx-10.11-x86_64-3.6
clang -bundle -undefined dynamic_lookup
build/temp.macosx-10.11-x86_64-3.6/build/temp.macosx-10.11-x86_64-3.6/_rtmixer.o
/Users/me/python-pa-ringbuffer/src/_pa_ringbuffer.abi3.so -o
build/lib.macosx-10.11-x86_64-3.6/_rtmixer.abi3.so
ld: can't link with bundle (MH_BUNDLE) only dylibs (MH_DYLIB) file
'/Users/me/python-pa-ringbuffer/src/_pa_ringbuffer.abi3.so' for
architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1

I guess this has something to do with the -bundle and -dynamiclib
options, but I don't really know how to fix this ...

Any ideas how I can get this to work?

See also this issue on Github: https://github.com/mgeier/python-rtmixer/pull/3.

cheers,
Matthias