Multiple DLLs - only first one is used - "no code attached yet with @ffi.def_extern()"

[M P]

I figured out the source of my other post. I have 4 build scripts for 4 different function implementations. They each take this format:

# file Build%STRATEGY_NAME%.py

# DEVELOPER'S NOTE: All files are found relative to the current working directory
# of the terminal or script that calls the `python` command.

import cffi

ffibuilder = cffi.FFI()

with open('AlgoServer.h') as f:
    # read plugin.h and pass it to embedding_api(), manually
    # removing the '#' directives and the CFFI_DLLEXPORT
    data = ''.join([line for line in f if not line.startswith('#')])
    data = data.replace('CFFI_DLLEXPORT', '')
    ffibuilder.embedding_api(data)

    with open('Bars.h') as f:
        # read plugin.h and pass it to embedding_api(), manually
        # removing the '#' directives and the CFFI_DLLEXPORT
        data = ''.join([line for line in f if not line.startswith('#')])
        data = data.replace('CFFI_DLLEXPORT', '')
        ffibuilder.embedding_api(data)

ffibuilder.set_source("Build%STRATEGY_NAME%", r'''
    #include "Bars.h"
    #include "AlgoServer.h"
''')

ffibuilder.embedding_init_code("""
from Build%STRATEGY_NAME% import ffi

# Add futbox import directory
import sys
import os
#sys.path.append(os.path.expandvars("$FUTBOX_PY_MODULE_DIR"))

%STRATEGY_IMPORTS%

# Python implementation of exposed C code

%STRATEGY_PYTHON_CODE%
""")

ffibuilder.compile(target="libFtbx%STRATEGY_NAME%.*", verbose=True)

Where each

%STUB%

is replaced by the python script's content. Below is example output of these scripts:

# file BuildTrain.py

# DEVELOPER'S NOTE: All files are found relative to the current working directory
# of the terminal or script that calls the `python` command.

import cffi

ffibuilder = cffi.FFI()

with open('AlgoServer.h') as f:
    # read plugin.h and pass it to embedding_api(), manually
    # removing the '#' directives and the CFFI_DLLEXPORT
    data = ''.join([line for line in f if not line.startswith('#')])
    data = data.replace('CFFI_DLLEXPORT', '')
    ffibuilder.embedding_api(data)

    with open('Bars.h') as f:
        # read plugin.h and pass it to embedding_api(), manually
        # removing the '#' directives and the CFFI_DLLEXPORT
        data = ''.join([line for line in f if not line.startswith('#')])
        data = data.replace('CFFI_DLLEXPORT', '')
        ffibuilder.embedding_api(data)

ffibuilder.set_source("BuildTrain", r'''
    #include "Bars.h"
    #include "AlgoServer.h"
''')

ffibuilder.embedding_init_code("""
from BuildTrain import ffi

# Add futbox import directory
import sys
import os
#sys.path.append(os.path.expandvars("$FUTBOX_PY_MODULE_DIR"))


# Python implementation of exposed C code

# from futbox.PyFriendly import make_python_friendly
# from futbox import CustomMetrics

@ffi.def_extern(name="whatever")
def whatever():
    print("in train function")
""")

ffibuilder.compile(target="libFtbxTrain.*", verbose=True)

This works perfect for the first DLL linked. There are no build errors. Except that the following DLLs are never properly referenced at run-time.

Example, if I have function "whatever()" defined in the first DLL and a function "doSomethingElse()" in the second, only "whatever()" will be properly defined. "doSomethingElse()" will print "no code attached yet..."

I'm linking them properly in my build script:

   gcc -std=gnu99 -I"$(pwd)" -I/usr/local/Cellar/postgresql -I/usr/local/Cellar/czmq/4.0.2/include -I"$(pwd)"/.. tests/TestAlgoServer.c Bars.c CQueue.c -L"$(pwd)"/tests -L"$(pwd)" -L/usr/local/Cellar/postgresql -L/usr/local/Cellar/czmq/4.0.2/lib  -lFtbxBasic -lFtbxMovingAverageCross -lFtbxVwap -lFtbxTrain -lpython2.7 -lczmq -lzmq -lpq -lssl -lcrypto -o Server

Each DLL has a different name. So what gives? What can be done?

[Armin Rigo]

Hi,

I tried to reproduce with no success. See attached files. I would
suggest that you start with the attached files and try to make it look
more like your own case until you hit the same problem again, while
keeping it independently compilable. Then post the resulting code.
(This is what I mean with "can I reproduce the problem?")


A bientôt,

Armin.

[M P]

I have successfully recreated the issue using your setup. See the attached file.

Run

sh build.sh -clean

to clean. Otherwise you can run the script

sh build.sh

To compile for your OS

------

It seems the issue comes from when sharing the same embedding api. What can we do about this? The header file is long and contain structs that must be referenced for the python scripts.

[Armin Rigo]

Hi,

On 21 March 2017 at 06:36, M P <bmpe...@gmail.com> wrote:
> I have successfully recreated the issue using your setup. See the attached
> file.

Ah! Thanks, that clears it up. The problem is that you're calling
ffibuilder.embedding_api() listing both the fx and the fy function, in
each of x.py and y.py. As a result, both _x and _y independently
export functions called fx and fy. We might expect that you'd get an
error or warning when linking, saying something like "the symbol fx is
defined in both libx.so and liby.so", but no. What occurs instead is
that it silently picks the version of fx and fy that are defined in
libx.so. So when xylink.c calls fy(), it actually calls the version
of fy that comes with libx.so, and indeed because we don't see a
@ffi.def_extern() for fy inside libx.so, then it complains.

The proper fix is to only list the functions that you actually
implement here, in each call to ffibuilder.embedding_api().

I'm not sure how to fix that to give a better error. I'll need to
think about it. Maybe simply:

extern "Python": function _x.fy() called, but no code was attached to it
yet with @ffi.def_extern(). Returning 0.

Note the '_x'.


A bientôt,

Armin.

[M P]

You were absolutely correct. I solved the issue by writing a multi step script.

First I generate the unique header file for that python script.
Second, I build the python script with the unique header; only defining the functions found in the python script.
Third, I generate the final header file that contains ALL the cdef functions.
Fourth, I compile the code with the final header file and link all the DLLs.

The program runs.