Providing Runtime Implementation for Externally Defined C Function

[Daniel Rozeboom]

Hi all,

   Please forgive my noobishness around this topic, my C / library linking knowledge is very spotty. :-P

   I'm using Cython to build a Python program ("client") that invokes functions of a C library ("utility", declared in various .h files and dynamically linked). However, the utility library is not available in my development environment and so I'd like to be able to (with a flag) optionally build my client to use local "stub" implementations of the utility instead.

   I have been able to build and run the client where the external utility library is available, but I haven't been able to get a local Cython stub implementation working. I keep getting an error of "Symbol not found" ... "Expected in: flat namespace." I've tried to simplify the problem down to the minimal case, so here are some sample files showing what I'm trying to do:

utility.h: (C function declaration)

extern int do_something(unsigned long* value);

utility.pxd: (Cython declaration of function in utility.h)

cdef extern from "utility.h":

    int do_something(unsigned long *value)

utility.pyx: (Stub implementation, only want to have this included in development build)

cdef int do_something(unsigned long *value):

    print('In Stub Implementation')

client.pyx: (Calls externally declared C function, which should be either the dynamically loaded "real" library or the Cython stub implementation)

from utility cimport do_something

def execute():

    print("Calling utility...")

    cdef unsigned long value = 0

    do_something(&value)

setup.py: (Cythonizes Extensions for use with build_ext, either including or excluding the stub .pyx file based on an environment variable)

from os import environ

from distutils.core import setup

from distutils.extension import Extension

from Cython.Build import cythonize

module_list = [Extension('client', ['client.pyx'])]

if environ.get('STUB'):

    print("Adding stub utility implementation to module list")

    module_list.append(Extension('utility', ['utility.pyx']))

else:

    print("Not including stub utility implementation")

setup(ext_modules=cythonize(module_list))

run.py: (Run the thing)

from client import execute

execute()

   So I want the client to be able to include "utility.pxd" for a single function declaration, which is implemented either by my stub (utility.pyx) or the dynamically linked library. The following are the steps I'm taking to build and run this with the 'STUB' flag enabled, where I get an error rather than the stub implementation running:

$ STUB=1 python3 setup.py build_ext --inplace
$ python3 run.py

Traceback (most recent call last):

  File "run.py", line 1, in <module>

    from client import execute

ImportError: dlopen(<path>/client.so, 2): Symbol not found: _do_something

  Referenced from: <path>/client.so

  Expected in: flat namespace

 in <path>/client.so

   I'm using Python 3.4.4, Cython 0.24 and setuptools 23.0.0. Based on other similar questions I've looked at, I'm guessing something is wrong with the way I'm declaring my Extension objects and/or a library or path environment variable problem for executing run.py. Please let me know if this question makes sense and if you can offer any tips as to how to get this working, or let me know if I'm not structuring this in a way that's appropriate for Cython.

   Thanks in advance for any help someone can offer on this!

- Daniel

[Daniel Rozeboom]

Hi, just wanted to bump this up - would really appreciate if anyone can offer some guidance on how I might be able to accomplish this, or maybe a link to an example project where the same thing is being done. Thanks so much!

- Daniel

[Stefan Behnel]

Daniel Rozeboom schrieb am 15.06.2016 um 21:54:
> I'm using Cython to build a Python program ("client") that invokes
> functions of a C library ("utility", declared in various .h files and
> dynamically linked). However, the utility library is not available in my
> development environment and so I'd like to be able to (with a flag)
> optionally build my client to use local "stub" implementations of the
> utility instead.
>
> I have been able to build and run the client where the external utility
> library is available, but I haven't been able to get a local Cython stub
> implementation working. I keep getting an error of "Symbol not found" ...
> "Expected in: flat namespace."

That suggests that you're on MacOS-X ?

You're not linking against any library here, so the external library symbol
that you are calling is not defined at runtime.

> if environ.get('STUB'):
> print("Adding stub utility implementation to module list")
> module_list.append(Extension('utility', ['utility.pyx']))
> else:
> print("Not including stub utility implementation")

This doesn't work on all operating systems. Specifically not on MacOS,
IIRC, which doesn't allow referencing symbols between different dynamically
loaded libraries (I really hope I remember that right).

What I would do instead is to write the stub function(s) in C (assuming
it's really something simple) and add the .c file to the list of sources of
the Extension object above, i.e.

module_list = [
Extension('client', sources=['client.pyx', 'utility.c'])
]

That way, it doesn't depend on dynamic linking behaviour at all, and you
only get a single Python module.

Stefan

[Daniel Rozeboom]

Hi Stefan,

   Thanks very much for your assistance on this issue. Sorry for not mentioning, but yes, I am on OSX.

   I was able to try the method you described successfully; however, the stub functionality is relatively complicated and so I was still hoping to write it in Cython if possible. I'm continuing to experiment with ways to incorporate the strategy you suggested, where I wonder if there may be at least two possible solutions:

1) First, Cythonize the stub Cython code into a C file (generating the equivalent of 'utility.c' in your example) that subsequently gets statically linked into the client module (e.g., two stages of Cythonization... not sure if Cython can be [ab]used for this purpose)

2) Pairing a small C "stub bridge" file like the one you mentioned with a Cython implementation, where the "stub bridge" simply redirects the method call to a Cython method exposed in the same way one would make any Cython code accessible to C code

   If you have any other suggestions for an approach that allows the stub to be written in Cython that would be much appreciated, but in the meantime I will see if I can get one of the above strategies working.

Many thanks,

Daniel

[Robert Bradshaw]

(2) would probably be easiest. Implement it as a "public" cdef
function in a pyx file (so its name doesn't get mangled), then call it
from your utility.c file.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Daniel Rozeboom]

Thanks for the suggestions. One other constraint I didn't originally mention (in an effort to simplify the repro case) is that I would like for the 'utility' module to be usable by any number of different 'client' modules, so I want to avoid Cythonizing the client and the utility into one Extension together (because I think ultimately, this would necessitate creating one huge Extension for the whole application).

Since I couldn't figure out a classier way (and also because I got tripped up around http://stackoverflow.com/questions/15451740/cython-c-header-file-paths-in-shared-pxd-file), what I ended up doing is a very hack-y set of pre/post Cythonization automated source code manipulations. For each utility module 'm' in path 'p', I do the following:

A) Before Cythonize:

* If p/m_stub.pyx exists, ensure that p/m.pyx and p/m.pxd also exist (m.pyx is typically empty), and also ensure that p/m.pxd contains at least one "cdef extern from" statement.

* Copy p/m.pyx and p/m.pxd to backup locations (in case something unexpected happens, and also to use for restoring the original state after Cythonizing).

* Move p/m_stub.pyx to p/m.pyx, overwriting anything that might have been there before.

* For every line of the form "cdef extern blah blah:" in p/m.pxd, replace it with something like "cdef: # extern blah blah", i.e. comment out the extern part. Doing it this way, the file could potentially be reverted in-place (with a sufficiently obscure token in the comment separation) and then verified against the backup afterwards. Alternatively, you could just put "cdef:" if the backup/original will always be used to clobber the modified version afterwards. I'm also currently assuming that the declarations always happen with "cdef ...:" and then the function signatures on additional lines following that, but a more robust solution might be required in cases where extern functions can be declared on one line.

B) Cythonize with all of the .pyx files in the various utility and client directories together (no explicit references to Extension required, for my case at least). I'm also using exclude=**/*_stub.pyx, but this may not be necessary depending on how good the file shifting logic is.

C) After Cythonize:

* Move p/m.pyx back to p/m_stub.pyx, and then move the backups of the non-stub .pxd and .pyx files back to their original locations (m.pxd and m.pyx).

Then, setup() can be called with ext_modules=<the list of Extensions generated in step B>. This seems to successfully create a module for each utility and each client, as desired.

While definitely brittle, one advantage of this approach vs. 2) is that it doesn't require a .c bridge file, so it's easy to just write a stub in Cython. Also, the compile will fail if the stub definition file doesn't satisfy the signatures specified in the declaration file, which is good. Compiling and running in this stub mode won't highlight if the declaration file doesn't match the specified header file, but one can still perform a compile in 'non-stub' mode (i.e. not doing any of the above file manipulation) and any problems with the declarations should then appear. This method just requires that developers on the project learn the pattern of required files for including 'stub' implementations, and of course the legwork to actually implement the file tomfoolery.

For the Stack Overflow issue referenced above, in the alternate case of Cythonizing/compiling in non-stub mode, I use a similar back-up-and-modify strategy to replace a placeholder in the header paths (in .pxd files only) with an actual, resolved directory path on the machine doing the compiling. In effect, this lets the developer use a relative path when specifying header files in a way that won't barf with multiple development environments, since by the time it gets Cythonized and compiled it's actually using an absolute path. I didn't see any responses to the OP on that Stack Overflow issue, but it would be awesome if a future version could include some kind of functionality for either variable substitution or relative paths in "extern from" declarations... the stub vs. non-stub issue is probably less broadly applicable and/or easy to incorporate.

Thanks again,

Daniel