Call Numba generated shared library from Cython

[Chase Coleman]

One of my classes gave an assignment to write a function “as fast as possible” using tools in Python. I have written both numba and cython versions and parallelized it on different dimensions.

As I was playing with this, I came across a new(ish) feature of numba which allows you to compile your function into a shared library. I thought it might be fun to try and call a numba function from cython — I haven’t been able to get it working yet though. I was wondering if people might know what is going wrong. I don’t have lots of experience with Cython, so there are probably obvious things I’m doing wrong and I’m sorry if this is either trivial or just generally a bad idea. Below is an example (simpler than what I’m really trying to do):
This is the code to compile the function using numba.

import numpy as np
from numba.pycc import CC

cc = CC('testmodule')

@cc.export('sum_rnv', 'f8(f8, f8, i8)')
def sum_rnv(mean, std, N):
    # Create variable that will hold total
    total = 0.0

    for i in range(N):
        total += mean + std*np.random.randn()

    return total

cc.compile()

Compiling this function will create a file called testmodule.cpython-35m-x86_64-linux-gnu.so. The function can then be run by doing

import testmodule
testmodule.sum_rnv(0.0, 1.0, 100000)

Now I could write a short .pyx file

from cython.parallel import prange

# I think I need to do a cdef extern (Unsure about this part)
cdef extern double sum_rnv(double mean, double std, int n)

def sum_many_rnv(double mean, double std, int N, int n)

    # Create place holder for big total
    cdef double bigtotal = 0.0
    cdef int i

    for i in prange(N, nogil=True):
        bigtotal += sum_rnv(mean, std, n)

    return bigtotal

When I substitute sum_rnv(mean, std, n) with 0.0 and comment out the cdef extern then I can compile this file with the following setup.py

from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension("testcython",
              ["testcython.pyx"],
              libraries=[],
              extra_compile_args=["-O3", "-march=native"],
              extra_link_args=["-fopenmp"],
             )
]

setup(name="testcython",
      cmdclass={"build_ext": build_ext},
      ext_modules=ext_modules
     )

My question is how to get the cython to see the shared library build by numba. I have tried a couple of different things: I added testmodule to the list of libraries in setup.py and added LDFLAGS="-L/path/to/numba/so/" \ python setup.py build_ext -i as seen here. Also tried something like this SO answer to no avail. Any thoughts?

​

[Yury V. Zaytsev]

On Mon, 29 Feb 2016, Chase Coleman wrote:

> Compiling this function will create a file called testmodule.cpython-35m-x86_64-linux-gnu.so. The function can then be run by doing
>
> import testmodule
> testmodule.sum_rnv(0.0, 1.0, 100000)
>
> Now I could write a short .pyx file
>
> from cython.parallel import prange
>
> # I think I need to do a cdef extern (Unsure about this part)
> cdef extern double sum_rnv(double mean, double std, int n)

You're right to be unsure about this part :-)

Unless Numba additionally exports some sort of function with (double,
double, int) signature (which is maybe also used internally in the Python
wrapper), you can only use the Python interface from Cython to call it.

Of course, this will result into having to box & unbox the numbers on the
border, but if there isn't another exported function that you can call
directly, there is nothing you can do about it.

--
Sincerely yours,
Yury V. Zaytsev

[Chase Coleman]

Ha! I'm glad I was at least unsure about the right part :-)

There doesn't seem to be an obvious way to do this. Thanks for the response!

[Marco Tröster]

Hey guys,

just a quick question because your thread is about 6 years old now, so maybe some things have changed since then.

So, is it still the case that Cython issues Python C-API overhead when calling into precompiled Numba-optimized code? I'm not so sure by the documentation how Cython and Numba actually interact.

I'm sorry to ask in that way, but apparently noone on the internet seems to be using both Numba and Cython together, so you seem to be the only guys who know things :D

Thanks for your help in advance.

Best wishes, Marco

[da-woods]

You can definitely call Cython functions from Numba. See https://stackoverflow.com/questions/43016076/calling-cython-functions-from-numba-jitted-code and https://numba.readthedocs.io/en/stable/extending/high-level.html#importing-cython-functions for a detailed explanation.

Obviously you can call a Numba function as a normal Python callable from Cython, but that does give you Python C-API overhead.

It looks like you can get a C function pointer for a Numba function: https://numba.readthedocs.io/en/stable/user/cfunc.html. Within Cython you'd convert the address to a function pointer with a two-step cast:

func_ptr_variable = <func_ptr_type><uintptr_t>cfunc_pyobject.address

You look to be a bit restricted to basic C types as the function arguments.

Hopefully that's of some use

David

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/e39f71fd-76c7-4897-b242-ae8bd488d341n%40googlegroups.com.