cfunc: how to call from C code

[Martin Fiers]

Hello,

I'm very happy with the new cfunc functionality! It may eventually replace a large chunk of custom-written code in one of our simulators to make it much more robust.

I have a question however: the documentation of cfunc talks about exposing the 'address' attribute. I didn't find so far how I could reuse that address inside C code to do an actuall call. I tried to create a small prototype to test this, but it gives a segmentation fault. Below are the two files.

run_cfunc.c: 

#include "Python.h"

#include <stdio.h>

double call_func(PyObject *self, PyObject *args) {

    printf("call_func\n'");

    const int *addr;

    double res;

    printf("call_func: converting to address\n");

    if (!PyArg_ParseTuple(args, "i", &addr)) {

        printf("Could not parse integer value in call_func.\n");

        return -1;

    }

    printf("call_func: converting address to func pointer\n'");

    //double (*func)(double, double) = addr;

    double (__cdecl *func)(double, double) = addr;

    printf("call_func: calling function\n");

    res = (double)func(1.0, 2.0);

    return res;

}

static PyMethodDef RunCFuncMethods[] = {

    {"call", call_func, METH_VARARGS,

     "Execute function."},

    {NULL, NULL, 0, NULL}        /* Sentinel */

};

PyMODINIT_FUNC

init_run_cfunc(void)

{

    printf("Initializing module\n");

    PyObject *m;

    m = Py_InitModule("_run_cfunc", RunCFuncMethods);

    if(m == NULL) {

        printf("Could not initialize module!\n");

    }

}

And then a Python file:

import numba

cfunc = numba.cfunc

def add(p1, p2):

    return p1 + p2

add_nb = cfunc("float64(float64, float64)", nopython=True, cache=True)(add)

add_nb.ctypes(2, 3)

print(add_nb.address)

import _run_cfunc

_run_cfunc.call(add_nb.address)

This code will crash when trying to call the function. ("call_func: calling function").

Compilation on Windows was like this:

gcc -IC:\Software\miniconda2\envs\numbatest\include run_cfunc.c -c -DMS_WIN64 -o run_cfunc.o && gcc -shared -l python27 -LC:\Software\miniconda2\envs\numbatest run_cfunc.o -o _run_cfunc.pyd

I prefer not to use ctypes to load the C code; as this code is eventually part of a much larger C++ extension. I understand if I would use ctypes and have a function signature the ctypes, then this would work. 

So it must be some fairly trivial conversion from the address to the function pointer (double (__cdecl *func)(double, double) = addr;). This is not explained in the doc, so I have a hard time figuring out how to properly do the conversion :).

Thanks for looking into this!

Martin

[stuart]

Hi,

Glad you like the functionality, thanks for the feedback.

Indeed what you describe is possible, just a few minor adjustments and your code should work. Main points:

1. The PyMethodDef struct isn't quite valid. https://docs.python.org/2/c-api/structures.html#c.PyMethodDef. The function `call_func` must return a `PyObject *` for it to be valid in the PyMethodDef struct.
2. A C `int` isn't wide enough to hold an address. Using a ` Py_ssize_t` for `addr` or perhaps a unsigned long (long) along with the appropriate parse format should work (see https://docs.python.org/2/c-api/arg.html).
3. The `call_func` would benefit from adhering to the `PyCFunction` behaviour of returning NULL if an exception is set (https://docs.python.org/2/c-api/structures.html#c.PyCFunction) and returning a new ref on successful completion. Your function could either end with `Py_RETURN_NONE` if you don't want the function to return anything to Python, or something like returning a `PyFloat_FromDouble` result instantiated from `res` could work.

Here's something that works for me (on linux :-) )

#include "Python.h"
#include <stdio.h>

typedef double (*func_t)(double, double);

static PyObject * call_func(PyObject *self, PyObject *args)
{
    printf("call_func\n'");
    const Py_ssize_t *addr;
    double res;
    func_t func;
    PyObject *ret;

    printf("call_func: converting to address\n");

    if (!PyArg_ParseTuple(args, "n", &addr))

    {
        printf("Could not parse integer value in call_func.\n");

        return NULL;

    }

    printf("call_func: converting address to func pointer\n'");

    func = (func_t)addr;

    printf("call_func: calling function\n");

    res = func(1.0, 2.0);
    printf("call_func: %f\n", res);

    /*  return the result */
    ret = PyFloat_FromDouble(res);
    if(PyErr_Occurred()) return NULL;
    return ret;

}

static PyMethodDef RunCFuncMethods[] =
{
    {
        "call", call_func, METH_VARARGS,
        "Execute function."
    },
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyMODINIT_FUNC
init_run_cfunc(void)
{
    printf("Initializing module\n");
    PyObject *m;
    m = Py_InitModule("_run_cfunc", RunCFuncMethods);
    if(m == NULL)
    {
        printf("Could not initialize module!\n");
    }
}

Hope this helps,

--
stuart

[Martin Fiers]

Dear Stuart,

That works! That's very much appreciated! And *oops* for the beginner mistakes :). One step close to adopting numba!

Kind regards,

Martin

[stuart]

Hi,

Great to hear that it's working. Thanks for your support.

Kind Regards,

--
stuart

[Martin Fiers]

Hello,

I'm picking up this thread again because we would like to take it 1 step further. Suppose, now, we have different building blocks, each building block is represented by a jitclass (there can be multiple types of classes for this). Each node/building block has a calculation function. We would like to call this calculation function from C. We've tried many things, but somethimes it would crash or just give wrong behavior. There's two questions now:

1. How should we - if possible - call a jitclass-backed class method (from C)? Basically, is there an equivalent of the cfunc '.address' for jitclass methods?

2. Is there a way to pass on C structs to a cfunc function? i.e., we do the same thing as in the original example, except now instead of two doubles as arguments, we now have a jitclass as argument (which, reading form the numba doc, should be a C-compatible struct). Whatever we tried crashed Python somehow, or gave nonsense results.

From a broader perspective, we want to loop over a large number of 'nodes/building blocks (which we thought would be easiest to represent using a jitclass)', and want to run a certain calculation method each time.

If the questions are too vague, please say so, and I'll try to compile a relevant example. Looking forward to your reply!

With kind regards,

Martin

On Sunday, 20 November 2016 11:46:25 UTC+1, Martin Fiers wrote: