Passing Python callbacks to an external C library.

[Björn Dahlgren]

Hi all!

I am writing a partial wrapper around sundials (and ODE solver written in C) for a project of mine and
I wanted to be able to pass python callbacks to the C library using Cython (even though
I mainly pass C pointers to native code).

It was not immediately apparent how I should go about achieving this but I ended up using
a global variable in the Cython module to point to the user provided Python callback.

I implemented a minimal working example around a fictitious "library" which has two functions
for computing the derivative and gradient of functions using a simple forward difference formula.

If my implementation is sound (criticism is very welcome) maybe it could serve as a self contained example
(I could rewrite it for inclusion in the documentation if there is any interest)?

The code is available at:
https://github.com/bjodah/expose_pycallback_to_a_clib

[Nils Bruin]

On Tuesday, January 13, 2015 at 7:49:59 AM UTC-8, Björn Dahlgren wrote:

Hi all!

I am writing a partial wrapper around sundials (and ODE solver written in C) for a project of mine and
I wanted to be able to pass python callbacks to the C library using Cython (even though
I mainly pass C pointers to native code).

It was not immediately apparent how I should go about achieving this but I ended up using
a global variable in the Cython module to point to the user provided Python callback.

A global pointer will obviously cause problems as soon as you need to register multiple/variable callback functions. What you really want is a "closure" in C, which doesn't exist. Newer versions of C++ have them. It might be interesting to see if such a closure can be used as a function pointer where a plain C function is expected.

There is a usual work-around in C for cases where closures would likely be useful, though: In many C interfaces where one specifies a callback, one not only gives a function pointer but also a (void) pointer to "user data". This pointer gets passed back to the callback function as a parameter when the callback happens. This is what you can use to pass variable data back to your own function.

In your case, the pointer to the python callback could be stored in the "user data" and your C-level specified callback function can lookup this pointer and call the python function as desired.

cdef ... callback( ...parameters ..., void * userdata):
    F=<PyObject *>userdata
    python_result = F( ... processed parameters ...)
    return <processed python result to match callback signature>

register_callback( &callback, pointer_to_python_object_specified_as_userdata)

It looks like the SUNDIALS API indeed has room for userdata pointers in its callback signatures.

[Björn Dahlgren]

On Wednesday, 14 January 2015 06:24:19 UTC+1, Nils Bruin wrote:

A global pointer will obviously cause problems as soon as you need to register multiple/variable callback functions. What you really want is a "closure" in C, which doesn't exist. Newer versions of C++ have them. It might be

interesting to see if such a closure can be used as a function pointer where a plain C function is expected.

 
I liked this idea so I ran with it until I realized C++ that capture cannot be cast to function pointers:
https://github.com/bjodah/expose_pycallback_to_a_clib/pull/1

Even though global pointers aren't pretty there is only one per "callback wrapper" so I am not sure what you mean that they will cause
problems for mulitple/variable callback functions?

There is a usual work-around in C for cases where closures would likely be useful, though: In many C interfaces where one specifies a callback, one not only gives a function pointer but also a (void) pointer to "user data". This pointer gets passed back to the callback function as a parameter when the callback happens. This is what you can use to pass variable data back to your own function.

In your case, the pointer to the python callback could be stored in the "user data" and your C-level specified callback function can lookup this pointer and call the python function as desired.

cdef ... callback( ...parameters ..., void * userdata):
    F=<PyObject *>userdata
    python_result = F( ... processed parameters ...)
    return <processed python result to match callback signature>

register_callback( &callback, pointer_to_python_object_specified_as_userdata)

It looks like the SUNDIALS API indeed has room for userdata pointers in its callback signatures.

Yes that would definitely work, but I am actually writing a C++ wrapper foremost so I don't want to reference
any Python related objects in the C++ wrapper. Using Cython to expose a C function pointer interface to a Python
callback lets me use the wrapper unmodified.

[Nils Bruin]

On Wednesday, January 14, 2015 at 1:32:51 AM UTC-8, Björn Dahlgren wrote:

Even though global pointers aren't pretty there is only one per "callback wrapper" so I am not sure what you mean that they will cause
problems for mulitple/variable callback functions?

If you have multiple instances of whatever object these callbacks get registered with, then using the same "callback wrapper"  for each instance would be the natural thing to do, but with a global pointer you cannot do that. Perhaps that doesn't happen for and/or perhaps SUNDIALS has been designed in a way that prevents that from ever happening (fortran based code perhaps?). However, a general example for how to wrap callback registration should definitely keep this in mind.

I am actually writing a C++ wrapper foremost so I don't want to reference
any Python related objects in the C++ wrapper. Using Cython to expose a C function pointer interface to a Python
callback lets me use the wrapper unmodified.

In that case you should probably define the C++ interface in terms of std::function, which allows arbitrary callables to be passed. Your callback wrapper will then just expect a pointer to the std::function in its user data. Your python interface to your C++ interface would then need to figure out how to put a python callable in a std::function, but that should be fairly straightforward to figure out. You'd get the corresponding inefficiencies of a double-stacked interface between languages with subtly different calling conventions, though.

The "void *user_data" really is there to help you avoid global variables. All well-designed C callback interfaces have them (as a functional equivalent to closures). Your scenario is exactly what they are for.

[Björn Dahlgren]

On Wednesday, 14 January 2015 17:37:49 UTC+1, Nils Bruin wrote:

If you have multiple instances of whatever object these callbacks get registered with, then using the same "callback wrapper"  for each instance would be the natural thing to do, but with a global pointer you cannot do that. Perhaps that doesn't happen for and/or perhaps SUNDIALS has been designed in a way that prevents that from ever happening (fortran based code perhaps?). However, a general example for how to wrap callback registration should definitely keep this in mind.

Ah yes, in the general case globals will be too limited..
 

The "void *user_data" really is there to help you avoid global variables. All well-designed C callback interfaces have them (as a functional equivalent to closures). Your scenario is exactly what they are for.

Yes, thinking more about this I think I should use it when it is available, but I am struggling:

https://gist.github.com/bjodah/1ad81ca218d21e941947

above is an attempt on a MWE of letting a Python callback modify a Numpy wrapped C-array.
But it is segfaulting in my invocation of:

PyObject * x_ = PyArray_SimpleNewFromData(1, dims, NPY_DOUBLE, (void *)x);

where PyArray_SimpleNewFromData is a nested macro definition eventually calling by indiraction in a API[] of function pointers...
Can any of you Cython/NumPy spot the mistake?

Thank you for your feedback Nils!

Best,
/Björn

[Björn Dahlgren]

On Friday, 16 January 2015 15:15:57 UTC+1, Björn Dahlgren wrote:

But it is segfaulting in my invocation of:

PyObject * x_ = PyArray_SimpleNewFromData(1, dims, NPY_DOUBLE, (void *)x);

For future reference. I was missing a "import_array();" invocation to enable PyArray_* functions.
Gist updated to working condition if anyone struggles with this in the future.