create a C function in Python

[Eric Frederich]

Hello,

I have a big binding library that I have built up.
Most of the parameter types to the functions are primitives and Cython makes them very easy to wrap.

I have been skipping over some functions though that take a void function pointer because I have no clue how to implement this.

Is it possible to create a C function at runtime from Python?
I know I can create C functions using Cython by doing something like...
    cdef api foo(int a):

But that is at compile time, not at runtime.

Any ideas here how to solve this problem?

Thanks,
~Eric

[Stefan Behnel]

Eric Frederich, 13.11.2012 20:27:

Sounds like you are looking for this kind of code:

https://github.com/cython/cython/tree/master/Demos/callback

In short, cdef functions are plain C functions that you can pass into C
code as a function pointer.

Stefan

[Eric Frederich]

Thanks, though it seems that this example works only because the API takes two pointers and they're using the 2nd one (void* user_data) to pass the actual Python function.
How would I wrap C functions that just take a single C function pointer?

~Eric

[Robert Bradshaw]

On Tue, Nov 13, 2012 at 1:28 PM, Eric Frederich
<eric.fr...@gmail.com> wrote:
> Thanks, though it seems that this example works only because the API takes
> two pointers and they're using the 2nd one (void* user_data) to pass the
> actual Python function.
> How would I wrap C functions that just take a single C function pointer?

That is a deficiency in the API. Essentially the only thing to do here
is either store the user data in some global (if you control the scope
of when the callback will be called) or write a whole set of cdef
functions, each of which calls a different Python level function under
your control.

You might be able to use an ffi to generate a C-level closure.

- Robert

[Eric Frederich]

You say it is a deficiency in the API if there is no void* for user data in addition to the void* to the C function?
Is it a standard practice to do so?
I don't come across these types of situations often so I don't know what standard practice is for APIs that take user defined functions.

In any case the 3rd party library I am creating Python bindings for doesn't provide an extra void*.
I had thought about using a global variable to store the Python callable.

[Dag Sverre Seljebotn]

On 11/13/2012 10:56 PM, Eric Frederich wrote:
> You say it is a deficiency in the API if there is no void* for user data
> in addition to the void* to the C function?
> Is it a standard practice to do so?

Yes, I think any experienced C programmer would say that a function
taking a callback without a void* for user data is essentially a bug, or
at least a major design flaw.

Dag Sverre

[Eric Frederich]

Interesting.
Other than wrapping the functions for use from Python what would be another use case?


Now I'm just thinking out loud here....

If the void* for user data has other uses in C perhaps it does in Python as well.
Creating the bindings as the cython/Demo/callback example does consumes the void* and makes it unavailable for use from Python.

Instead of making the void* user data point to the Python callable, it could be a tuple of length 2.  One for the callable and the other for the user data.

In Python you can always pass user data using functools.partial

[Robert Bradshaw]

On Tue, Nov 13, 2012 at 2:13 PM, Eric Frederich
<eric.fr...@gmail.com> wrote:
> Interesting.
> Other than wrapping the functions for use from Python what would be another
> use case?

Essentially any function that wants to use non-global runtime data.

> Now I'm just thinking out loud here....
>
> If the void* for user data has other uses in C perhaps it does in Python as
> well.
> Creating the bindings as the cython/Demo/callback example does consumes the
> void* and makes it unavailable for use from Python.
>
> Instead of making the void* user data point to the Python callable, it could
> be a tuple of length 2. One for the callable and the other for the user
> data.
>
> In Python you can always pass user data using functools.partial

In C, function pointers are just integers, addresses at which the CPU
should start reading instructions to execute from. It's also hard(er)
to create new C functions at runtime. Python functions are much
richer, e.g. you can attach data to them and they can enclose state,
so you don't need this. (As you note, once you have a user-supplied
void* you can pass as much data as you want.)

[Stefan Behnel]

Robert Bradshaw, 13.11.2012 23:29:

> On Tue, Nov 13, 2012 at 2:13 PM, Eric Frederich wrote:
>> If the void* for user data has other uses in C perhaps it does in Python as
>> well.
>> Creating the bindings as the cython/Demo/callback example does consumes the
>> void* and makes it unavailable for use from Python.
>>
>> Instead of making the void* user data point to the Python callable, it could
>> be a tuple of length 2. One for the callable and the other for the user
>> data.
>>
>> In Python you can always pass user data using functools.partial
>
> In C, function pointers are just integers, addresses at which the CPU
> should start reading instructions to execute from. It's also hard(er)
> to create new C functions at runtime. Python functions are much
> richer, e.g. you can attach data to them and they can enclose state,
> so you don't need this. (As you note, once you have a user-supplied
> void* you can pass as much data as you want.)

Exactly, you can pass a Python object reference to a nested function, for
example, which inherits its closure (and thus state) from the environment
it was defined in. You could even pass a running Python generator or
coroutine and instead of calling a function in the C callback you'd send
some data into the coroutine or call next() on it. All you really need for
this is a way to pass a single void* through C space, and that's the
problem here.

A less comfortable but still common way to handle this is the use of thread
local storage, in case the callback occurs during a call into the C library
and not at an arbitrary point, e.g. timer triggered. It's a bit more
overhead, but at least it fills the gap of a missing user data pointer in
the C API.

Stefan

[Eric Frederich]

Can you elaborate on this solution using thread local storage to get around the fact that the API does not support user data?

Thanks,

~Eric

[Eric Frederich]

Could someone point me to a reference stating something to this effect?
I'd like to report this as a bug to the 3rd party library I am dealing with here.
Some kind of guidelines, standard practice, etc.

On Tue, Nov 13, 2012 at 5:02 PM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no> wrote:

[Bradley Froehle]

Returning to what Robert said a few posts ago, you might be able to survive with a run-time generated C function using, e.g., using the callback interface in ctypes.  See http://docs.python.org/2/library/ctypes.html#callback-functions

I'd suggest asking the developers of the 3rd party library how they would handle this situation.  They might be able to provide more insight or an alternative approach.

-Brad