Accessing python functions in callbacks with return values

252 views
Skip to first unread message

Toni Barth

unread,
Jan 30, 2015, 1:25:20 AM1/30/15
to cython...@googlegroups.com
Hello Cython users,

i'm currently facing a quite interesting problem.
I already wrote a callback function (declared as cdef) which gets called
by a binary c file I linked to. This callback, without a return value
(means void) can access python functions, means it can call a user
defined python function, so it can be used as callback bridge to call a
python function as real callback, just like the cython example.
But when this C callback function gets a return value, things get much
stranger. Since the cython source doesn't contain a demo with return
value, I wasn't able to check it out.
Each time the C binary wants to execute my c function which now has a
return value (which is compatible to the binary's definitions), the code
causes a segfault at the point where a pythonic function is called.
I also tried commenting this step out and try to find a solution, but
then the code crashes at the next pythonic function call, so it seems to
have problems calling into the python namespace when called by the
binary callback thread.
I also read that a c library which wants to call cython code needs to
execute Py_Initialize() first, but since it's a binary i'm unable to do so.
It's also curious that a cython callback without return value is able to
call into the python namespace, but a cython callback with return value
isn't.
Since my c callback with return value needs to call a python function
(since it's a callback bridge again) there is no way to get around
calling a pythonic function.

Did anyone of you face that problem before and maybe found a, if
possible, cython-python-only solution?

Best Regards.

Toni Barth

Robert Bradshaw

unread,
Jan 30, 2015, 1:50:32 AM1/30/15
to cython...@googlegroups.com
I'm not quite following exactly how things are set up. Could you post
a bare-bones example of what works and the minimal change to break it?

Toni Barth

unread,
Jan 30, 2015, 3:09:34 AM1/30/15
to cython...@googlegroups.com
Hi,

of course I can.

Imagine a binary library method called RegisterCallback which takes a
callback declared like this:
ctypedef void (*Callback)(char *somestring);

And image the same library containing a function
RegisterCallbackWithReturn accepting a callback like this:
ctypedef unsigned long (*Callback2)(char *somestring2);

Then I'll write these two callbacks in cython:

import somepythonmodule
cdef void c_Callback(char *somestring):
somepythonmodule.firework()

Adding this callback with:
...
RegisterCallback(<Callback>c_Callback)

works fine. The callback is called correctly, does it's stuff and
finished correctly, everything okay. But:

Declaring c_Callback2 as follows:

import somepythonmodule
cdef unsigned long c_Callback2(char *somestring2):
somepythonmodule.firework()
return 0

Registering this with:
RegisterCallback2(<Callback2>c_Callback2)

will still work of course, but the whole program crashes when the binary
calls the callback.
The crash occures in __pyx_PyObject_Call, as gdb states.

As I stated before, I read that the binary should use Py_Initialize()
before to access the python namespace, but as I said, the binary is a
binary, no way to change it for me.
I'm currently trying to write a c library wrapping the functionality of
being the callback function and calling the cython function with the
Py_Initialize() and Py_Finalize() thing, but i'm stuck doing that, c is
quite new to me and i'm sure that's not the thing to do to wrap an
external c lib in python... not writing a new c lib to wrap an existing
c lib, so you need to actually wrap 2 libs in cython to make them
accessible in python, lol.

If these examples aren't precise enough I will extract the real code
part from my project if you request it.

Thanks for your help, really!

Best Regards.

Toni Barth

Robert Bradshaw

unread,
Feb 3, 2015, 1:47:59 AM2/3/15
to cython...@googlegroups.com
On Fri, Jan 30, 2015 at 12:07 AM, Toni Barth <hiho...@googlemail.com> wrote:
> Hi,
>
> of course I can.
>
> Imagine a binary library method called RegisterCallback which takes a
> callback declared like this:
> ctypedef void (*Callback)(char *somestring);
>
> And image the same library containing a function RegisterCallbackWithReturn
> accepting a callback like this:
> ctypedef unsigned long (*Callback2)(char *somestring2);
>
> Then I'll write these two callbacks in cython:
>
> import somepythonmodule
> cdef void c_Callback(char *somestring):
> somepythonmodule.firework()
>
> Adding this callback with:
> ...
> RegisterCallback(<Callback>c_Callback)
>
> works fine. The callback is called correctly, does it's stuff and finished
> correctly, everything okay.

Despite the fact that you never called Py_Initialize()? That's surprising.

FWIW, the cast to Callback should be unneeded.

> But:
>
> Declaring c_Callback2 as follows:
>
> import somepythonmodule
> cdef unsigned long c_Callback2(char *somestring2):
> somepythonmodule.firework()
> return 0
>
> Registering this with:
> RegisterCallback2(<Callback2>c_Callback2)
>
> will still work of course, but the whole program crashes when the binary
> calls the callback.
> The crash occures in __pyx_PyObject_Call, as gdb states.

It literally calls exactly the same code, but crashes this time? That
is quite strange.

> As I stated before, I read that the binary should use Py_Initialize() before
> to access the python namespace, but as I said, the binary is a binary, no
> way to change it for me.
> I'm currently trying to write a c library wrapping the functionality of
> being the callback function and calling the cython function with the
> Py_Initialize() and Py_Finalize() thing, but i'm stuck doing that,

Call Py_Initialize() wherever you're calling the RegisterCallbacks.
> --
>
> --- You received this message because you are subscribed to the Google
> Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Toni Barth

unread,
Feb 3, 2015, 7:16:03 AM2/3/15
to cython...@googlegroups.com
Hello Robert,

it isn't the same code, the first method doesn't return anything (void,
no return value), the second function returns an unsigned int and also
uses return 0. That's, as I debugged, what crashes everything. At least
the return value changes something cython or python internally.
Can you explain where to call Py_Initialize() exactly? Because cython
doesn't know these Py_Initialize() and Py_Finalize() functions and I
can't change the binary, as stated. Thanks.

Best Regards.

Toni Barth

Robert Bradshaw

unread,
Feb 3, 2015, 12:07:17 PM2/3/15
to cython...@googlegroups.com
On Tue, Feb 3, 2015 at 2:52 AM, Toni Barth <hiho...@googlemail.com> wrote:
> Hello Robert,
>
> it isn't the same code, the first method doesn't return anything (void, no
> return value), the second function returns an unsigned int and also uses
> return 0. That's, as I debugged, what crashes everything. At least the
> return value changes something cython or python internally.

OK, so it's the same code *except* for the return value? And you're
returning the constant 0? What if you omit the return value (which
will return 0)?

> Can you explain where to call Py_Initialize() exactly? Because cython
> doesn't know these Py_Initialize() and Py_Finalize() functions and I can't
> change the binary, as stated. Thanks.

Presumably you're calling

RegisterCallback(<Callback>c_Callback);

somewhere, change that to

Py_Initialize();
RegisterCallback(<Callback>c_Callback);

Toni Barth

unread,
Feb 3, 2015, 1:47:25 PM2/3/15
to cython...@googlegroups.com
Hi,

it's nearly the exact code, yes, just the return value specification
(void and unsigned int) differs and the existing return value in the
second case. Fact is, that not the value of the return value matters,
but the fact that it returns anything. I could even return -21405 or
some other random number, the function will crash the program anyway.

And at next the problem:
RegisterCallback(...)

That's a cython function, located in a .pyx file, wrapped by a .pxd file
which retrieved this function from a binary via:
cdef extern from "file.h":
RegisterCallback(...)

You see, now way to implement the Py_Initialize() except in the cython
code, and as I stated before, cython doesn't recognize the function
names Py_Initialize() or Py_Finalize(). Do I have anything special to
include or how can I do this?

Best Regards.
Toni Barth

Nils Bruin

unread,
Feb 3, 2015, 2:41:52 PM2/3/15
to cython...@googlegroups.com
On Tuesday, February 3, 2015 at 10:47:25 AM UTC-8, Toni Barth wrote:
You see, now way to implement the Py_Initialize() except in the cython
code, and as I stated before, cython doesn't recognize the function
names Py_Initialize() or Py_Finalize(). Do I have anything special to
include or how can I do this?

A simple

cdef extern from "Python.h":
    void Py_Initialize()

should do the trick then!. You can probably leave the "Python.h" off, since that header file is included anyway.
You should probably also try to let your void-returning Callback actually do some non-trivial python things (e.g., build a list and print it or something like that). If that fails too, you have can have a reasonable suspicion that Py_Initialize hasn't run yet. If that succeeds, though, then Py_Initialize must somehow have executed already and then that is obviously not the obstacle.

(you can write a cdef cython function that makes no calls whatsoever to the python library and hence can execute without Py_Initialize)

How does the execution path ever get to the RegisterCallback? The right place to run Py_Initialize would of course be at the top of your main() function, but apparently you're not providing that.
 

Toni Barth

unread,
Feb 3, 2015, 3:37:15 PM2/3/15
to cython...@googlegroups.com
Hello,

thanks, I will test that later.

The RegisterCallback() will be called when the user wants that. I'm currently creating a wrapper for the binary lib, and if the user calls mywrapper.ThisExplicitFunction(callback), then callback will be saved in some background pythonic class and the c_callback function will be registered with RegisterCallback() to be called each time the binary wants to do that. The c_callback() will then retrieve the pythonic callback() function the user once registered and call that, since binary libraries aren't able to call real pythonic functions directly.

I will post back after testing your Py_Initialize() and Py_Finalize() attempts. Thanks for your response.

Best Regards.
Toni Barth

Toni Barth

unread,
Feb 5, 2015, 9:36:01 AM2/5/15
to cython...@googlegroups.com
Hello,

I tested the code by using Py_Initialize() and Py_Finalize().

I have no choice but using it directly as first and last statement inside the callback function, because I can't edit the binary as stated.
I don't know if this is supposed to work anyway, but it fails for me inside PyErr_Restore. I also don't see the cython source code in gdb where the problem fails, since my cython debug on linux seems to fail and on Windows it doesn't debug anyway (linux always says ".cygdbinit: no such file or directory").
So I think it fails when calling Py_Initialize(), since the last traceback also states that it failed in _start().

Writing a function which doesn't call any python library function isn't possible, since the c callback needs to retrieve the python function the user defined and call that, and calling a python function is kind of pythonic...

So, here's the backtrace from gdb for you:
#0  PyErr_Restore (traceback=0x0, value="no locals when loading 'True'",
    type=<type at remote 0x915cc0>) at ../Python/errors.c:39
#1  PyErr_SetObject (value="no locals when loading 'True'",
    exception=<type at remote 0x915cc0>) at ../Python/errors.c:57
#2  PyErr_Format () at ../Python/errors.c:551
#3  0x0000000000426a47 in PyEval_EvalFrameEx (f=<unknown at remote 0x2>,
    throwflag=18) at ../Python/ceval.c:2037
#4  0x00000000004def64 in PyEval_EvalCodeEx () at ../Python/ceval.c:3253
#5  0x0000000000518d7f in PyEval_EvalCode (
    locals={'Bass4Py': None, 'b': None, 'd': None, 'f': None, '__builtins__': <m
odule at remote 0x7ffff7fa9b08>, 'cb': None, '__file__': None, '__package__': No
ne, 'url': None, 's': None, 'exit': None, 'time': None, '__name__': None, '__doc
__': None},
    globals={'Bass4Py': None, 'b': None, 'd': None, 'f': None, '__builtins__': <
module at remote 0x7ffff7fa9b08>, 'cb': None, '__file__': None, '__package__': N
one, 'url': None, 's': None, 'exit': None, 'time': None, '__name__': None, '__do
c__': None}, co=<code at remote 0x7ffff7ee4c30>) at ../Python/ceval.c:667
#6  run_mod.lto_priv () at ../Python/pythonrun.c:1371
#7  0x000000000050eeda in PyRun_FileExFlags () at ../Python/pythonrun.c:1357
#8  0x000000000050dbf7 in PyRun_SimpleFileExFlags ()
    at ../Python/pythonrun.c:949
#9  0x00000000004b0b5b in Py_Main () at ../Modules/main.c:640
#10 0x00007ffff6f14b45 in __libc_start_main (main=0x4b0430 <main>, argc=2,
    argv=0x7fffffffe6d8, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffe6c8) at libc-start.c:287
#11 0x00000000004b0352 in _start ()

Does any of you have any clue of what to do next? :(.

Thanks for help.

Best Regards.

Toni Barth

Am 03.02.2015 um 20:41 schrieb Nils Bruin:

Stefan Behnel

unread,
Feb 6, 2015, 1:54:43 AM2/6/15
to cython...@googlegroups.com
Toni Barth schrieb am 03.02.2015 um 20:48:
> The RegisterCallback() will be called when the user wants that. I'm
> currently creating a wrapper for the binary lib, and if the user calls
> mywrapper.ThisExplicitFunction(callback), then callback will be saved in
> some background pythonic class and the c_callback function will be
> registered with RegisterCallback() to be called each time the binary wants
> to do that. The c_callback() will then retrieve the pythonic callback()
> function the user once registered and call that, since binary libraries
> aren't able to call real pythonic functions directly.

Ah, so the setup is that you first start your Python runtime and that
imports everything else? In that case, you must call neither
Py_Initialize() nor Py_Finalize(). Python does it all for you.

In fact, calling Py_Finalize() will definitely crash your program, as it
kills Python. (And I see in your last email that that's what happened.)

Stefan

Toni Barth

unread,
Feb 6, 2015, 2:20:04 AM2/6/15
to cython...@googlegroups.com
Hello Stefan,

Okay, so that resolves the error I encountered. But now i'm totally
confused.
My c callback crashes anyway when calling some pythonic function,
encountering an pythonic error in __pyx_PyObject_Call(), if I do
remember right. So, if it doesn't have something to do with
Py_Initialize(), what can be the problem then?

Best Regards.

Toni Barth

Stefan Behnel

unread,
Feb 6, 2015, 3:23:22 AM2/6/15
to cython...@googlegroups.com
First of all, as others have noted, whether the callback function returns a
value or not cannot make a difference here. You should concentrate your
debugging efforts on other factors.

Does the native code execute your callback directly (e.g. while you are
calling into it), or does it store it away and call it later? Are threads
involved? Natively started threads require a bit of initialisation to run
Python code.

Is the way the two callbacks are called in any way different for the two
types of callbacks?

Are you sure that the function (or whatever Python object) you are calling
from your C callback is a live object (i.e. hasn't been garbage collected
yet) with a valid object reference? Have you tried calling some other
object instead, just for testing?


> If these examples aren't precise enough I will extract the real code part
> from my project if you request it.

That would help, yes.

Stefan


PS: when replying, please take the time to strip down the original email to
the parts that are necessary to understand your reply, and then reply below
the parts that you refer to.

Toni Barth

unread,
Feb 9, 2015, 2:25:35 AM2/9/15
to cython...@googlegroups.com
Hello,
> the c callback is registered with RegisterCallback() which stores the
> pythonic function in a pythonic class which the c callback knows about
> and can/will find to execute the pythonic function.
>
> After registering the callback I don't have any clue what the binary
> does with it, if it executes it in another thread or whatever.
>
> Just on purpose, what can I do to enable thread support? Just if the
> binary executes the callback in an external thread, which would explain
> why it is unable to find any pythonic stuff.
>
> I don't know if the way they're called is different however, it's a
> binary. I just know how they are registered and I know the prototypes of
> the debug functions.
>
> Yep, i'm sure they aren't garbage collected. The module to call was just
> imported by a simple pythonic import statement. Also, a cythonic call
> doesn't work, namely creating a new class inside that function, example:
>
> # file.pxd
>
> cdef int cb(void *data)
>
> cdef class MyData:
>
> cdef int somefunc(int test)
>
> # file.pyx
>
> cdef int cb(void *data):
>
> cdef MyData cdata=MyData()
> # ... remaining class stuff here
>
> Also this call seems to fail, in the same function,
> __pyx_PyObject_Call(), but i'm not sure, I didn't dig into this bug. I
> just commented it out and found it at the python code execution again,
> which is my main problem.
>
>
>
>
>> If these examples aren't precise enough I will extract the real code part
>> from my project if you request it.
> That would help, yes.
>
> Stefan
>
>
> PS: when replying, please take the time to strip down the original email to
> the parts that are necessary to understand your reply, and then reply below
> the parts that you refer to.

Thanks for your support.

Best Regards.
Toni Barth

PS: Is it okay this way, quoting?


Reply all
Reply to author
Forward
0 new messages