crash when returning from a callback function

[Sarvi Shanmugham]

I am working with a multithreaded communication library libmylib.so which starts multiple threads to do deal with external communication.

This library that has been in use for a while by other programs and works fine. I am now wrapping it with cffi.

The library needs a C callback function defined as 

typedef int (*mylib_client_resp_cb) (int, my_client_resp_t ev, my_client_resp_attr_t rattr);

I have defined the callback as following

@ffi.callback('int(*)(oirlib_handle, my_client_resp_t, my_client_resp_attr_t)')

def my_client_resp_cb(ohandle, ev, rattr):

     print ev

     time.sleep(10)

     print "callback done"

     return 0

The library has an API that needs to register a callback function. I am calling it as follows

mylib.mylib_server_register(server_handle, my_client_resp_cb)

mylib.mylib_sendevent(event) # This sends out a message and expect the response to be available in the callback function

while(True):

    time.sleep(1)

This program crashes the moment I return from the callback function.

 I know because I adjusted the sleep in the callback function

Trouble is, there is no traceback or core. All I see is an exit code of 11 which I read is SEGV. 

Confirmed by installing a siignal 11 handler in the python program which gets called.

Question:

    Has the CFFI been tested with wrapping C libraries that are themselves multithreaded and doing callbacks

I am gonna try to debug this further an may be try to reproduce it in a simpler problem. 

But just wanted to check if this is known problem.

Thx,

Sarvi 

I am trying to see if this can be reproruced

[Sarvi Shanmugham]

Was able to reproduce it

>>> import nova.cffitest

{'dummy_func1': <cdata 'struct test_struct_ *(*)(struct test_struct_ *)' 0xf71f2700>, 'dummy_func2': <function newfunc at 0x8991e2c>, 'spi_iterate_error_code': <cdata 'int(*)(struct tdlhandle_s *, structtdlhandle_s *, struct spi_error_code_ *, int(*)(struct tdlhandle_s *, struct tdlhandle_s *, struct tdlhandle_s *, int, void *), void *)' 0xf71f2730>, 'threaded_ballback_test': <cdata 'int(*)(int, int(*)(int, int))' 0xf71f27a0>}

>>>

>>> def mycallback(x,y):

...     print x, y

...     return 0

...

>>> x=nova.cffitest.callback('int(*)(int,int)', mycallback)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

AttributeError: 'module' object has no attribute 'callback'

>>> x=nova.cffitest.ffi.callback('int(*)(int,int)', mycallback)

>>> print x

<cdata 'int(*)(int, int)' calling <function mycallback at 0x8991e64>>

>>> nova.cffitest.threaded_ballback_test(1000,x)

0

>>> Beginning Wait ....

Ending Wait ....

10 10

Segmentation fault (core dumped)

[128:~]$

My code library header code

typedef int (*mycallback_func_t)(int, int);

int threaded_ballback_test(int handle, mycallback_func_t mycb);

My library C code

void *my_wait_function( void *ptr )

{

     mycallback_func_t cbfunc;

     int x;

     cbfunc = (mycallback_func_t) ptr;

     printf("Beginning Wait .... \n");

     sleep(5);

     printf("Ending Wait .... \n");

     x=cbfunc(10, 10);

     printf("Doing Callback Wait .... \n");

     return NULL;

}

int threaded_ballback_test(int handle, mycallback_func_t mycb)

{

    pthread_t thread;

    int x;

    x = pthread_create( &thread, NULL, my_wait_function, (void*) mycb);

    return 0;

}

Thx,

Sarvi

[Armin Rigo]

Hi Sarvi,

On Thu, Sep 27, 2012 at 2:41 AM, Sarvi Shanmugham <sarv...@gmail.com> wrote:
> x = pthread_create( &thread, NULL, my_wait_function, (void*) mycb);

Your library is creating threads from C, without notifying
CPython/PyPy that it does so. This is what leads to crashes. New
threads are either expected to be created from Python, or there is
some C API that can be called from a CPython extension module. Of
course, in the case of CFFI, it cannot work like that. I'm unfamiliar
with it, but we might have to look it up and try to design something
else for CFFI.


A bientôt,

Armin.

[Armin Rigo]

ReHi,

For CPython C extensions, this is documented here:
http://docs.python.org/c-api/init.html#non-python-created-threads


A bientôt,

Armin.

[Armin Rigo]

Hi again,

Sorry, it seems that _cffi_backend is already doing the right thing.
Investigating more...


Armin

[Armin Rigo]

Hi,

Bah. It seems everything works, but under the conditions that Python
threads have already been internally initialized. Try adding this
line at the start of the .py script:

import thread; thread.start_new_thread(lambda: None, ())


A bientôt,

Armin.

[Armin Rigo]

Hi,

Fixed (caae153920ef).


Armin

[Sarvi Shanmugham]

Awesome. Thanks.

On Thursday, September 27, 2012 1:50:05 AM UTC-7, Armin Rigo wrote:

Hi,

Fixed (caae153920ef).


Armin