I am working with a multithreaded communication library libmylib.so which
starts multiple threads to do deal with external communication.
This library that has been in use for a while by other programs and works
fine. I am now wrapping it with cffi.

The library needs a C callback function defined as

typedef int (*mylib_client_resp_cb) (int, my_client_resp_t ev,
my_client_resp_attr_t rattr);

I have defined the callback as following
@ffi.callback('int(*)(oirlib_handle, my_client_resp_t,
my_client_resp_attr_t)')
def my_client_resp_cb(ohandle, ev, rattr):
     print ev
     time.sleep(10)
     print "callback done"
     return 0

The library has an API that needs to register a callback function. I am
calling it as follows
mylib.mylib_server_register(server_handle, my_client_resp_cb)
mylib.mylib_sendevent(event) # This sends out a message and expect the
response to be available in the callback function

while(True):
    time.sleep(1)

This program crashes the moment I return from the callback function.
 I know because I adjusted the sleep in the callback function

Trouble is, there is no traceback or core. All I see is an exit code of 11
which I read is SEGV.
Confirmed by installing a siignal 11 handler in the python program which
gets called.

Question:
    Has the CFFI been tested with wrapping C libraries that are themselves
multithreaded and doing callbacks

I am gonna try to debug this further an may be try to reproduce it in a
simpler problem.
But just wanted to check if this is known problem.
Thx,
Sarvi

I am trying to see if this can be reproruced

Was able to reproduce it

{'dummy_func1': <cdata 'struct test_struct_ *(*)(struct test_struct_ *)'
0xf71f2700>, 'dummy_func2': <function newfunc at 0x8991e2c>,
'spi_iterate_error_code': <cdata 'int(*)(struct tdlhandle_s *,
structtdlhandle_s *, struct spi_error_code_ *, int(*)(struct tdlhandle_s *,
struct tdlhandle_s *, struct tdlhandle_s *, int, void *), void *)'
0xf71f2730>, 'threaded_ballback_test': <cdata 'int(*)(int, int(*)(int,
int))' 0xf71f27a0>}
...     print x, y
...     return 0
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'callback'
<cdata 'int(*)(int, int)' calling <function mycallback at 0x8991e64>>
Ending Wait ....
10 10
Segmentation fault (core dumped)
[128:~]$

My code library header code

typedef int (*mycallback_func_t)(int, int);
int threaded_ballback_test(int handle, mycallback_func_t mycb);

My library C code
void *my_wait_function( void *ptr )
{
     mycallback_func_t cbfunc;
     int x;

     cbfunc = (mycallback_func_t) ptr;
     printf("Beginning Wait .... \n");
     sleep(5);
     printf("Ending Wait .... \n");
     x=cbfunc(10, 10);
     printf("Doing Callback Wait .... \n");
     return NULL;

int threaded_ballback_test(int handle, mycallback_func_t mycb)
{
    pthread_t thread;
    int x;

    x = pthread_create( &thread, NULL, my_wait_function, (void*) mycb);

    return 0;

Thx,
Sarvi

Hi Sarvi,

Your library is creating threads from C, without notifying
CPython/PyPy that it does so.  This is what leads to crashes.  New
threads are either expected to be created from Python, or there is
some C API that can be called from a CPython extension module.  Of
course, in the case of CFFI, it cannot work like that.  I'm unfamiliar
with it, but we might have to look it up and try to design something
else for CFFI.

A bientôt,

Armin.

ReHi,

For CPython C extensions, this is documented here:
http://docs.python.org/c-api/init.html#non-python-created-threads

A bientôt,

Armin.

Hi again,

Sorry, it seems that _cffi_backend is already doing the right thing.
Investigating more...

Armin

Hi,

Bah.  It seems everything works, but under the conditions that Python
threads have already been internally initialized.  Try adding this
line at the start of the .py script:

import thread; thread.start_new_thread(lambda: None, ())

A bientôt,

Armin.

Hi,

Fixed (caae153920ef).

Armin

Awesome. Thanks.