Help with implementing callback functions using ctypes

[jamadagni]

Hello. I am running Kubuntu Raring with Python 3.3.1. (I mostly don't myself use Py2.)

I have the below C program spiro.c (obviously a simplified testcase) which I compile to a sharedlib using clang -fPIC -shared -o libspiro.so spiro.c, sudo cp to /usr/lib and am trying to call from a Python script spiro.py using ctypes. However it would seem that the first call to the callback results in a segfault. (The line before the callback is executed.) I tried debugging using gdb but perhaps because of my insufficient knowledge I couldn't find out what is causing the problem.

I hope some of the kind knowledgeable souls here can help me out:

-- spiro.c

# include <stdbool.h>
# include <stdio.h>

typedef struct _bezctx bezctx ;
struct _bezctx {
void ( * moveto ) ( bezctx * bc, double x, double y, bool starts_open_path ) ;
void ( * lineto ) ( bezctx * bc, double x, double y ) ;
void ( * curveto ) ( bezctx * bc, double c1x, double c1y, double c2x, double c2y, double x, double y ) ;
void ( * track_seg ) ( bezctx * bc, unsigned int start_node_idx ) ;
} ;

typedef struct {
double x, y ;
char ty ;
} spiro_cp ;

bool spiro_to_bezier_strict ( const spiro_cp src [], int src_len, bezctx * bc ) {
for ( int i = 0 ; i < src_len ; ++i ) {
const spiro_cp * p = &(src[i]) ;
printf ( "%g %g %c\n", p->x, p->y, p->ty ) ;
if ( i % 2 == 0 )
bc -> moveto ( bc, p->x, p->y, i % 3 ) ;
else
bc -> lineto ( bc, p->x, p->y ) ;
bc -> curveto ( bc, 233, 150, 267, 150, 300, 100 ) ;
bc -> track_seg ( bc, i ) ;
}
return src_len % 2 == 0 ;
}

-- spiro.py

from ctypes import *

# incomplete struct definition needed for pointer below
class bezctx ( Structure ) : pass

# pointer to function types for callback
MoveToFnType = CFUNCTYPE ( None, POINTER ( bezctx ), c_double, c_double, c_bool )
LineToFnType = CFUNCTYPE ( None, POINTER ( bezctx ), c_double, c_double )
CurveToFnType = CFUNCTYPE ( None, POINTER ( bezctx ), c_double, c_double, c_double, c_double, c_double, c_double )
TrackSegmentFnType = CFUNCTYPE ( None, POINTER ( bezctx ), c_uint )

# completing the struct definition
bezctx._fields_ = [ ( "moveto" , MoveToFnType ),
( "lineto" , LineToFnType ),
( "curveto" , CurveToFnType ),
( "track_seg", TrackSegmentFnType ) ]

# the python callbacks
def moveTo ( bc, x, y, startsOpenPath ) :
print ( "moveTo", bc, x, y, startsOpenPath )
def lineTo ( bc, x, y ) :
print ( "lineTo", bc, x, y )
def curveTo ( bc, c1x, c1y, c2x, c2y, x, y ) :
print ( "curveTo", bc, c1x, c1y, c2x, c2y, x, y )
def trackSegment ( bc, start_node_idx ) :
print ( "trackSegment", bc, start_node_idx )

# instance of struct
bc = bezctx ()
bc . moveto = MoveToFnType ( moveTo )
bc . lineto = LineToFnType ( lineTo )
bc . curveto = CurveToFnType ( curveTo )
bc . track_seg = TrackSegmentFnType ( trackSegment )

# another struct
class spiro_cp ( Structure ) :
_fields_ = [ ( "x", c_double ), ( "y", c_double ), ( "ty", c_char ) ]

# array of struct instances
srclist = [ spiro_cp ( 147, 215, b'o' ),
spiro_cp ( 168, 136, b'o' ),
spiro_cp ( 294, 184, b'[' ),
spiro_cp ( 381, 136, b']' ),
spiro_cp ( 445, 204, b'c' ),
spiro_cp ( 364, 235, b'c' ),
spiro_cp ( 266, 285, b'v' ) ]
src = ( spiro_cp * len ( srclist ) ) ( *srclist )

# open the sharedlib
libspiro = CDLL ( "libspiro.so" )
spiro_to_bezier_strict = libspiro.spiro_to_bezier_strict

# call the C function
spiro_to_bezier_strict ( src, len ( src ), bc )

-- bash session

$ python3 spiro-ctypes.py
147 215 o
Segmentation fault (core dumped)

[dieter]

jamadagni <sam...@gmail.com> writes:
> ...
I cannot help you with "ctypes". But, if you might be able to use
"cython", then calling callbacks is not too difficult
(you can find an example in e.g. my "dm.xmlsec.binding").

Note, however, that properly handling the GIL ("Global Interpreter Lock")
may be of great importance when the Python/C boundary is crossed --
at least when you intend to use your code in a multi thread environment.

[Stefan Behnel]

dieter, 09.05.2013 07:54:

> jamadagni writes:
>> ...
> I cannot help you with "ctypes". But, if you might be able to use
> "cython", then calling callbacks is not too difficult

+1 for using Cython. It also has (multi-)source level gdb support, which
greatly helps in debugging crashes like this one.

http://docs.cython.org/src/userguide/debugging.html

> (you can find an example in e.g. my "dm.xmlsec.binding").

An "official" example is here:

https://github.com/cython/cython/tree/master/Demos/callback

> Note, however, that properly handling the GIL ("Global Interpreter Lock")
> may be of great importance when the Python/C boundary is crossed --
> at least when you intend to use your code in a multi thread environment.

Cython makes this really easy. You can use the "with" statement to release
the GIL when you don't need it, and the compiler will produce errors when
you try to do things that require the GIL while you don't own it.

Stefan

[Nobody]

On Wed, 08 May 2013 04:19:07 -0700, jamadagni wrote:

> I have the below C program spiro.c (obviously a simplified testcase)
> which I compile to a sharedlib using clang -fPIC -shared -o libspiro.so
> spiro.c, sudo cp to /usr/lib and am trying to call from a Python script
> spiro.py using ctypes. However it would seem that the first call to the
> callback results in a segfault.

> # call the C function
> spiro_to_bezier_strict ( src, len ( src ), bc )

This line should be:

spiro_to_bezier_strict ( src, len ( src ), byref(bc) )

Without the byref(...), it will try to pass a copy of the structure rather
than passing a pointer to it.

[jamadagni]

On Friday, May 10, 2013 12:02:08 AM UTC+5:30, Nobody wrote:
> This line should be:
> spiro_to_bezier_strict ( src, len ( src ), byref(bc) )
> Without the byref(...), it will try to pass a copy of the structure rather
> than passing a pointer to it.

Wow silly silly mistake of mine, passing an object where a pointer is expected. Alternatively, if I specify:

spro_to_bezier_strict.argtypes = [ POINTER(spiro_cp), c_int, POINTER(bezctx) ]

it works equally good and apparently automatically converts the object specified to a pointer to it (which is good and intuitive, I guess, though against C practice).

I guess if one wants to be safe when interfacing to C functions, one should always specify argtypes (even if not restype).

But really, a segfault? I would expect a more meaningful error or an exception to be raised. Or is the black magic occurring at a level lower than where Python's control goes?

If I had seen this post I might not have even spent time on Cython. (For some reason Google Groups doesn't send me email updates even if I ask it to.) Using CTYpes I was able to implement a callback with less coding (for this particular requirement) than Cython which required an extra wrapper layer on top of the library to pass through a void* representing the Python callback (or a state variable would have to be used which may mean no thread-safety).

But I guess Cython would be faster for those who want that. Even though there is an extra function call every time (for the wrapper) it would be faster since it's at the C level rather than at the Python level (or something like that).

Thanks to all who helped on this, and also the people over at the cython-users list who helped me on the Cython wrapper too.

[Dan Stromberg]

On Wed, May 8, 2013 at 10:54 PM, dieter <die...@handshake.de> wrote:

jamadagni <sam...@gmail.com> writes:
> ...
I cannot help you with "ctypes". But, if you might be able to use
"cython", then calling callbacks is not too difficult

(you can find an example in e.g. my "dm.xmlsec.binding").

Note, however, that properly handling the GIL ("Global Interpreter Lock")
may be of great importance when the Python/C boundary is crossed --
at least when you intend to use your code in a multi thread environment.

Cython is good.

So is the new cffi, which might be thought of as a safer (API-level) version of ctypes (which is ABI-level).

[Shriramana Sharma]

On Friday, May 24, 2013 8:56:28 AM UTC+5:30, Dan Stromberg wrote:
> Cython is good. So is the new cffi, which might be thought of as a
> safer (API-level) version of ctypes (which is ABI-level).

Hi -- can you clarify what is this new CFFI and where I can get it? In the Python 3 library reference I seem to see only CTypes.

[Carlos Nepomuceno]

https://cffi.readthedocs.org/en/release-0.6/

----------------------------------------
> Date: Sun, 26 May 2013 09:12:10 -0700
> Subject: Re: Help with implementing callback functions using ctypes
> From: sam...@gmail.com
> To: pytho...@python.org

> --
> http://mail.python.org/mailman/listinfo/python-list

[Dan Stromberg]

CFFI is like CTypes, but it allows you to do things like see C preprocessor symbols.

https://cffi.readthedocs.org/en/release-0.6/

I don't think it comes with CPython, at least not yet.  But I believe it comes with the latest Pypy.