boxing/unboxing pointer to pass between C and python

[Lawrence Mitchell]

Hi all,

I'm using cython to wrap an existing C library is being used as
part of a python library for runtime code generation.

The basic setup is that we have a bunch of python objects, each
of which has a C counterpart (a struct)

The C struct is wrapped in a cython cdef class. So far, so good.

Now, we're generating C code at runtime that manipulates the data
associated with these objects and compiling and calling that with
instant.

For almost all use cases, this works fine: the data we need to
pass into the function started life on the python side. But for
one type of object, the data pointer that needs to be passed to
the compiled function lives in C land.

So we need a way of wrapping this pointer up as a python object
to pass back to python, and then unwrapping it in the C code.

At the moment we're doing this:


cdef cython_class:
cdef core.c_struct _handle
def __cinit__(self, python_class):
...
self._handle = core.build_c_struct(...)

The python class looks like:

def python_class(object):
def __init__(self, ...):
...
self._handle = cython_class(self)

Now we build some C code to compile:

_fun = instant.inline("code_that_needs_c_struct")

_fun(python_class_instance._handle)

To get the C struct from this object we're currently doing:


typedef struct {
PyObject_HEAD;
c_struct _handle;
} cython_c_struct;
void code_that_needs_c_struct(PyObject *o)
{
c_struct foo = ((cython_c_struct *)o)->_handle;
...
}

This works. But obviously we're worried that we're relying on an
internal Cython implementation detail of how it lays its data
structures out. Is this the best way to do things, or have we
missed some obvious correct method?

Cheers,

Lawrence
--
Lawrence Mitchell <we...@gmx.li>

[Henry Gomersall]

On Wed, 2012-07-25 at 15:31 +0100, Lawrence Mitchell wrote:
> Is this the best way to do things, or have we
> missed some obvious correct method?

What happens if you just naively cast the pointer to an intptr_t and
pass it around as an integer?

I'm not saying this is the right way to do it, just that I'm curious
myself as to why it's not a reasonable way.

Cheers,

Henry

[Dag Sverre Seljebotn]

Have a look at the CPython Capsule API.

Passing an uintptr_t works just fine, it's just that it's "wrong"; e.g.
when printing the object Python side it will be more informative to know
that it's a capsule than just have some semi-random number.

Dag

[Dag Sverre Seljebotn]

If you really want to avoid the overhead of doing malloc/free and embed
the data in the Python object's memory, creating a cdef class is the way
to do it. But you could write a module-level cdef function in Cython:

cdef c_struct get_struct_from_cython_wrapper(cython_class o):
return o._handle

# call function implemented in C passing the address to the Cython
# 'getter'
set_c_side_cython_class_getter(&get_struct_from_cython_wrapper)

Then you don't rely on Cython internals any longer.

Dag

[Lawrence Mitchell]

Dag Sverre Seljebotn wrote:
> On 07/25/2012 05:48 PM, Dag Sverre Seljebotn wrote:

[...]

>> Have a look at the CPython Capsule API.

> If you really want to avoid the overhead of doing malloc/free and
> embed the data in the Python object's memory, creating a cdef class is
> the way to do it. But you could write a module-level cdef function in
> Cython:

> cdef c_struct get_struct_from_cython_wrapper(cython_class o):
> return o._handle

> # call function implemented in C passing the address to the Cython
> # 'getter'
> set_c_side_cython_class_getter(&get_struct_from_cython_wrapper)

> Then you don't rely on Cython internals any longer.

> Dag

>> Passing an uintptr_t works just fine, it's just that it's "wrong"; e.g.
>> when printing the object Python side it will be more informative to know
>> that it's a capsule than just have some semi-random number.

Thanks for these varied suggestions. In the end, I went with the
uintptr_t approach. I expect that the python-side object will
never need printing anyway (except from a C level debugger).