can pointers be stored in Python's dict or list?

[Mark Summerfield]

Hi,

(1)

I am using Cython to provide a Python wrapper to a C library.
The C library has a function which gives a handle (a pointer) which is
then used to call other functions.

I want to be able to store a dictionary of name-pointer pairs so that I
can resuse handles.

This doesn't seem to work with dict or list because of the
Cannot convert 'Thing*' to Python object
error.

Right now I have a static array and a dictionary:

cdef Thing* pointers[30]
indexForThingName = {}

but I'd really like to be able to store an arbitrary number of pointers,
ideally in a dict.

Is this possible?

(2)

Also, I tried doing

from libc.stdio cimport printf

and then in my code:

printf("%p", pointer)

but although it compiled without error, nothing was printed.

Thanks!

--
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy
"Programming in Python 3" - ISBN 0321680561
http://www.qtrac.eu/py3book.html

[Robert Bradshaw]

On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <li...@qtrac.plus.com> wrote:
> Hi,
>
> (1)
>
> I am using Cython to provide a Python wrapper to a C library.
> The C library has a function which gives a handle (a pointer) which is
> then used to call other functions.
>
> I want to be able to store a dictionary of name-pointer pairs so that I
> can resuse handles.
>
> This doesn't seem to work with dict or list because of the
> Cannot convert 'Thing*' to Python object
> error.
>
> Right now I have a static array and a dictionary:
>
> cdef Thing* pointers[30]
> indexForThingName = {}
>
> but I'd really like to be able to store an arbitrary number of pointers,
> ideally in a dict.
>
> Is this possible?

Sounds like you want http://docs.python.org/release/3.1.5/c-api/capsule.html

> (2)
>
> Also, I tried doing
>
> from libc.stdio cimport printf
>
> and then in my code:
>
> printf("%p", pointer)
>
> but although it compiled without error, nothing was printed.

Did it perhaps not flush? You also might want to try adding a newline.

- Robert

[Mark Summerfield]

Hi Robert,

On Fri, 14 Sep 2012 10:45:06 -0700
Robert Bradshaw <robe...@gmail.com> wrote:
> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <li...@qtrac.plus.com>
> wrote:
> > Hi,
> >
> > (1)
> >
> > I am using Cython to provide a Python wrapper to a C library.
> > The C library has a function which gives a handle (a pointer) which is
> > then used to call other functions.
> >
> > I want to be able to store a dictionary of name-pointer pairs so that I
> > can resuse handles.
> >
> > This doesn't seem to work with dict or list because of the
> > Cannot convert 'Thing*' to Python object
> > error.
> >
> > Right now I have a static array and a dictionary:
> >
> > cdef Thing* pointers[30]
> > indexForThingName = {}
> >
> > but I'd really like to be able to store an arbitrary number of
> > pointers, ideally in a dict.
> >
> > Is this possible?
>
> Sounds like you want
> http://docs.python.org/release/3.1.5/c-api/capsule.html

But that means I have to drop down to C. I want to stay in Python and
Cython. I was hoping that Cython offered some kind of wrapper type,
something that at the Python level would look like:

class Pointer(object):
def __init__(self, void *pointer):
cdef void *self.pointer = pointer # Doesn't work

that could be put in a dict or list. Then I could do things like this:

Pointer p = <void*>c_func_that_returns_a_pointer()
mylist.append(p)
...
Foo *q = <Foo*>mylist.pop()

Oh well, I'll see if I can use libcpp's map...

> > (2)
> >
> > Also, I tried doing
> >
> > from libc.stdio cimport printf
> >
> > and then in my code:
> >
> > printf("%p", pointer)
> >
> > but although it compiled without error, nothing was printed.
>
> Did it perhaps not flush? You also might want to try adding a newline.

I changed it to printf("%p\n", pointer) and now I get
Cannot convert 'void *' to Python object

--
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy

"Programming in Go" - ISBN 0321774639
http://www.qtrac.eu/gobook.html

[Robert Bradshaw]

You're already "in C" if you're dealing with pointers.

> I was hoping that Cython offered some kind of wrapper type,
> something that at the Python level would look like:
>
> class Pointer(object):
> def __init__(self, void *pointer):
> cdef void *self.pointer = pointer # Doesn't work
>
> that could be put in a dict or list. Then I could do things like this:
>
> Pointer p = <void*>c_func_that_returns_a_pointer()
> mylist.append(p)
> ...
> Foo *q = <Foo*>mylist.pop()

How about (untested)

from cpython.cobject cimport PyCObject_FromVoidPtr, PyCObject_AsVoidPtr

cdef void* p = <void*>c_func_that_returns_a_pointer()
mylist.append(PyCObject_FromVoidPtr(p, NULL)
...
Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop())

This should work for Python 2, use capsules (at the provided link) for
Python 3+. For a portable solution, you could cast the void* to a
intptr_t (or long long, if that's sufficient) which converts back and
forth from Python just fine (but is more hackish).

> Oh well, I'll see if I can use libcpp's map...
>
>> > (2)
>> >
>> > Also, I tried doing
>> >
>> > from libc.stdio cimport printf
>> >
>> > and then in my code:
>> >
>> > printf("%p", pointer)
>> >
>> > but although it compiled without error, nothing was printed.
>>
>> Did it perhaps not flush? You also might want to try adding a newline.
>
> I changed it to printf("%p\n", pointer) and now I get
> Cannot convert 'void *' to Python object

Are you still cimporting printf?

- Robert

[Bradley Froehle]

On Friday, September 14, 2012 12:55:03 PM UTC-7, Mark wrote:

Hi Robert,

On Fri, 14 Sep 2012 10:45:06 -0700
Robert Bradshaw <robe...@gmail.com> wrote:
> On Fri, Sep 14, 2012 at 8:11 AM, Mark Summerfield <li...@qtrac.plus.com>
> wrote:
> > Hi,
> >
> > (1)
> >
> > I am using Cython to provide a Python wrapper to a C library.
> > The C library has a function which gives a handle (a pointer) which is
> > then used to call other functions.
> >
> > I want to be able to store a dictionary of name-pointer pairs so that I
> > can resuse handles.
> >
> > This doesn't seem to work with dict or list because of the
> >     Cannot convert 'Thing*' to Python object
> > error.
> >
> > Right now I have a static array and a dictionary:
> >
> > cdef Thing* pointers[30]
> > indexForThingName = {}
> >
> > but I'd really like to be able to store an arbitrary number of
> > pointers, ideally in a dict.
> >
> > Is this possible?
>
> Sounds like you want
> http://docs.python.org/release/3.1.5/c-api/capsule.html

You should be able to do this in the realm of Cython (but still calling the CPython API).

Try something like the following:

from cython.cpython.pycapsule import PyCapsule_New, PyCapsule_GetPointer

p = PyCapsule_New(<void*> c_func_that_returns_a_pointer(), "Foo*", NULL)

mylist.append(p)

# ...

cdef Foo* q = <Foo*> PyCapsule_GetPointer(mylist.pop(), "Foo*")

You need to be careful with memory allocation here.  PyCapsule_New offers an argument for a destructor which you may need to use.

-Brad

[Mark Summerfield]

Hi Robert,

On Fri, 14 Sep 2012 13:11:16 -0700

I know but I only want to deal with pointers as handles to pass between
functions.

[snip]

> How about (untested)
>
> from cpython.cobject cimport PyCObject_FromVoidPtr,
> PyCObject_AsVoidPtr
>
> cdef void* p = <void*>c_func_that_returns_a_pointer()
> mylist.append(PyCObject_FromVoidPtr(p, NULL)
> ...
> Foo *q = <Foo*>PyCObject_AsVoidPtr(mylist.pop())
>
> This should work for Python 2, use capsules (at the provided link) for
> Python 3+. For a portable solution, you could cast the void* to a
> intptr_t (or long long, if that's sufficient) which converts back and
> forth from Python just fine (but is more hackish).

I'm using Python 3. The Cython docs are a bit frustrating at times.

However, thanks to your info I can now store void*s in Python dicts
etc:-D Here's the skeleton:

cimport cpython.pycapsule as pycapsule

d = {}
...
Foo *p = make_foo()
...
d[name] = pycapsule.PyCapsule_New(<void*>p, NULL, NULL)
...
capsule = d[name]
if not pycapsule.PyCapsule_IsValid(capsule, NULL):
raise ValueError("unexpected null ptr")
Foo *q = <Foo*>pycapsule.PyCapsule_GetPointer(capsule, NULL)


[snip]

> >> Did it perhaps not flush? You also might want to try adding a newline.
> >
> > I changed it to printf("%p\n", pointer) and now I get
> > Cannot convert 'void *' to Python object
>
> Are you still cimporting printf?

Ooops.. sorry, it now works perfectly.

Thanks v. much for all your help!

--
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy

[Czarek Tomczak]

Good to know there is such a thing as capsule. I would probably have used C++ map:

from libcpp.map cimport map 

[Mark Summerfield]

Hi Bradley,

On Fri, 14 Sep 2012 13:13:35 -0700 (PDT)
Bradley Froehle <brad.f...@gmail.com> wrote:
>
> On Friday, September 14, 2012 12:55:03 PM UTC-7, Mark wrote:

[snip]

> You should be able to do this in the realm of Cython (but still calling
> the CPython API).
>
> Try something like the following:
>
> from cython.cpython.pycapsule import PyCapsule_New, PyCapsule_GetPointer
> p = PyCapsule_New(<void*> c_func_that_returns_a_pointer(), "Foo*", NULL)
> mylist.append(p)
> # ...
> cdef Foo* q = <Foo*> PyCapsule_GetPointer(mylist.pop(), "Foo*")
>
> You need to be careful with memory allocation here. PyCapsule_New
> offers an argument for a destructor which you may need to use.

Good point. In my particular case make_foo() returns a "handle" pointer
and when I've finished I have to call a free_foo() function. I do this
by iterating over my dict of capsules in an atexit()-registered
function.

:-)

--
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy

"Advanced Qt Programming" - ISBN 0321635906
http://www.qtrac.eu/aqpbook.html

[Mark Summerfield]

On Fri, 14 Sep 2012 13:13:35 -0700 (PDT)
Bradley Froehle <brad.f...@gmail.com> wrote:

[snip]

> You should be able to do this in the realm of Cython (but still calling
> the CPython API).
>
> Try something like the following:
>
> from cython.cpython.pycapsule import PyCapsule_New, PyCapsule_GetPointer
> p = PyCapsule_New(<void*> c_func_that_returns_a_pointer(), "Foo*", NULL)
> mylist.append(p)
> # ...
> cdef Foo* q = <Foo*> PyCapsule_GetPointer(mylist.pop(), "Foo*")
>
> You need to be careful with memory allocation here. PyCapsule_New
> offers an argument for a destructor which you may need to use.

BTW I tried:

capsule = pycapsule.PyCapsule_New(<void*>p, NULL, _delete)

with

cdef void _delete(capsule):
cdef Foo *p
if pycapsule.PyCapsule_IsValid(capsule, NULL):
p = <Foo*>pycapsule.PyCapsule_GetPointer(capsule, NULL)
free_foo(p)
print("_delete")

But _delete() was never called. So I've stuck to using an
atexit()-registered function.

--
Mark Summerfield, Qtrac Ltd, www.qtrac.eu
C++, Python, Qt, PyQt - training and consultancy

[Golgauth]

I did it that way : (added cython exposable function to my C++ object)


string MyObj::ThisToString() {
stringstream ss;
ss << (void *)this;
cout << "addr" << ss.str() << endl;
return ss.str();
}