How to declare an array of extension type cython objects

[VictorNorman]

I have a Cython extension type ResProperties:

cdef class ResProperties:

    cdef public float endDayUtilities

    cdef public list marginalUtilities

    cdef public int held

    ... etc, etc. ...

and I have a Cython extension type Agent:

cdef class Agent(object):

    cdef public ResProperties[c.NUM_RESOURCES] resProp

In my current implementation the resProp is a python list of ResProperties objects, always of length c.NUM_RESOURCES.  I'd like to make that more efficient by declaring it to be an array of ResProperties objects.

Is there a way to do this?  Is there a web page / tutorial where I should have found this info myself?  

Thanks. 

[Robert Bradshaw]

No, there's currently no way to do this. Note, however, that getting
from/setting to a Python list is already rather efficient (it indexes
into the underlying PyObject*).

- Robert

[shaunc]

Hmm...

Couldn't one do something like this?

from cpython cimport PyObject, Py_INCREF, Py_DECREF

ctypedef PyObject *PyObject_p

cdef class Agent( object ):

    cdef public PyObject_p resProp[ c.NUM_RESOURCES ]

    def __cinit__( self ):

        cdef int i

        for i in range( c.NUM_RESOURCES ):

            tmp = ResProperties( ... )

            Py_INCREF( tmp )

            self.resProp[ i ] = <PyObject *>tmp

    def __getitem__( self, int item ):

        return <ResProperties>self.resProp[ item ]

    def __dealloc__( self ):

        cdef int i

        for i in range( c.NUM_RESOURCES ):

            Py_DECREF( <object>self.resProp[ i ] )

[Stefan Behnel]

shaunc, 12.06.2012 21:30:

> Couldn't one do something like this?
>
> from cpython cimport PyObject, Py_INCREF, Py_DECREF
>
> ctypedef PyObject *PyObject_p
>
> cdef class Agent( object ):
> cdef public PyObject_p resProp[ c.NUM_RESOURCES ]

"public" doesn't work here because the type is not compatible with a Python
object.

> def __cinit__( self ):
> cdef int i
> for i in range( c.NUM_RESOURCES ):
> tmp = ResProperties( ... )
> Py_INCREF( tmp )
> self.resProp[ i ] = <PyObject *>tmp
>
> def __getitem__( self, int item ):
> return <ResProperties>self.resProp[ item ]
>
> def __dealloc__( self ):
> cdef int i
> for i in range( c.NUM_RESOURCES ):
> Py_DECREF( <object>self.resProp[ i ] )

Sure, one could do that. But what makes you think that this has an
advantage over using a Python list? It's certainly a lot less obvious, less
readable, much harder to use and more error prone.

Stefan

[shaunc]

umm... slightly faster because of lack of bounds checking code? (pace "much more error prone" :)). Just in case that was important to Victor in view of other disadvantages.

[shaunc]

umm... slightly faster because of lack of bounds checking code? (pace "much more error prone" :)). Just in case that was important to Victor in view of other disadvantages.

also, could be used in "nogil" blocks -- essential, e.g. if using prange 

[Stefan Behnel]

shaunc, 13.06.2012 12:58:

> umm... slightly faster because of lack of bounds checking code? (pace "much
> more error prone" :)).

You can remove that performance hit more explicitly and clearly with the
"boundscheck" option.

Stefan

[Stefan Behnel]

shaunc, 13.06.2012 13:07:

No, because you are still dealing with Python references at the moment
where you make any use if them.

Stefan

[shaunc]

On Wednesday, June 13, 2012 6:36:09 PM UTC+2, Stefan Behnel wrote:

> 
> umm... slightly faster because of lack of bounds checking code? (pace "much 
> more error prone" :)). 

You can remove that performance hit more explicitly and clearly with the 
"boundscheck" option. 

Ah -- didn't know that helped with reference into list as well (I thought it was ndarray specific) -- good to know.

Since the "ResProperties" are themselves extension types, you can get data out of them without the gil with a bit of hacking. Not that this is pretty .... But this is exactly why I did this once. You may be right though, that if you need nogil access it might be better to keep the data in c structures -- implementing "ResProperties" using "flyweight" pattern w/ pointer into these structures (which would be stored in Agent).

[Robert Bradshaw]

On Thu, Jun 28, 2012 at 12:31 AM, Niko Skrypnik <nskr...@gmail.com> wrote:
>
>
> суббота, 26 мая 2012 г., 1:56:41 UTC+3 пользователь Robert Bradshaw написал:

> As I understand it's also impossible now to do something like
>
> cdef class Agent(object):
>     cdef ResProperties* resProp
>
> i.e. to make pointer of extension type. Or I'm wrong? For me it was clear
> that it possible cause as I got it from docs and generated C files,
> extension type becomes a C struct after translation so I thought that I can
> do this way. But in fact I've get error from cython translator - "Pointer
> base type cannot be a Python object". Hm, why is this Python object rather
> than C struct?

Extension types compile to C structs, but all extension type variables
are already pointers (to be compatible with PyObject* and have the
reference semantics one is used to from Python). We don't however let
you take a pointer of that (though it's tempting to borrow the pointer
and address-of syntax for borrowed references). Often it's better to
think about how you would write it in Python instead of thinking about
how you would write it in C.

- Robert

[Stefan Behnel]

Niko Skrypnik, 28.06.2012 09:31:

> суббота, 26 мая 2012 г., 1:56:41 UTC+3 пользователь Robert Bradshaw написал:

>> On Fri, May 25, 2012 at 3:52 PM, VictorNorman wrote:
>>> I have a Cython extension type ResProperties:
>>>
>>> cdef class ResProperties:
>>> cdef public float endDayUtilities
>>> cdef public list marginalUtilities
>>> cdef public int held
>>> ... etc, etc. ...
>>>
>>> and I have a Cython extension type Agent:
>>>
>>> cdef class Agent(object):
>>> cdef public ResProperties[c.NUM_RESOURCES] resProp
>>>
>>> In my current implementation the resProp is a python list of
>>> ResProperties
>>> objects, always of length c.NUM_RESOURCES. I'd like to make that more
>>> efficient by declaring it to be an array of ResProperties objects.

I should reiterate that this is not necessarily making it more efficient.
It depends on what the code actually does.

>>> Is there a way to do this? Is there a web page / tutorial where I
>>> should have found this info myself?
>>
>> No, there's currently no way to do this. Note, however, that getting
>> from/setting to a Python list is already rather efficient (it indexes
>> into the underlying PyObject*).
>>

>> As I understand it's also impossible now to do something like
>
> cdef class Agent(object):
> cdef ResProperties* resProp
>
> i.e. to make pointer of extension type.

Correct, although you can take a generic <void*> or <PyObject*> from a
Python object. It just doesn't have attributes or methods.

> Or I'm wrong? For me it was clear
> that it possible cause as I got it from docs and generated C files,
> extension type becomes a C struct after translation so I thought that I can
> do this way. But in fact I've get error from cython translator - "Pointer
> base type cannot be a Python object". Hm, why is this Python object rather
> than C struct?

It's not that simple. Object references are reference counted. Pointers are
not. Language support for borrowed (i.e. non reference counted) references
is still not implemented due to lack of interest and/or sponsorship.

However, borrowed references are a rather tricky thing to use anyway. For
one, they barely work in PyPy, but also in CPython, you have to make sure
there actually is an owned reference to them somewhere as long as you make
use of them. That leads to duplication that can make code hard to
understand at times and introduce subtle bugs that only show coincidentally
way after writing the code.

Stefan