Cython arrays of cython classes ...

[Jerome Kieffer]

Dear Cython developers,

I looked in the documentation and did not find any example of memory
view on an array of cython objects.

Actually my problem is how to "declare" the format of such an object ib
the view.array constructor. Does a cdef class have a "get_format" ? how
can we build it ?

In the test of Cython, there are some examples which suggest this is
possible but I did not manage by myself to do it.

Here is a simple example of what I would like to do: a 1D array of
objects of type Vector2, but of course the code is not functional :(

"""
from cython cimport view
cdef class Vector2:
cdef:
float[:] coef
int[:] idx
int size, allocated

def __cinit__(self, int min_size=10):

self.allocated = min_size
self.coef = np.empty(self.allocated, dtype=np.float32)
self.idx = np.empty(self.allocated, dtype=np.int32)
self.size = 0

cdef class Array2:
cdef:
Vector2[:] lines
int size

def __cinit__(self, int nlines, min_size=10):
self.size = nlines
self.lines = view.array(shape=(nlines,), itemsize=sizeof(Vector2), format=Vector2)
for i in range(nlines):
self.lines[i] = Vector2(min_size=min_size)
"""

Thanks in advance for you help.

Cheers,

--
Jérôme Kieffer
tel +33 476 882 445

[Robert Bradshaw]

Unfortunately, memoryviews don't support arrays of specific subclasses
of object. I would recommend simply using list in this case, which are
actually quite efficient containers of refcoutned objects. (We don't
have a good solution for homogeneously lists, one difficulty being how
to restrict assignment from other contexts, so you'd have to cast on
access...)

Looking at your code, another option might be to create an actual 2D
NumPy array. (I don't have enough context to see if that would work
well though.)

[Jerome Kieffer]

Dear Robert,

Thanks for you answer.

On Wed, 31 Aug 2016 09:42:40 -0700
Robert Bradshaw <robe...@gmail.com> wrote:

> Unfortunately, memoryviews don't support arrays of specific subclasses
> of object. I would recommend simply using list in this case, which are
> actually quite efficient containers of refcoutned objects.

This was the original implementation and I confirm it works and the
performances are decent.
My hope was to be able to have a GIL free implementation and maybe to
have it running in parallel :(

> (We don't
> have a good solution for homogeneously lists, one difficulty being how
> to restrict assignment from other contexts, so you'd have to cast on
> access...)
>
> Looking at your code, another option might be to create an actual 2D
> NumPy array. (I don't have enough context to see if that would work
> well though.)

The context is the generation of sparse matrix, ultimately in CSR format.
the number of lines is known in advance (~1e3 to 1e5), the number of
element per line is unknown and varies a lot (from 0 to 1e6 elements).
The inner vector needs to be extended on demand like python lists.

[Robert Bradshaw]

On Thu, Sep 1, 2016 at 2:50 AM, Jerome Kieffer <goo...@terre-adelie.org> wrote:
>
> Dear Robert,
>
> Thanks for you answer.
>
>
> On Wed, 31 Aug 2016 09:42:40 -0700
> Robert Bradshaw <robe...@gmail.com> wrote:
>
>> Unfortunately, memoryviews don't support arrays of specific subclasses
>> of object. I would recommend simply using list in this case, which are
>> actually quite efficient containers of refcoutned objects.
>
> This was the original implementation and I confirm it works and the
> performances are decent.
> My hope was to be able to have a GIL free implementation and maybe to
> have it running in parallel :(

Getting a reference to an object requires the GIL, no matter the
container. That being said, you could use PyList_GET_ITEM which
shouldn't require the GIL, i.e.

(<YourClass>PyList_GET_ITEM (L, ix)).member

Note that in

cdef list L = ...
(<YourClass>L[ix]).member

the indexing itself doesn't require the GIL, but it inserts a check
for None (with possible error) that does. Assignment to an (object)
intermediate would require the GIL as well; above you'd need to do

PyObject* item = PyList_GET_ITEM(L, ix)
(<YourClass>item).member1.something
(<YourClass>item).member2()
...

to avoid the GIL. Assuming, of course, the list is immutable for safety.

>> (We don't
>> have a good solution for homogeneously lists, one difficulty being how
>> to restrict assignment from other contexts, so you'd have to cast on
>> access...)
>>
>> Looking at your code, another option might be to create an actual 2D
>> NumPy array. (I don't have enough context to see if that would work
>> well though.)
>
> The context is the generation of sparse matrix, ultimately in CSR format.
> the number of lines is known in advance (~1e3 to 1e5), the number of
> element per line is unknown and varies a lot (from 0 to 1e6 elements).
> The inner vector needs to be extended on demand like python lists.

Yeah, a 2D array wouldn't work here.

Another option, though less palatable, is to make your vector a struct
and play with pointers directly...