Cython Memoryviews of Arrays of Structs bugged?

803 views
Skip to first unread message

Leo

unread,
Sep 27, 2013, 12:38:05 PM9/27/13
to cython...@googlegroups.com
Hi, there seem to be some bugs regarding memoryviews of structured arrays in Cython when the struct itself contains arrays. I have a Cython fix for one of them (the most relevant to me), but first I would like to make sure that these are really bugs and I am not doing anything stupid. This question is related to http://stackoverflow.com/questions/17239091/cython-memoryviews-from-array-of-structs, but the answer there works only for char arrays. An array of doubles breaks it again.

It seems to me that Cython implementation solves some special cases, but not the general one. My proposed fix would be to insert
ndim = ctx->head->field->type->ndim;
after
/* Process the previous element */
if (__Pyx_BufFmt_ProcessTypeChunk(ctx) == -1) return NULL;
in Cython\Utility\Buffer.c

This fixes tests 2, 4 and 7, but not 8 and 9 (see below). I am not sure about the details of what the code in Buffer.c is supposed to do and whether my modification would break some other functionality, but my code (not only the tests) works with this fix.

Or am I possibly simply missing "the right way" to handle arrays of structs containing arrays in Cython? But in either case the compiler shouldn't crash like in test 9...



Here is the code of my tests with and without numpy with results ranging from "ok" to "compiler crash".
The numpy tests require these imports:
import numpy as np
cimport numpy as np

Test 1 (OK)
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[:] my_view = <My_Struct[:1]>&my_struct

Test 2 (FAILS)
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[1]
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[:] my_view = <My_Struct[:1]>&my_struct

ValueError: Expected 1 dimension(s), got 1

Test 3 (OK)
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[1]
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_first_value','float64'), ('my_second_value','float64',1)], align=False)
    cdef My_Struct[:] my_view = my_array

Test 4 (FAILS)
cdef packed struct My_Struct:
    double my_first_value
    double my_second_value[2]
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_first_value','float64'), ('my_second_value','float64',2)], align=False)
    cdef My_Struct[:] my_view = my_array

ValueError: Expected 1 dimension(s), got 1

Test 5 (OK)
cdef packed struct My_Struct:
    double my_first_value
    char my_second_value[2]
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_first_value','float64'), ('my_second_value','a',2)], align=False)
    cdef My_Struct[:] my_view = my_array

Test 6 (OK)
cdef packed struct My_Array_Struct:
    double values[1]
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_first_value','float64'), ('my_second_value','float64',1)], align=False)
    cdef My_Struct[:] my_view = my_array

Test 7 (FAILS)
cdef packed struct My_Array_Struct:
    double values[2]
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value
cpdef test():
    cdef np.ndarray my_array = np.ndarray(1, dtype=np.dtype([('my_first_value','float64'), ('my_second_value','float64',2)], align=False)
    cdef My_Struct[:] my_view = my_array

ValueError: Expected 1 dimension(s), got 1

Test 8 (FAILS)
cdef packed struct My_Struct:
    double my_first_value[1]
    double my_second_value
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[:] my_view = <My_Struct[:1]>&my_struct

ValueError: Buffer dtype mismatch, expected 'double' but got end in 'My_Struct.my_second_value'

Test 9 (COMPILER CRASH)
cdef packed struct My_Array_Struct:
    double values[2]
cdef packed struct My_Struct:
    double my_first_value
    My_Array_Struct my_second_value[1]
cpdef test():
    cdef My_Struct my_struct
    cdef My_Struct[:] my_view = <My_Struct[:1]>&my_struct

Gregorio Bastardo

unread,
Jan 26, 2015, 4:24:51 AM1/26/15
to cython...@googlegroups.com
Hello All,
 
This problem just hit me (with 0.20.2), does anyone know if there's any progress? Is it a numpy or a cython bug?
 
Thanks,
Gregorio

ratom...@gmail.com

unread,
Mar 23, 2015, 3:27:19 PM3/23/15
to cython...@googlegroups.com
+1

Stefan Behnel

unread,
Jun 29, 2015, 9:40:50 AM6/29/15
to cython...@googlegroups.com
I created a ticket for this so that it doesn't get lost:

http://trac.cython.org/ticket/853

Stefan

Sturla Molden

unread,
Jun 30, 2015, 9:18:57 AM6/30/15
to cython...@googlegroups.com
Just a short question: Does it work correctly with normal structs, i.e. not
using 'packed' and using aligned=True?

Sturla

Gio Piantoni

unread,
Jul 2, 2015, 9:57:44 AM7/2/15
to cython...@googlegroups.com
I was able to replicate this error on Cython 0.22.1 and 0.23.dev (Windows 7, Python 3.4, Numpy 1.9.1, MSVC 2010).

The error occurred with or without options "packed" and "aligned=True"

The bugfix solved it for me, thanks for providing it. Hopefully the fix will be included in the main repo soon.

zhe qu

unread,
Dec 16, 2015, 4:29:44 AM12/16/15
to cython-users
hi, I can not open http://trac.cython.org/ticket/853, is that bug fixed? thank you!


在 2015年7月2日星期四 UTC+8下午9:57:44,Gio Piantoni写道:

Chiu Speq

unread,
May 19, 2016, 1:47:16 AM5/19/16
to cython-users, stef...@behnel.de
I need this feature, too. 
Thanks for patch.

Chiu Speq

unread,
May 20, 2016, 1:55:28 PM5/20/16
to cython-users, stef...@behnel.de
Dear Leo/Stefan,

It seems an issue on Buffer.c:695.
It tries merge field with same type which causes the problems. 
(trying to fixed some bug #630 w/ numpy, maybe for char type only?).

anyway, after applied the patch, below code works fine for me.

cdef struct zz:
    int     a[10]
    int     b
    float   c
    double  d[1]
    double  e
    double  f[2]
    double  g
    char    h
    char    i[10]
    char    k

cpdef zz[:] gogo():
    cdef int N = 10
    cdef zz * m = <zz *>malloc(N * sizeof(zz))
    cdef zz[:] a = <zz[:N]>m

    a[0].a = [8] * 10
    a[0].b = 20
    a[0].c = 30.0

    return a





On Monday, June 29, 2015 at 9:40:50 PM UTC+8, Stefan Behnel wrote:
buffer.diff
Reply all
Reply to author
Forward
0 new messages