How to declare and use PyArray_NewFromDescr

[Daily, Jeff A]

I need to create an ndarray with specific strides. From this link
http://blog.enthought.com/python/numpy-arrays-with-pre-allocated-memory/
it sounds like I should use PyArray_NewFromDescr. However, when I

import numpy as np
cimport numpy as np
...
array = np.PyArray_NewFromDescr(...)

the function isn't available, or rather I get errors such as "cannot convert np.npy_intp* to Python object". I looked at the numpy.pxd file in Cython 0.15.1 and sure enough, the function isn't declared. So I tried declaring the function myself. I have the following code at the top of my *.pyx file.

import numpy as np
cimport numpy as np
# first attempt
from cpython.ref cimport PyTypeObject
import __builtin__

np.import_array()

# second attempt
#cdef extern from "Python.h":
# ctypedef struct PyTypeObject:
# pass

# third attempt
#cdef extern from "Python.h":
# struct PyTypeObject:
# pass

cdef extern from "numpy/arrayobject.h":
PyTypeObject PyArray_Type
object PyArray_NewFromDescr(PyArray_Type * subtype,
np.dtype descr,
int nd,
np.npy_intp * dims,
np.npy_intp * strides,
void * data,
int flags,
object obj)

I've tried three different ways of declaring PyTypeObject. Nothing works. "Variable type 'PyTypeObject' is incomplete," says Cython. Or "'PyArray_Type' is not a type identifier".

But if I do the following, it magically works. At least, it seems like magic to me.

cdef extern from "numpy/arrayobject.h":
struct WhatTheHell:
pass
WhatTheHell PyArray_Type
object PyArray_NewFromDescr(WhatTheHell * subtype,
np.dtype descr,
int nd,
np.npy_intp * dims,
np.npy_intp * strides,
void * data,
int flags,
object obj)
array = PyArray_NewFromDescr(&PyArray_Type,
np.PyArray_DescrFromType(typenum), nd, dims, strides, ptr,
np.NPY_DEFAULT, None)

What was I doing wrong? Why did the last version of my code work?

_______________________________________________
Jeff Daily
Scientist
DATA INTENSIVE SCIENTIFIC COMPUTING GROUP

Pacific Northwest National Laboratory
902 Battelle Boulevard
P.O. Box 999, MSIN K7-90
Richland, WA 99352 USA
Tel: 509-372-6548
Fax: 509-372-4720
jeff....@pnnl.gov
www.pnnl.gov

[mark florisson]

Your predicament is understandable. Firstly, it is important to
understand that PyArray_Type is not a C type, it is an instance of a
PyTypeObject. As the NumPy documentation suggests, the signature wants
a PyTypeObject *, of which PyArray_Type (numpy.ndarray) is a instance.
If you change the signature accordingly, it should work.

Note however that your life gets much easier by just typing everything
as object instead. Here goes:

cimport numpy as np

cdef extern from "numpy/arrayobject.h":
object PyArray_NewFromDescr(object subtype, np.dtype descr,

int nd, np.npy_intp* dims, np.npy_intp* strides,

void* data, int flags, object obj)

import numpy
np.import_array()

cdef np.npy_intp dims[2]
cdef np.npy_intp strides[2]
cdef float data[10][10]

dims[0] = 10
dims[1] = 10
strides[0] = 10 * 4
strides[1] = 4
for i in range(10):
for j in range(10):
data[i][j] = i * 10 + j

print PyArray_NewFromDescr(numpy.ndarray, numpy.dtype('f'), 2, dims, strides,
<void *> data, 0, None)

# output:
[[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
[ 10. 11. 12. 13. 14. 15. 16. 17. 18. 19.]
[ 20. 21. 22. 23. 24. 25. 26. 27. 28. 29.]
[ 30. 31. 32. 33. 34. 35. 36. 37. 38. 39.]
[ 40. 41. 42. 43. 44. 45. 46. 47. 48. 49.]
[ 50. 51. 52. 53. 54. 55. 56. 57. 58. 59.]
[ 60. 61. 62. 63. 64. 65. 66. 67. 68. 69.]
[ 70. 71. 72. 73. 74. 75. 76. 77. 78. 79.]
[ 80. 81. 82. 83. 84. 85. 86. 87. 88. 89.]
[ 90. 91. 92. 93. 94. 95. 96. 97. 98. 99.]]

> But if I do the following, it magically works.  At least, it seems like magic to me.
>
> cdef extern from "numpy/arrayobject.h":
>    struct WhatTheHell:
>        pass
>    WhatTheHell PyArray_Type
>    object PyArray_NewFromDescr(WhatTheHell * subtype,
>                                np.dtype descr,
>                                int nd,
>                                np.npy_intp * dims,
>                                np.npy_intp * strides,
>                                void * data,
>                                int flags,
>                                object obj)
> array = PyArray_NewFromDescr(&PyArray_Type,
>        np.PyArray_DescrFromType(typenum), nd, dims, strides, ptr,
>        np.NPY_DEFAULT, None)
>
> What was I doing wrong?  Why did the last version of my code work?

Your last version works because Cython sees the right type, and you
take the address of it, so it expects the type to be there at the C
level but it isn't. This would break as soon as you try to cast it, as
it would insert a C cast into the code.

> _______________________________________________
> Jeff Daily
> Scientist
> DATA INTENSIVE SCIENTIFIC COMPUTING GROUP
>
> Pacific Northwest National Laboratory
> 902 Battelle Boulevard
> P.O. Box 999, MSIN K7-90
> Richland, WA 99352 USA
> Tel: 509-372-6548
> Fax: 509-372-4720
> jeff....@pnnl.gov
> www.pnnl.gov

Note that there are two other ways to get what you want, besides using
this numpy function:

a) write a cdef class with __getbuffer__ and __releasebuffer__ methods
(see https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#L105
for an example)

b) checkout cython from github and use memoryviews:
numpy.asarray(<float[:, :]> data) (where data is a 2D C array)

If data were a pointer you'd have to insert shape information like
<float[:10, :10]> data. You can then slice the result appropriately
(before or after passing it into NumPy). Note that for both these
methods you need a NumPy version that supports PEP 3118.

Good luck!

Mark

[Brian Rossa]

Mark,

Thanks for this and the related discussion on the other thread.
Unfortunately, this code fails to compile when the C++ extensions are
enabled:

$cython test.pyx --embed --cplus && gcc test.cpp -I/usr/include/
python2.7/ -lpython2.7 -o test && ./test
test.cpp: In function ‘void inittest()’:
test.cpp:2946:146: error: cannot convert ‘PyObject*’ to
‘PyTypeObject*’ in argument passing

It's just fine without the C++ extensions, though some warnings are a
harbinger of trouble:

$ cython test.pyx --embed --cplus && gcc test.c -I/usr/include/
python2.7/ -lpython2.7 -o test && ./test
test.c: In function ‘inittest’:
test.c:2951:3: warning: passing argument 1 of ‘(struct PyObject * (*)
(struct PyTypeObject *, struct PyArray_Descr *, int, npy_intp *,
npy_intp *, void *, int, struct PyObject *))*(PyArray_API + 752u)’
from incompatible pointer type
test.c:2951:3: note: expected ‘struct PyTypeObject *’ but argument is
of type ‘struct PyObject *’

[[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
[ 10. 11. 12. 13. 14. 15. 16. 17. 18. 19.]
[ 20. 21. 22. 23. 24. 25. 26. 27. 28. 29.]
[ 30. 31. 32. 33. 34. 35. 36. 37. 38. 39.]
[ 40. 41. 42. 43. 44. 45. 46. 47. 48. 49.]
[ 50. 51. 52. 53. 54. 55. 56. 57. 58. 59.]
[ 60. 61. 62. 63. 64. 65. 66. 67. 68. 69.]
[ 70. 71. 72. 73. 74. 75. 76. 77. 78. 79.]
[ 80. 81. 82. 83. 84. 85. 86. 87. 88. 89.]
[ 90. 91. 92. 93. 94. 95. 96. 97. 98. 99.]]

Any direction you can offer would be greatly appreciated.

Cheers!
~br

On Dec 3, 8:13 am, mark florisson <markflorisso...@gmail.com> wrote:

> > jeff.da...@pnnl.gov

> >www.pnnl.gov
>
> Note that there are two other ways to get what you want, besides using
> this numpy function:
>
> a) write a cdef class with __getbuffer__ and __releasebuffer__ methods

> (seehttps://github.com/cython/cython/blob/master/Cython/Utility/MemoryVie...

[Brian Rossa]

Sorry, bad copy-n-paste job on that last email. That second command
line snippet should read:

$ cython test.pyx --embed && gcc test.c -I/usr/include/

python2.7/ -lpython2.7 -o test && ./test
test.c: In function ‘inittest’:
test.c:2951:3: warning: passing argument 1 of ‘(struct PyObject * (*)
(struct PyTypeObject *, struct PyArray_Descr *, int, npy_intp *,
npy_intp *, void *, int, struct PyObject *))*(PyArray_API + 752u)’
from incompatible pointer type
test.c:2951:3: note: expected ‘struct PyTypeObject *’ but argument is
of type ‘struct PyObject *’
[[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
[ 10. 11. 12. 13. 14. 15. 16. 17. 18. 19.]
[ 20. 21. 22. 23. 24. 25. 26. 27. 28. 29.]
[ 30. 31. 32. 33. 34. 35. 36. 37. 38. 39.]
[ 40. 41. 42. 43. 44. 45. 46. 47. 48. 49.]
[ 50. 51. 52. 53. 54. 55. 56. 57. 58. 59.]
[ 60. 61. 62. 63. 64. 65. 66. 67. 68. 69.]
[ 70. 71. 72. 73. 74. 75. 76. 77. 78. 79.]
[ 80. 81. 82. 83. 84. 85. 86. 87. 88. 89.]
[ 90. 91. 92. 93. 94. 95. 96. 97. 98. 99.]]

~br

[mark florisson]

On 3 December 2011 21:28, Brian Rossa <b...@co57.com> wrote:
> Mark,
>
> Thanks for this and the related discussion on the other thread.
> Unfortunately, this code fails to compile when the C++ extensions are
> enabled:
>
> $cython test.pyx --embed --cplus && gcc test.cpp -I/usr/include/
> python2.7/ -lpython2.7 -o test && ./test
> test.cpp: In function ‘void inittest()’:
> test.cpp:2946:146: error: cannot convert ‘PyObject*’ to
> ‘PyTypeObject*’ in argument passing
>
> It's just fine without the C++ extensions, though some warnings are a
> harbinger of trouble:

Ah, you're right. A PyTypeObject is a PyObject, but in Python 2 isn't
not a struct in a struct, but the members themselves are in there from
the macro and C++ is picky about these things. You can solve it by
typing your things as a PyTypeObject:

from cpython.ref cimport PyTypeObject

cdef extern ...:
object PyArray_NewFromDescr(PyTypeObject *subtype, ...)

PyArray_NewFromDescr(<PyTypeObject *> numpy.ndarray, ...)

[Daily, Jeff A]

Thanks for that. Your code got me 99% of the way there, but then I started
getting ref count errors about trying to dereference the 'f' dtype. After
some googling I found the problem was as described here:
http://wiki.cython.org/tutorials/numpy

So I added
from python_ref cimport Py_INCREF
...
cdef np.dtype dtype = np.dtype('f')
Py_INCREF(dtype)
array = PyArray_NewFromDescr(numpy.ndarray,
numpy.dtype('f'), 2, dims, strides, ptr, np.NPY_DEFAULT, None)

Oh, and I'm using np.NPY_DEFAULT instead of 0 since the data getting wrapped
by the ndarray should be modifiable. Hmm, the data is in C order but not
contiguous since I'm modifying the strides. Is using NPY_DEFAULT correct?

*snip*

> Note that there are two other ways to get what you want, besides using
> this numpy function:
>
> a) write a cdef class with __getbuffer__ and __releasebuffer__ methods
> (see
>
https://github.com/cython/cython/blob/master/Cython/Utility/MemoryView.pyx#L10>
5
> for an example)
>
> b) checkout cython from github and use memoryviews:
> numpy.asarray(<float[:, :]> data) (where data is a 2D C array)
>
> If data were a pointer you'd have to insert shape information like
> <float[:10, :10]> data. You can then slice the result appropriately
> (before or after passing it into NumPy). Note that for both these
> methods you need a NumPy version that supports PEP 3118.
>
> Good luck!
>
> Mark

I wasn't aware of memoryviews. I'll take a look, but I'd like to be able to
support users who don't use bleeding-edge versions of cython and numpy.