creating a memoryview from scratch

[Chris Barker - NOAA Federal]

Can I create a memoryvie from scratch (i.e have it allocate it's own memory)

What I really want to do:

I'm working on some wrappers that I'd like to work well with numpy,
but ideally wouldn't require numpy to use or build.

for method that take an ndarray-like object, it works great to use a memoryview

but I need to return something that can be turned inot an ndarray --

ie.

arr = np.asarray(my_object)

so I can specify an __array__ method, but what to I return? I'm
thinking a memoryview object.

but how do I allocate the memory block for that object?

If I use regular old malloc, and copy to the memoryview, then how/when
do I free that memory?

any ideas?

(note -- it turns out that that lib I"m wrapping does not have a
compatible block of memory already, so I need to create a new block
for this..)

-Chris











--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Gregor Thalhammer]

Am 20.6.2013 um 01:44 schrieb Chris Barker - NOAA Federal:

Can I create a memoryvie from scratch (i.e have it allocate it's own memory)

What I really want to do:

I'm working on some wrappers that I'd like to work well with numpy,
but ideally wouldn't require numpy to use or build.

for method that take an ndarray-like object, it works great to use a memoryview

but I need to return something that can be turned inot an ndarray --

ie.

arr = np.asarray(my_object)

so I can specify an __array__ method, but what to I return? I'm
thinking a memoryview object.

but how do I allocate the memory block for that object?

You can use a cython array, e.g.

from cython cimport view

my_array = view.array(shape=(10, 2), itemsize=sizeof(int), format="i")
cdef int[:, :] my_slice = my_array

(see http://docs.cython.org/src/userguide/memoryviews.html#cython-arrays) and return my_array or np.asarray(my_array). The format argument to view.array is the same as for the standard python array.

If I use regular old malloc, and copy to the memoryview, then how/when
do I free that memory?

Memory will be released, when the view.array is destroyed. A numpy array created with asarray will share memory and keeps a reference, so memory keeps allocated as long the numpy array exists.

Gregor

[Sturla Molden]

On 20.06.2013 09:57, Gregor Thalhammer wrote:

>> so I can specify an __array__ method, but what to I return? I'm
>> thinking a memoryview object.
>>
>> but how do I allocate the memory block for that object?
>
>
> You can use a cython array, e.g.
>
> from cython cimport view
>
> my_array = view.array(shape=(10, 2), itemsize=sizeof(int), format="i")
> cdef int[:, :] my_slice = my_array

We can also use a pointer, e.g. from malloc:

cdef int[:,:] memview = <int[:m,:n]> pointer

Just remember to declare the shape in the cast.

Also, to access char** data in libgd images (if that is still needed),
it would be something like this:

from cython cimport view
cdef int sx, sy
cdef char **image = image_from_libgd
cdef char[::view.indirect_contiguous, ::1] image_view = \
<char[:sy:view.indirect_contiguous, :sx:1]> image

Unlike NumPy, typed memoryviews can actually access these indirect 2d
arrays :)

Then a copy to NumPy will be something like this:

numpy_image = np.empty((sy,sx),dtype=np.uint8)
numpy_image[...] = image_view



Sturla

[Chris Barker - NOAA Federal]

On Thu, Jun 20, 2013 at 12:57 AM, Gregor Thalhammer
<gregor.t...@gmail.com> wrote:
> You can use a cython array, e.g.
>
> from cython cimport view
>
> my_array = view.array(shape=(10, 2), itemsize=sizeof(int), format="i")
> cdef int[:, :] my_slice = my_array

OK -- simple enough

> and return my_array or np.asarray(my_array).

> Memory will be released, when the view.array is destroyed.

if this is in a method, then the view.array will be destroyed when
that method returns, yes?

> A numpy array
> created with asarray will share memory and keeps a reference, so memory
> keeps allocated as long the numpy array exists.

so the view.array works with the numpy machinery to know that it
shouldn't be destroyed? For that to work:

np.asarray(my_array)

would have to increment the reference count of my_array, yes? And does it?

(OK -- I'll test it!)

[Chris Barker - NOAA Federal]

On Thu, Jun 20, 2013 at 2:53 AM, Sturla Molden <stu...@molden.no> wrote:

>>> so I can specify an __array__ method, but what to I return? I'm
>>> thinking a memoryview object.
>>>
>>> but how do I allocate the memory block for that object?

>> You can use a cython array, e.g.

>> my_array = view.array(shape=(10, 2), itemsize=sizeof(int),
>> format="i")

> We can also use a pointer, e.g. from malloc:
>
> cdef int[:,:] memview = <int[:m,:n]> pointer
>
> Just remember to declare the shape in the cast.

but then I don't get any nifty python memory management for that
pointer -- which is my concern -- I either free it when there may be a
numpy array still trying to use it, or I don't delete and get a memory
leak..(though the wrapper class you suggested earlier may be the
solution to that)

> Also, to access char** data in libgd images (if that is still needed),

indeed, that's what I'm working on

> would be something like this:
>
> from cython cimport view
> cdef int sx, sy
> cdef char **image = image_from_libgd
> cdef char[::view.indirect_contiguous, ::1] image_view = \
> <char[:sy:view.indirect_contiguous, :sx:1]> image
>
> Unlike NumPy, typed memoryviews can actually access these indirect 2d arrays
> :)

cool! nice to not have to write that memcpy loop...

> Then a copy to NumPy will be something like this:
>
> numpy_image = np.empty((sy,sx),dtype=np.uint8)
> numpy_image[...] = image_view

couldn't I do a simple:

numpy_image[...] = np.asarray(image_view)

here?

[Chris Barker - NOAA Federal]

On Thu, Jun 20, 2013 at 2:53 AM, Sturla Molden <stu...@molden.no> wrote:

> Also, to access char** data in libgd images (if that is still needed), it
> would be something like this:
>
> from cython cimport view
> cdef int sx, sy
> cdef char **image = image_from_libgd
> cdef char[::view.indirect_contiguous, ::1] image_view = \
> <char[:sy:view.indirect_contiguous, :sx:1]> image

trying this, but I'm getting confused. This is what I now have:

def __array__(self):
"""
Returns a numpy array object with a copy of the data

Note that the array is (height, width) in size, in
keeping with image storage standards (e.g. PIL)
"""
# cdef cnp.ndarray[char, ndim=2, mode='c'] arr
# arr = np.zeros((self.height, self.width), dtype=np.uint8)
# cdef unsigned int row

# ##copy the data, row by row
# for row in range(self.height):
# memcpy( &arr[row, 0], self._image.pixels[row], self.width)
# return arr

# create a cython array for the pixels
pixel_array = view.array(shape=(height, width),
itemsize=sizeof(unsigned char), format="b")

# create a memoryview object to wrap the same memory
cdef unsigned char[:, :] image_view = pixel_array

##copy the data from the image pixel array:
cdef int sx = self.width
cdef int sy = self.height

cdef unsigned char **image = self._image.pixels

<unsigned char[::view.indirect_contiguous, ::1]> image_view = \
<unsigned char[:sy:view.indirect_contiguous, :sx:1]> image

return pixel_array

But Cython is giving me:
Error compiling Cython file:
------------------------------------------------------------
...
cdef int sx = self.width
cdef int sy = self.height

cdef unsigned char **image = self._image.pixels

<unsigned char[::view.indirect_contiguous, ::1]> image_view = \
^
------------------------------------------------------------

py_gd/py_gd.pyx:126:8: Cannot assign to or delete this

Error compiling Cython file:
------------------------------------------------------------
...
cdef int sy = self.height

cdef unsigned char **image = self._image.pixels

<unsigned char[::view.indirect_contiguous, ::1]> image_view = \
<unsigned char[:sy:view.indirect_contiguous, :sx:1]> image
^
------------------------------------------------------------

py_gd/py_gd.pyx:127:70: Pointer base type does not match cython.array base type

Error compiling Cython file:
------------------------------------------------------------
...
cdef int sx = self.width
cdef int sy = self.height

cdef unsigned char **image = self._image.pixels

<unsigned char[::view.indirect_contiguous, ::1]> image_view = \
^
------------------------------------------------------------

py_gd/py_gd.pyx:126:68: Can only create cython.array from pointer or array


so clearly I'm confused. Sturla suggested:

cdef char[::view.indirect_contiguous, ::1] image_view = \
<char[:sy:view.indirect_contiguous, :sx:1]> image

but if I read that right, it's creating a memoryview called
image_view, then assigning it to the image buffer.

But I need a copy -- to a contiguous buffer, which I was trying to
create with the view.array constructor. with the right typecasting,
can I do that as a single assignment?

I'm also having trouble figuring out how to copy the data by loping
through the rows:

cdef int row
##loop through the rows to copy data from pixel buffer to new array
for row in range(sy):
image_view[row, :] = image[row]


Error compiling Cython file:
------------------------------------------------------------
...


cdef int row
##loop through the rows to copy data from pixel buffer to new array
for row in range(sy):
image_view[row, :] = image[row]
^
------------------------------------------------------------

py_gd/py_gd.pyx:129:38: Cannot convert 'unsigned char *' to memoryviewslice


Any hints appreciated...

[Dag Sverre Seljebotn]

On 06/20/2013 01:44 AM, Chris Barker - NOAA Federal wrote:
> Can I create a memoryvie from scratch (i.e have it allocate it's own memory)
>
> What I really want to do:
>
> I'm working on some wrappers that I'd like to work well with numpy,
> but ideally wouldn't require numpy to use or build.
>
> for method that take an ndarray-like object, it works great to use a memoryview
>
> but I need to return something that can be turned inot an ndarray --
>
> ie.
>
> arr = np.asarray(my_object)
>
> so I can specify an __array__ method, but what to I return? I'm
> thinking a memoryview object.
>
> but how do I allocate the memory block for that object?
>
> If I use regular old malloc, and copy to the memoryview, then how/when
> do I free that memory?
>
> any ideas?

My suggestion would be to not touch either __array__ or memoryviews, but
either one of:

a) Implement the Cython special __getbuffer__ method on a cdef class
which has a pointer

b) Allocate the memory as a NumPy array in the first place (you can
almost always do this if you need to copy anyway?)

Dag Sverre

[Sturla Molden]

Den 20. juni 2013 kl. 19:41 skrev Dag Sverre Seljebotn <d.s.se...@astro.uio.no>:

> b) Allocate the memory as a NumPy array in the first place (you can almost always do this if you need to copy anyway?)
>

Except libgd seems to call malloc and free in a number of places, replacing lines in the image in different places. Images are allocated line-wise, with one calloc per line, not as one contiuous buffer. I briefly looked at the code, and I don't think it is doable to just change the image constructor and destructor.

Sturla

[mark florisson]

On 20 June 2013 17:09, Chris Barker - NOAA Federal

<chris....@noaa.gov> wrote:
> On Thu, Jun 20, 2013 at 2:53 AM, Sturla Molden <stu...@molden.no> wrote:
>
>>>> so I can specify an __array__ method, but what to I return? I'm
>>>> thinking a memoryview object.
>>>>
>>>> but how do I allocate the memory block for that object?
>
>>> You can use a cython array, e.g.
>
>>> my_array = view.array(shape=(10, 2), itemsize=sizeof(int),
>>> format="i")
>
>> We can also use a pointer, e.g. from malloc:
>>
>> cdef int[:,:] memview = <int[:m,:n]> pointer
>>
>> Just remember to declare the shape in the cast.
>
> but then I don't get any nifty python memory management for that
> pointer -- which is my concern -- I either free it when there may be a
> numpy array still trying to use it, or I don't delete and get a memory
> leak..(though the wrapper class you suggested earlier may be the
> solution to that)

cython.view.array handles that through 'callback_free_data', here's an example:

from cython.view cimport array as cyarray
from libc.stdlib cimport malloc, free

import numpy as np

cdef double *p = <double *> malloc(10 * sizeof(double))

for i in range(10):
p[i] = i

cdef void myfree(void *data):
print "deallocating array!"
free(data)

cdef cyarray cy_array = <double[:10]> p
cy_array.callback_free_data = myfree
print np.asarray(cy_array)
del cy_array
print "bye!"


This code prints:

[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]
deallocating array!
bye!

The cython arary can be turned into a numpy array (like above) or a
memoryview (by assignment).

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[mark florisson]

(BTW, I'm not necessarily advocating memoryviews for what you're
doing, just responding to the deallocation issue).

[Dag Sverre Seljebotn]

OK, so copy the image in __getbuffer__?

Dag Sverre

[Sturla Molden]

On 21.06.2013 07:09, Dag Sverre Seljebotn wrote:

> OK, so copy the image in __getbuffer__?

But then the NumPy array would not be a view of the image, so he could
just as well implement __setitem__ and __getitem__ (including slice
support for speed).

Sturla

[Chris Barker - NOAA Federal]

On Fri, Jun 21, 2013 at 7:01 AM, Sturla Molden <stu...@molden.no> wrote:
>> OK, so copy the image in __getbuffer__?

I think that's what I'll need to do. I'll give it a shot.

> But then the NumPy array would not be a view of the image, so he could just
> as well implement __setitem__ and __getitem__ (including slice support for
> speed).

would that be a "just as well" or an "in addition"? -- I can see the
benefit of a nice fast __setitem__ and __getitem__ -- and do want to
do that (but will I get around to it ?), but the need-at-hand is for
the user to be able to get a proper numpy array, with all its glory,
that mirrors the data in the image. I'd rather that be a view, but as
Sturla points our, that's not possible the way gd is currently
designed (apparently they intend to change this is the next minor
version, but no idea of time scale)

So for now, it will need to be a copy. I currently have it implemented with:
def __array__(self):
cdef cnp.ndarray[cnp.uint8_t, ndim=2, mode='c'] arr

arr = np.zeros((self.height, self.width), dtype=np.uint8)

cdef unsigned int row

##copy the data, row by row

for row in range(self.height):

memcpy( &arr[row, 0], self._image.pixels[row], self.width)

return arr

Which seems to work just fine.

However, another goal here (which I may give up on) is to write the
wrapper with no numpy dependencies, but with the ability to work
efficiently with numpy -- which is what PEP 3118 buffers are all
about, yes?

So I think DAG's suggestion about __getbuffer__ is the way to go. i"M
trying that now....

-Chris













> Sturla

>
>
> --
>
> --- You received this message because you are subscribed to the Google
> Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Chris Barker - NOAA Federal]

On Fri, Jun 21, 2013 at 11:26 AM, Chris Barker - NOAA Federal
<chris....@noaa.gov> wrote:
> So I think DAG's suggestion about __getbuffer__ is the way to go. I'm
> trying that now....

All I could find for examples is tests/run/buffer.pyx, referred to in:

http://wiki.cython.org/enhancements/buffer

However, that's a pretty trivial example, so I ran into some issues.
I've created a totally self-contained example, so that you can test it
if you're so inclined, and if I get this working right, I can put this
more complete example in the wiki or somewhere.

See the enclosed code for the example, but my questions are:

In the buffer struct, you need pointers to small things like .shape
and .strides -- where should those get allocated? I've now got it
allocated in the __getbuffer__, and freed in __releasebuffer__, but I
could also keep those around in the cdef class....

I'm getting errors when things like memoryview.to_bytes() is called:

Python(85194) malloc: *** error for object 0x3c4ef0: pointer being
freed was not allocated
*** set a breakpoint in malloc_error_break to debug

I tried wrapping the free() call with a check for NULL, but that
didn't work -- and is this a sign that I did something wrong?

if buffer.buf is not NULL:
free(buffer.buf)

Also, it looks like the main object is not deleted until after the
__releasebuffer__ is called -- a good thing! But somehow this makes me
nervous anyway!

I'm also a little concerned that I'm allocating a new buffer each time
a view is taken, so am I going to get multiple copies -- I do have a
big loop in teh test function that calls asarray() over an over again,
and dont see a memory climb, so I guess I"ve got it right.

If this looks like I've done it right -- should I put it in the Wiki?

Thanks
-Chris