Reshaping typed memoryviews

[Toni Barth]

Hi,

I'm searching for a most-efficient way to declare an already allocated
memory view or, if this isn't possible, work around it.
Thats why I used a command like this:
cdef int[:, :] = cython.view.array(shape=(1280, 960),
itemsize=sizeof(int), format='i', allocate_buffer = True)

That gives me an allocated memoryview with defined shape at least.
I now profiled it and found out that the function using this aproach
is rather slow. There is of course much more going on within this
function, so i'm not entirely sure if this is actually the course of
the slowdown here, but creating an array object just to define a
memoryview with a given shape seems rather extreme to me.
Thats why I tried a different aproach:

The final result of this function should be to read in pixel data from
a pgm file and put them into a two-dimensional view (no matter if
array, memory view or whatever), but I need to be as native as
possible, so no numpy dependency.

I now created a C buffer using malloc which is large enough to take
all pixel data after another, shaped just C-contiguous so that reading
it in should be no problem.

I'm just struggling to move it into a cython-native memoryview now.
I tried the following, which works, but leaves me with a 1d view:
cdef int[:] view = <int[:sizeof(int) * width * height]>buf

Numpy would now allow me to simply reshape the 1d view to get a 2d
view, just as I need, but the typed memoryview doesn't have a reshape
method.

Does anyone have an idea how to solve this? Maybe there already is a
way to read the 1d buffer in as a 2d view, i'm just not getting the
hang of it?

Thanks.

Best Regards.

Toni

[Chris Barker]

On Wed, Apr 25, 2018 at 2:50 AM, 'Toni Barth' via cython-users <cython...@googlegroups.com> wrote:

I'm searching for a most-efficient way to declare an already allocated
memory view or, if this isn't possible, work around it.
Thats why I used a command like this:
cdef int[:, :] = cython.view.array(shape=(1280, 960),
itemsize=sizeof(int), format='i', allocate_buffer = True)

 

That gives me an allocated memoryview with defined shape at least.
I now profiled it and found out that the function using this aproach
is rather slow. There is of course much more going on within this
function, so i'm not entirely sure if this is actually the course of
the slowdown here, but creating an array object just to define a
memoryview with a given shape seems rather extreme to me.

well, an array object isn't that heavyweight -- I'll guess that allocating the memory buffer is as or more expensive than the array object itself. You'll need to do some profiling if you really want to know, but I doubt this is noticably slow.

 

Thats why I tried a different aproach:

The final result of this function should be to read in pixel data from
a pgm file and put them into a two-dimensional view (no matter if
array, memory view or whatever), but I need to be as native as
possible, so no numpy dependency.

I now created a C buffer using malloc

that should be about as fast as you can get -- though now you need to manage that memory -- python won't do it for you.

 

I'm just struggling to move it into a cython-native memoryview now.

I tried the following, which works, but leaves me with a 1d view:
cdef int[:] view = <int[:sizeof(int) * width * height]>buf

did you try simply:

cdef int[:,:] view = <int[:sizeof(int) * width * height]>buf

that should give you a 2D memoryview, just like it does when you use an array.array (which are 1D as well)

 

Numpy would now allow me to simply reshape the 1d view to get a 2d
view, just as I need,

when you reshape a numpy array, all yu are doing is chanign the strides, the  memory buffer is not effected, it's not really doing much.

if you want a memoryview with a different shape pointing ot the same buffer, you can simply make a new one.

Does anyone have an idea how to solve this? Maybe there already is a
way to read the 1d buffer in as a 2d view,

try the above -- there is no such thing as a "1D buffer" a buffer is always 1D (or non-dimensional, depending on how you want to think about it -- it's just a block of allocated memory. The whole point of a memoryview it to provide a way to work with a block of memory wit a bit more "smarts"

 

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Toni Barth]

Hi Chris,

thanks for your help.

Yeah, you're right, it works for me that way now (more below).

>> I'm just struggling to move it into a cython-native memoryview now.
>>
> I tried the following, which works, but leaves me with a 1d view:
>> cdef int[:] view = <int[:sizeof(int) * width * height]>buf
>>
>
> did you try simply:
>
> cdef int[:,:] view = <int[:sizeof(int) * width * height]>buf
>
> that should give you a 2D memoryview, just like it does when you use an
> array.array (which are 1D as well)
>
>

I believe I tried this earlier and I got a compilation error, but
thanks to your hint, I finally found a solution:
cdef int[:, ::1] = <int[:width, :height]>buf

Since this command just sets up a memory view on top of the already
existing memory, I needed to use copy() on it to get a self-managed
memory view out of it, because I want to free the original buffer
right afterwards, since I can't know when the user actually trashes
the memory view and the gc recycles it.

Or is there any way to give a function to the view which automatically
frees the memory when the last handle gets released, so I don't need
to copy all of the data again but just write a simple function
instead?

>> Numpy would now allow me to simply reshape the 1d view to get a 2d
>> view, just as I need,
>
>
> when you reshape a numpy array, all yu are doing is chanign the strides,
> the memory buffer is not effected, it's not really doing much.
>
> if you want a memoryview with a different shape pointing ot the same
> buffer, you can simply make a new one.
>
> Does anyone have an idea how to solve this? Maybe there already is a
>> way to read the 1d buffer in as a 2d view,
>
>
> try the above -- there is no such thing as a "1D buffer" a buffer is always
> 1D (or non-dimensional, depending on how you want to think about it -- it's
> just a block of allocated memory. The whole point of a memoryview it to
> provide a way to work with a block of memory wit a bit more "smarts"
>
> -CHB

Thanks. With a bit of optimization, including this solution above, I
already reduced the processing time of a 1280 x 960 pgm file with 4
megabytes size from 0.6 seconds down to 73 msecs, which is absolutely
enough for my current situation.

Thanks again.

Best Regards.

Toni

>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris....@noaa.gov
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Chris Barker]

On Thu, Apr 26, 2018 at 11:10 PM, 'Toni Barth' via cython-users <cython...@googlegroups.com> wrote:

thanks to your hint, I finally found a solution:
cdef int[:, ::1] = <int[:width, :height]>buf

Since this command just sets up a memory view on top of the already
existing memory, I needed to use copy() on it to get a self-managed
memory view out of it, because I want to free the original buffer
right afterwards, since I can't know when the user actually trashes
the memory view and the gc recycles it.

Or is there any way to give a function to the view which automatically
frees the memory when the last handle gets released, so I don't need
to copy all of the data again but just write a simple function
instead?

that's one reason to use a pyton object to create (and manage) the memory. HAve you profiled it with this vs using an array.array object? I can't imagine that createing an array.array is going to take long enough to notice.

There is also the "Cython array":

from cython.view cimport array as cvarray

# Memoryview on a Cython array
cyarr = cvarray(shape=(3, 3, 3), itemsize=sizeof(int), format="i")
cdef int [:, :, :] cyarr_view = cyarr

from:

http://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html

That may be the lightest weight way to get python-managed memory.

In my use cases, I pretty much always want a numpy array for users' access in Python code, so I don't have any real experience with allocing the memory by hand in Cython...

Thanks. With a bit of optimization, including this solution above, I
already reduced the processing time of a 1280 x 960 pgm file with 4
megabytes size from 0.6 seconds down to 73 msecs, which is absolutely
enough for my current situation.

good news! -- have you made sure you're not leaking memory? That can be the trick when you malloc by hand...

-CHB