"Cython memoryview" on existing buffer

[Gabriel Fougeron]

Hi,

I have a large memory buffer, which I'd like to understand as several memoryviews in cython in a nogil block.

In vanilla python, what I mean would be similar to doing somethin like :

import numpy as np

import math as m

shape_0 =(2,3)

shape_1 = (2,2,2)

shifts = np.zeros((3),dtype=np.intp)

shifts[1] = m.prod(shape_0)

shifts[2] = shifts[1] + m.prod(shape_1)

buf = np.random.random((shifts[2]))

arr_0 = buf[shifts[0]:shifts[1]].reshape(shape_0)

arr_1 = buf[shifts[1]:shifts[2]].reshape(shape_1)

print(buf)

print(arr_0)

print(arr_1)

I tried to have a go at it in cython and wrote the following:

cpdef void create_memview(

    double [::1] buf,

    int m,

    int n,

) noexcept nogil:

    cdef double[:,::1] view = <double[:m,:n:1]> &buf[0]

This code does not compile as-is but does compile if I remove the nogil statement.

However, I don't really understand why the GIL would be required here. 

Some questions : 

- Does my cython code do what I think it does?

- Why does this code required the GIL?

- Is there a better way to accomplish what I want?

Thanks in advance

[da-woods]

> - Does my cython code do what I think it does?

Probably. Although it loses any connection between the lifetimes of `view` and `buf`. So `view` is only valid as long as you keep `buf` alive.

> - Why does this code required the GIL?

<double[:m,:n:1]> &buf[0]

This requires the GIL because Cython memoryviews are ultimately underpinned by a Python object. When you create a memoryview from a pointer it makes a new Python object to represent the memoryview. That object doesn't do much but it does need to exist.

> - Is there a better way to accomplish what I want?

I don't think there's a way to do `reshape` with Cython memoryviews alone and without the GIL.

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/3f951e36-17e1-47b9-8958-a2486ef8fca3n%40googlegroups.com.

[Gabriel Fougeron]

Oh wait sorry I read your message too fast my bad! I guess it was wishful thinking on my part.

Anyway, if you d'ont think this is possible, it's enough for me to stop trying. Thanks again :-)

[Peter Schay]

Hi.  There *are* ways to use cython memoryviews without the GIL.

In case you have not already done so, search for "nogil" here: https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html and there are examples.

Will they help with your task?  I don't know :-)  

So far in my project I have always used the memoryview with a python object underneath.

[da-woods]

There's a lot you can do without the GIL but I don't think it's possible to reshape a memoryview without it.

With that said, I don't see a good reason why it isn't possible (at least for 1D -> any shape, and contiguous -> any shape). So it might be worth making a feature request to https://github.com/cython/cython/issues

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cython-users/CALKyFjYLhpjaLPM3X78Yj%3DxHAL2czGoVXnTyYHS5fRF3kUTEyQ%40mail.gmail.com.

[Gabriel Fougeron]

Thanks for the suggestion, I just opened a feature request for this: https://github.com/cython/cython/issues/5995

[Leo Hstone]

Hi! I am trying to achieve something similar, but I already have a 1d memoryview and want to cast it to a 3d memoryview. I understand, from the answer of Woods, that every memoryview need a Python object as base. In my case, is it possible to reuse the base object of the 1d memoryview? From my understanding it should be possible, as we only need to change ndim and strides... Maybe by recreating the memoryview manually and pointing it to the same base?

I have posted a similar reply previously in this thread: https://groups.google.com/g/cython-users/c/jBO_kSBKs7I but I think this one here is more suited.

Thanks a lot!