Creating typed memoryview from pointer without the GIL

[ajfr...@gmail.com]

Hi, I've got a use case where I'm trying to create a typed memoryview from a raw C pointer. My impression was that this should all be possible without the GIL, but it doesn't seem to be the case. Could anyone help me understand what I'm missing?

I'm getting an error message like:

cdef int[:] _copy_to_mv(int* ptr, size_t n) nogil: cdef array arr = <int[:n]> ptr ^ ------------------------------------------------------------ /Users/ajfriend/.cache/ipython/cython/_cython_magic_cc24b6a67fe400606664b7c5a8b1c440.pyx:5:21: Operation not allowed without gil

But I'm not sure why the GIL would even be involved here, based on what I understand from https://cython.readthedocs.io/en/latest/src/userguide/memoryviews.html#memoryviews-and-the-gil

Thanks!

A minimal example:

Code:

%%cython
from libc.stdlib cimport calloc, free
from cython.view cimport array

cdef int[:] _copy_to_mv(int* ptr, size_t n) nogil:
    cdef array arr = <int[:n]> ptr
    arr.callback_free_data = free
    return arr

cdef size_t n = 10
cdef int* ptr = <int*> calloc(n, sizeof(int))
_copy_to_mv(ptr, n)

[ajfr...@gmail.com]

Thanks! I had thought I was avoiding that with the `from cython.view cimport array`, but maybe I'm wrong about that.

If indeed that is the issue, I guess my follow-up question would be if there is a way to wrap C-allocated memory by a Cython typed memoryview that avoids the GIL?

AJ

[ajfr...@gmail.com]

^ Oops. Just realized I had replied to an email directed at me personally, and not the group. The gist was that cython.view.array might be a Python type, which would need the GIL.

[da-woods]

Ultimately Cython memoryviews wrap Python objects that implement the
buffer protocol. You can do things like cast a C pointer into a Cython
memoryview:

cdef double[::1] xview = <double[:10:1]>(xptr)

However, all this does is creates a dummy Python object to "wrap" the
pointer, so it doesn't really avoid need for the GIL.

The upshot is that you can use (and create subslices from) memoryviews
without the GIL, but I don't think you can do the initial creation
without it.

[ajfr...@gmail.com]

Ok, that's helpful. Thanks!

So it sounds like I need to stay in C pointer land if I want to work with the memory without getting the GIL.

Just out of curiosity: Is there a reason we couldn't *in principle* let use the nice memoryview interface from pure Cython without acquiring the GIL?

Thanks again,

AJ

[da-woods]

> Just out of curiosity: Is there a reason we couldn't *in principle*
> let use the nice memoryview interface from pure Cython without
> acquiring the GIL?
>

It's mostly just that the core idea of the typed memoryview is that it
wraps a Python object that has the buffer protocol. Having another
option for what it could wrap would be a fairly significant change to
the memoryview internals I suspect. Obviously not impossible though.