Can I switch from one array to another without gil?

[Marko Loparic]

Hello,

I would like to select a numpy array from a list without using gil. Is
it possible? Or do I have to use c arrays and pointers instead?

Here is a sample code showing more precisely what I want to do. The
line I want to avoid is marked with "# XXX".

$ cat f.pyx
def compute(names, arrays, int[:,:] keys, double[:] result):
cdef int key_idx, array_idx, i, j
cdef double[:,:] array
array_list = [arrays[name] for name in names]

with nogil:
for idx in range(keys.shape[0]):
array_idx = keys[idx, 0]
i = keys[idx, 1]
j = keys[idx, 2]
with gil:
array = array_list[array_idx] # XXX
result[idx] = array[i, j]

$ cat call.py
import numpy as np
from f import compute

a1 = np.arange(6, dtype=float).reshape((3, 2))
a2 = np.arange(10, dtype=float).reshape((2, 5))
keys = np.array([
[0, 2, 0],
[1, 1, 4],
])
result = np.zeros((2,))
arrays = {'a1': a1, 'a2': a2}

compute(['a1', 'a2'], arrays, keys, result)
print('result =', result)

$ python call.py
result = [ 4. 9.]

Note the for the moment array_list is a python list. The idea would be
to use a numpy array with references (?) to the arrays instead of it.

Combining all the arrays in a single big array is not an option in my
real case (they have different shapes and a big volume of data would
have to be duplicated).

Thanks a lot in advance!
Marko

[Stefan Behnel]

In that case, I don't see a way to avoid unpacking the views separately,
for which you need the GIL. But:

If your code is really doing as little as in your example above, then I
would guess that going through NumPy's Python interface and avoiding the
overhead of unpacking the buffers at all could be much faster.

And, if this problem is pressing enough for you, then I think the best you
can do is to avoid NumPy arrays completely and write your own data memory
wrapper that knows about the list of differently sized arrays, but still
stores them in a contiguous memory buffer (if possible). Then you can move
the indirect item access code in the loop above into an inline nogil method
that does the index offset calculations itself. It's not difficult, just
needs good tests to avoid the usual off-by-1 errors etc.

Stefan