How to instantiate a multidimensional indirect memoryview?

[Gaël]

Hi there,

I've been trying to define, instantiate and modify a 3D memoryview pointing to continuous arrays of integers. In principle, I want something like this:

int[:, :, :, ::1] data

but the size of that last dimension depends on the first three ones... For instance:

data[0, 0, 0, :]

could be a slice of size 10 while

data[0, 0, 1, :]

could be a slice of size 100000.

In other words, I would need the following

(int*)[:, :, ::1] array_of_arrays

which, in valid Cython, should translate to something like:

from cython cimport view
cdef int[:, :, ::view.indirect_contiguous, ::1] data

from what I can tell. However I can't find the way to properly instantiate such a memoryview... Something like:

from cython cimport view
nx, ny, nz = 2, 7, 9
cdef int[:, :, ::view.indirect_contiguous, ::1] data
data = <int[:nx, :ny, :nz:1]> malloc(nx * ny * nz * sizeof(int*))

fails at compile time when assigning the newly created array to data:

Memoryview 'int[:, :, ::1]' not conformable to memoryview 'int[:, :, ::indirect_contiguous, ::contiguous]'

Something like that doesn't work either:

from cython cimport view
nx, ny, nz = 2, 7, 9
cdef int[:, :, ::view.indirect_contiguous, ::1] data
data = <int[:nx, :ny, :nz, :1:1]> malloc(nx * ny * nz * sizeof(int*))

Memoryview 'int[:, :, :, ::1]' not conformable to memoryview 'int[:, :, ::indirect_contiguous, ::contiguous]'.

I guess (though I haven't tried yet), that I should be able to work with raw pointers instead but memoryviews would be so much neater!

So is it possible to allocate such an array of arrays from within Cython?

[Gaël]

Oh, I just realised I forgot the last case:

from cython cimport view
nx, ny, nz = 2, 7, 9
cdef int[:, :, ::view.indirect_contiguous, ::1]

data
data = <int[:nx, :ny, :nz:view.indirect_contiguous, :1:1]> malloc(nx * ny * nz * sizeof(int*))

Strides may only be given to indicate contiguity. Consider slicing it after conversion

[Stefan Behnel]

Gaël schrieb am 21.03.2018 um 20:53:
> I've been trying to define, instantiate and modify a 3D memoryview pointing
> to continuous arrays of integers. In principle, I want something like this:
>
> int[:, :, :, ::1] data
>
> but the size of that last dimension depends on the first three ones... For
> instance:
>
> data[0, 0, 0, :]
>
> could be a slice of size 10 while
>
> data[0, 0, 1, :]
>
> could be a slice of size 100000.

Such a highly irregular structure cannot be represented by Python's buffer
interface, which is what memoryviews are based on.

> In other words, I would need the following
>
> (int*)[:, :, ::1] array_of_arrays

Well, that seems to be precisely what you want, according.to your initial
description above: a 3D array of pointers to int(s).

> which, in valid Cython, should translate to something like:
>
> from cython cimport view
> cdef int[:, :, ::view.indirect_contiguous, ::1] data

No, this is a 4D view. You only have a 3D view, and the int arrays that the
buffer points to are not part of the buffer.

> I guess (though I haven't tried yet), that I should be able to work with raw pointers instead but memoryviews would be so much neater!

You can work with a memory view as long as you stay within the three
dimensions that point to the int arrays. You cannot make the int arrays
themselves part of the same view, because they are not part of the same
memory area.

However, note that Cython has several nice features also for arrays and
pointers, such as slicing when looping over them:

for x in c_int_ptr[:100]:
print(x)

So, feel free to stick to the memory view for finding the arrays, but then
take the pointer that you find in the view and process its data separately.

Stefan

[Gaël]

Rho... Nope, not working... Cython really doesn't like to cast things to array of pointers:

cdef void* _buff_addr
cdef int[:nx, :ny, :nz:view.indirect_contiguous] buf
...
_buf_addr = malloc(nx*ny*nz * sizeof(int*))
...
buf = <int[:nx, :ny, :nz:view.indirect_contiguous]>_buf_addr

results in:

Strides may only be given to indicate contiguity. Consider slicing it after conversion

:(

I understand the problem. The error is clear. However I can't figure out a workaround... (int*)[...] definitely won't work... int[:nx, :ny, :nz] won't work or would assume that sizeof(int*) == sizeof(int) which is not true in general.

[Gaël]

Hi Stefan, 

Thanks for your input! Things are now much clearer.

Should I understand then, that this: 

from cython cimport view
cdef int[:, :, ::view.indirect_contiguous] array

is Cython's equivalent of

int** array

but with neat indexing?

If so it turned out much simpler than what I was expecting. Thanks again!

Gaël