Getting a pointer to the memory behind a numpy structured dtype

[Nathaniel Smith]

Hi all,

I have a problem that's probably pretty simple, but I'm not sure how
to go about it. I have a numpy array with an arbitrary structured
dtype:

a = np.zeros(10, dtype=["i2,i4"])

Each entry in this array is a packed struct -- a 2-byte integer
followed by a 4-byte integer. But, I want to be able to handle
arbitrary dtypes of this form, and don't care about their internal
structure. I want to be able to take an index

a[i]

and get a void* or char* to the 6 bytes representing that entry in the
array, suitable for passing to memcmp() and such. Regardless of the
actual dtype of the array.

How do I efficiently, in Cython, convert an array of arbitrary dtype +
an index into a void*?

Cheers,
-n

[mark florisson]

If the size of the dtype is variable it isn't really possible. If it's
static you can probably declare a (packed) struct of char[n] and use
the numpy view or astype method.

[Dag Sverre Seljebotn]

Not sure what Nathaniel means, but just doing it manually should work? I.e.

cdef np.ndarray a
a = ...
# Leaving out bounds checking
cdef char *element = <char*>PyArray_DATA(a) + i * PyArray_DIMS(a)[0]


Dag

[Dag Sverre Seljebotn]

s/DIMS/STRIDES/

Dag

[Nathaniel Smith]

Right, it's obviously possible to do directly, but then I have to
think about how to reimplement all the stuff that cython normally does
for me -- bounds checking (or not), handling multi-dimensional
indices, etc. So I was hoping for something more terse. Maybe like

def get_voidptr(void [:] a, Py_ssize_t i):
return &a[i]

but of course this doesn't work. (BTW the np.ndarray[void] version of
this hits an assertion error in the cython compiler rather than
producing a real error message.)

Since the item size is variable, I can't statically declare a Cython
struct to use. I guess the problem is that Cython wants to check that
the ndarray's dtype is compatible with whatever C type it's using, and
to do that it needs to be able to introspect that C type? I can't see
any other reason why this *couldn't* work, just going ahead and
generating C code for the above function in the ordinary way should
basically just DTRT.

-n

[Chris Barker]

On Mon, Aug 13, 2012 at 2:46 AM, Nathaniel Smith <n...@pobox.com> wrote:

>>>> a = np.zeros(10, dtype=["i2,i4"])
>>>>
>>>> Each entry in this array is a packed struct -- a 2-byte integer
>>>> followed by a 4-byte integer. But, I want to be able to handle
>>>> arbitrary dtypes of this form, and don't care about their internal
>>>> structure. I want to be able to take an index
>>>>
>>>> a[i]
>>>>
>>>> and get a void* or char* to the 6 bytes representing that entry in the
>>>> array, suitable for passing to memcmp() and such. Regardless of the
>>>> actual dtype of the array.
>>>>
>>>> How do I efficiently, in Cython, convert an array of arbitrary dtype +
>>>> an index into a void*?

well, for some value of "efficient", maybe as simple as:

b = a[1:i+1] # this should give you a size (1,) array

my_pointer = <void*>b[0]

(untested, syntax may be wrong, but I hope you get the idea)

if the b[0] doesn't work for a untyped array, maybe the .data attribute?

with a typed as a non-specific ndarray.

-Chris



--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov