Recommendations for copying/processing large byte arrays

[Mark Melvin]

Hi Everyone,

Now that I have my callbacks working with a Cython-generated extension module, I can write Python code to grab streaming data from my logic analyzer.  My motivation here is to write a real-time data analyzer in Python (or Cython...).  The problem is, it is going to have to be *fast*.  Essentially, my callback function will be called regularly (approximately 20-30 times per second) with a pointer to a large array of bytes.  I need to do some pretty heavy processing on these values, but I can't afford to bottleneck the acquisition so I will need to probably do the analysis in a background thread and hope I can keep up.  The size of the array could be as large as 1 million bytes for higher sampling rates.  

Not being very good with C/C++/Cython at the moment, I would love some pointers (pun intended) on how to deal with this amount of data.  I need to dispose of the arrays of data as I am done with them to avoid running out of memory, and the logic analyzer provides a method that I can use to delete the arrays when I no longer need them.  So, given the breadth of experience on this list, I was wondering:

- Do I have any hope of succeeding here? :)

- Would it be possible to convert this amount of data to a Python array.array, (or numpy - but I have never used this before so I don't know if it will help me or not) quickly? Looping over the C array and appending to a Python array.array is not fast enough (no big surprise there) but I haven't found another method yet.

- Does anyone have any sample code that does something similar they can point me to?

- I'll need to examine bits within the bytes when analyzing the data - does anyone have any experience doing this efficiently?

I'm hoping I can be successful here and implement something that works in Python/Cython.

Thanks for the support so far - it is very much appreciated!

Regards,

Mark.

[Robert Bradshaw]

On Wed, Jul 27, 2011 at 2:43 PM, Mark Melvin <mark....@gmail.com> wrote:
> Hi Everyone,
> Now that I have my callbacks working with a Cython-generated extension
> module, I can write Python code to grab streaming data from my logic
> analyzer.  My motivation here is to write a real-time data analyzer in
> Python (or Cython...).  The problem is, it is going to have to be *fast*.
>  Essentially, my callback function will be called regularly (approximately
> 20-30 times per second) with a pointer to a large array of bytes.

20-30MB/s is pretty data intensive, but Cython should be able to handle that.

> I need to
> do some pretty heavy processing on these values, but I can't afford to
> bottleneck the acquisition so I will need to probably do the analysis in a
> background thread and hope I can keep up.  The size of the array could be as
> large as 1 million bytes for higher sampling rates.
> Not being very good with C/C++/Cython at the moment, I would love some
> pointers (pun intended) on how to deal with this amount of data.  I need to
> dispose of the arrays of data as I am done with them to avoid running out of
> memory, and the logic analyzer provides a method that I can use to delete
> the arrays when I no longer need them.  So, given the breadth of experience
> on this list, I was wondering:
> - Do I have any hope of succeeding here? :)

Sure, assuming your actual analysis code is fast enough to keep up (or
you can drop buffers).

> - Would it be possible to convert this amount of data to a Python
> array.array, (or numpy - but I have never used this before so I don't know
> if it will help me or not) quickly? Looping over the C array and appending
> to a Python array.array is not fast enough (no big surprise there)

Yeah, that'd probably be quadratic.

> but I haven't found another method yet.
> - Does anyone have any sample code that does something similar they can
> point me to?
> - I'll need to examine bits within the bytes when analyzing the data - does
> anyone have any experience doing this efficiently?
> I'm hoping I can be successful here and implement something that works in
> Python/Cython.

You could look into making a simple cdef class that is essentially a
queue of char* (or char* + length + any other data you need). Your
callback would insert things into the front of the queue, and then
your background thread would pop them off and process/free them. You
could probably even let the GIL take care of locking for you.

It might also look into making NumPy arrays out of the data. You can
even do it in such way as to avoid copying the actual data (it will
take a bit of care to register deallocation) but would probably be the
right way to go to expose things to Python. I would still do this in a
queue (either creating the NumPy arrays directly on the callback or
storing the raw char* and creating NumPy arrays on popping).

- Robert

[Mark Melvin]

Thanks for the suggestions, Robert.  I'm going to mess around with this while on vacation this weekend and next week.  I'm sure I'll have lots of inspirations whilst sitting by the lake! :)

Regards,

Mark.

[Mark Melvin]

On Wed, Jul 27, 2011 at 6:17 PM, Robert Bradshaw <robe...@math.washington.edu> wrote:

You could look into making a simple cdef class that is essentially a
queue of char* (or char* + length + any other data you need). Your
callback would insert things into the front of the queue, and then
your background thread would pop them off and process/free them. You
could probably even let the GIL take care of locking for you.

I've got this sort of working, given the following extension code:

cdef class PyLogic16Interface(PyGenericInterface):

    cdef Logic16Interface *thisptr

    cdef queue[char*] *data_queue

    def __dealloc__(self):

        del self.thisptr

        del self.data_queue

    def read_start(self, ):

        self.thisptr.ReadStart()

    def stop(self, ):

        self.thisptr.Stop()

    cdef void push_data(self, char *data):

        cdef char *dat

        cdef queue[char*] q = deref(self.data_queue)

        q.push(data)

        # Do the equivalent of this in the background analyzer thread

        #if not q.empty():

        #   dat = q.front()

        #  q.pop()

        # DeleteU8ArrayPtr(<U8*> dat)

But based on various searches and documentation bits, I am forced to create my instance variables in a wrapper function like this:

cdef PyLogic16Interface PyLogic16Interface_factory(Logic16Interface *cppLogic16Interface):

    cdef PyLogic16Interface instance = PyLogic16Interface.__new__(PyLogic16Interface)

    instance.thisptr = cppLogic16Interface

    cdef queue[char*] *q = new queue[char*]()

    instance.data_queue = q

    return instance

Is this still required with Cython 0.15, or am I using outdated information?  I'd love to be able to do this more cleanly.

Also, feel free to add suggestions on my above code - i am doing a lot of this by trial and error.  In fact the above gives me all kinds of warnings from the MSVC compiler like this:

C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1386) : warning C4530: C++ exception handler used, but unwind semantics are not enabled. Specify

/EHsc

        C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1358) : see reference to function template instantiation 'void std::deque<_Ty>::_Insert<_It>(std::_Deque_const_iterator<_Ty,_Alloc>,_It,_It,std::input_iterator_tag)' being compiled

        with

        [

            _Ty=char *,

            _It=std::_Deque_const_iterator<char *,std::allocator<char *>>,

            _Alloc=std::allocator<char *>

        ]

(there are multiple warnings of this type).

I will try passing the compiler the argument specified in the warning.

Thanks!

Mark.

[Robert Bradshaw]

That depends where cppLogic16Interface is coming from. If you're
making a new one, or have some Python handle to one, then no.

>  I'd love to be able to do this more cleanly.
> Also, feel free to add suggestions on my above code - i am doing a lot of
> this by trial and error.  In fact the above gives me all kinds of warnings
> from the MSVC compiler like this:
> C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1386) :
> warning C4530: C++ exception handler used, but unwind semantics are not
> enabled. Specify
> /EHsc
>         C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1358)
> : see reference to function template instantiation 'void
> std::deque<_Ty>::_Insert<_It>(std::_Deque_const_iterator<_Ty,_Alloc>,_It,_It,std::input_iterator_tag)'
> being compiled
>         with
>         [
>             _Ty=char *,
>             _It=std::_Deque_const_iterator<char *,std::allocator<char *>>,
>             _Alloc=std::allocator<char *>
>         ]
> (there are multiple warnings of this type).
> I will try passing the compiler the argument specified in the warning.

Yes. These are probably requirements of the libraries you're
including. Read up on the except+ syntax if you expect C++ exceptions
to be thrown.

- Robert

[Mark Melvin]

I also could use some help with the following:

ctypedef unsigned char U8

cdef queue[U8*] *q = new queue[U8*]()

This fails to build with an "expected type or identifier".  If I change it to:

cdef queue[unsigned char*] *q = new queue[unsigned char*]()

It works fine.  I could live with doing this, but not being able to use the "U8" type causes problems for me later.  This is because it is defined in my external library I am trying to wrap, and if I try to pass pointers to U8s around to my extensions, I get an error about being unable to convert U8 * to a Python object.  I still seem to be missing something about types here...

Thanks,

Mark.

[Mark Melvin]

OK.  It isn't created by me - it comes from the external library so I guess this is still required.  I could probably create my queue in the extension __cinit__ then.

 

>  I'd love to be able to do this more cleanly.
> Also, feel free to add suggestions on my above code - i am doing a lot of
> this by trial and error.  In fact the above gives me all kinds of warnings
> from the MSVC compiler like this:
> C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1386) :
> warning C4530: C++ exception handler used, but unwind semantics are not
> enabled. Specify
> /EHsc
>         C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\deque(1358)
> : see reference to function template instantiation 'void
> std::deque<_Ty>::_Insert<_It>(std::_Deque_const_iterator<_Ty,_Alloc>,_It,_It,std::input_iterator_tag)'
> being compiled
>         with
>         [
>             _Ty=char *,
>             _It=std::_Deque_const_iterator<char *,std::allocator<char *>>,
>             _Alloc=std::allocator<char *>
>         ]
> (there are multiple warnings of this type).
> I will try passing the compiler the argument specified in the warning.

Yes. These are probably requirements of the libraries you're
including. Read up on the except+ syntax if you expect C++ exceptions
to be thrown.

OK - thanks. I'll look into this.

Mark. 

[Mark Melvin]

I figured out why I was getting the 'can't convert to Python object' errors. When creating the instance of my extension object I was not cdef'ing it and declaring the type. Adding the extra info made it work.

I still can't seem to declare a queue that contains a ctypedef'd type, but I can work around this by declaring the actual type. I don't know if this is a bug, limitation, or my own lack of knowledge.

Thanks,
Mark

---

From: Mark Melvin <mark....@gmail.com>

Date: Tue, 2 Aug 2011 16:19:46 -0400

To: <cython...@googlegroups.com>

Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

[Robert Bradshaw]

On Tue, Aug 2, 2011 at 7:30 PM, Mark Melvin <mark....@gmail.com> wrote:
> I figured out why I was getting the 'can't convert to Python object' errors.
> When creating the instance of my extension object I was not cdef'ing it and
> declaring the type. Adding the extra info made it work.
>
> I still can't seem to declare a queue that contains a ctypedef'd type, but I
> can work around this by declaring the actual type. I don't know if this is a
> bug, limitation, or my own lack of knowledge.

This works for me:

from libcpp.queue cimport queue

ctypedef unsigned char U8
cdef queue[U8*] *q = new queue[U8*]()

> OK. It isn't created by me - it comes from the external library so I guess this is still required. I could probably create

> my queue in the extension __cinit__ then.

Yes. You could even assign thisptr to NULL, and the have a function
that sets it. (Make sure your __dealloc__ checks for NULL before
deleting it...)

- Robert

[Gregor Thalhammer]

> Hi Everyone,
>
> Now that I have my callbacks working with a Cython-generated extension module, I can write Python code to grab streaming data from my logic analyzer. My motivation here is to write a real-time data analyzer in Python (or Cython...). The problem is, it is going to have to be *fast*. Essentially, my callback function will be called regularly (approximately 20-30 times per second) with a pointer to a large array of bytes. I need to do some pretty heavy processing on these values, but I can't afford to bottleneck the acquisition so I will need to probably do the analysis in a background thread and hope I can keep up. The size of the array could be as large as 1 million bytes for higher sampling rates.

This seems similar to a problem I was working on, analyzing and displaying images from a video camera. 60 images per second, 1 MByte each. Handling this amount of data was no problem.

> Not being very good with C/C++/Cython at the moment, I would love some pointers (pun intended) on how to deal with this amount of data. I need to dispose of the arrays of data as I am done with them to avoid running out of memory, and the logic analyzer provides a method that I can use to delete the arrays when I no longer need them. So, given the breadth of experience on this list, I was wondering:
>
> - Do I have any hope of succeeding here? :)

depends on your data analysis, but with numpy/cython I would say: yes, easily. A modern CPU can handle a data rate of about 10 GB/s.

> - Would it be possible to convert this amount of data to a Python array.array, (or numpy - but I have never used this before so I don't know if it will help me or not) quickly? Looping over the C array and appending to a Python array.array is not fast enough (no big surprise there) but I haven't found another method yet.

I prefer using numpy arrays. I made a copy on array creation (numpy.frombuffer), so I didn't need to care about releasing the memory. At the data rate you mentioned I didn't experience any limitation due to the copy, but there are possibilities to create a numpy array from a pointer without copying data and proper memory release, there exists a detailed description (by Travis Oliphant?), but I lost the link.

> - Does anyone have any sample code that does something similar they can point me to?

You get your data in quite big chunks. I would convert them to a numpy array, put them into a Queue, and use another thread for data analysis.

> - I'll need to examine bits within the bytes when analyzing the data - does anyone have any experience doing this efficiently?

Just a guess: use the usual bit operators in numpy: bitwise_XXX()

Gregor

[Mark Melvin]

I still can't get this to work:

from libcpp.queue cimport queue
ctypedef unsigned char U8
cdef queue[U8*] *q = new queue[U8*]()

Even if I boil it down to these three lines only.

I get an 'Expected an identifier or literal' compile error pointing at the asterisk after 'U8'. Is it possible this only works in 0.15, and not in 0.14? I'm starting to wonder if I installed the latest RC properly. I did get an error from the MSVC compiler about a missing manifest file (which I haven't figured out how to fix yet), but the link succeeds. It's possible this made the install bail before it was done, though.

I'll go back and double-check this.

Regards,
Mark.

-----Original Message-----
From: Robert Bradshaw <robe...@math.washington.edu>
Sender: cython...@googlegroups.com
Date: Tue, 2 Aug 2011 21:20:03
To: <cython...@googlegroups.com>
Reply-To: cython...@googlegroups.com
Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

[Mark Melvin]

Problem solved. It turns out 0.15 was not fully installed. And I fixed my manifest issue. It was due to this:

http://bugs.python.org/issue4431

Thanks,
Mark

-----Original Message-----
From: "Mark Melvin" <mark....@gmail.com>
Date: Wed, 3 Aug 2011 15:33:24
To: <cython...@googlegroups.com>
Reply-To: mark....@gmail.com
Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

I still can't get this to work:

from libcpp.queue cimport queue
ctypedef unsigned char U8
cdef queue[U8*] *q = new queue[U8*]()

Even if I boil it down to these three lines only.

I get an 'Expected an identifier or literal' compile error pointing at the asterisk after 'U8'. Is it possible this only works in 0.15, and not in 0.14? I'm starting to wonder if I installed the latest RC properly. I did get an error from the MSVC compiler about a missing manifest file (which I haven't figured out how to fix yet), but the link succeeds. It's possible this made the install bail before it was done, though.

I'll go back and double-check this.

Regards,
Mark.


-----Original Message-----
From: Robert Bradshaw <robe...@math.washington.edu>
Sender: cython...@googlegroups.com
Date: Tue, 2 Aug 2011 21:20:03
To: <cython...@googlegroups.com>

Reply-To: cython...@googlegroups.com
Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

[Mark Melvin]

Hi Gregor,

(Sorry for the top-post; I don't have internet and am replying from my smartphone.)

Thanks for your reply - it is very useful. It also gives me more hope that what I am trying to do is possible.

Having prototyped a bit with raw C arrays, I think having numpy arrays would be more flexible moving forward. Without the ability to search online for a day or two I am struggling a bit.

How exactly can I use numpy.frombuffer to get a numpy array from a pointer to an unsigned char, for instance? Cython is giving me the 'can't convert to Python object error'. I imagine there is some magic to tell Cython it is a numpy array, and I do recall a numpy example on the wiki (which I will look at as soon as I am back online), but if you could give me a hint, I would very much appreciate it!

I'm hoping this is fast enough. If not, I'll cross that bridge when I come to it.

Thanks,
Mark

-----Original Message-----
From: Gregor Thalhammer <Gregor.T...@i-med.ac.at>
Sender: cython...@googlegroups.com
Date: Wed, 3 Aug 2011 16:11:36
To: <cython...@googlegroups.com>
Reply-To: cython...@googlegroups.com
Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

[Francesc Alted]

2011/8/3, Mark Melvin <mark....@gmail.com>:

> Hi Gregor,
>
> (Sorry for the top-post; I don't have internet and am replying from my
> smartphone.)
>
> Thanks for your reply - it is very useful. It also gives me more hope that
> what I am trying to do is possible.
>
> Having prototyped a bit with raw C arrays, I think having numpy arrays would
> be more flexible moving forward. Without the ability to search online for a
> day or two I am struggling a bit.
>
> How exactly can I use numpy.frombuffer to get a numpy array from a pointer
> to an unsigned char, for instance? Cython is giving me the 'can't convert
> to Python object error'. I imagine there is some magic to tell Cython it is
> a numpy array, and I do recall a numpy example on the wiki (which I will
> look at as soon as I am back online), but if you could give me a hint, I
> would very much appreciate it!

Once you declare a NumPy object as such in Python (via
np.ndarray[dtype...]) you can access the pointer to actual data via
`your_array.data` (a pointer to unsigned char, I think). That's very
fast, and you don't need a specific call at all.

--
Francesc Alted

[Mark Melvin]

Hi Francesc,

Thank you for the reply. I am stuck, however at getting the numpy array created to point to my raw C array. I can't pass a C pointer into numpy.frombuffer or numpy.array without the 'can't covert to Python object' error. I assume there is an easy way to accomplish this without creating an empty numpy array and copying each value into it?

Thanks,
Mark

-----Original Message-----
From: Francesc Alted <fal...@pytables.org>
Sender: cython...@googlegroups.com
Date: Wed, 3 Aug 2011 22:21:36
To: <cython...@googlegroups.com>
Reply-To: cython...@googlegroups.com
Subject: Re: [cython-users] Recommendations for copying/processing large byte arrays

[Robert Bradshaw]

On Wed, Aug 3, 2011 at 2:26 PM, Mark Melvin <mark....@gmail.com> wrote:
> Hi Francesc,
>
> Thank you for the reply. I am stuck, however at getting the numpy array created to point to my raw C array.  I can't pass a C pointer into numpy.frombuffer or numpy.array without the 'can't covert to Python object' error. I assume there is an easy way to accomplish this without creating an empty numpy array and copying each value into it?

Try creating an empty numpy array of the right size and then using
memcpy(ndarray.data, source, length). You can then deallocate the
source. (Yes, it is possible to avoid the copy, but that's a lot
messier and falls under what I would call "premature optimization.")

[Christopher Barker]

On 8/3/11 2:32 PM, Robert Bradshaw wrote:
> Try creating an empty numpy array of the right size and then using
> memcpy(ndarray.data, source, length). You can then deallocate the
> source. (Yes, it is possible to avoid the copy, but that's a lot
> messier and falls under what I would call "premature optimization.")

indeed. But if you really want to, one numpy C API call to do it is:

PyArray_SimpleNewFromData (PyObject*)(int nd npy_intp* dims, int typnum,
void data*)

data* is the pointer to your data. Numpy won't de-allocate that memory,
and if you do de-allocate it when the numpy array is still around, ugly
things will happen.

but it will get you a numpy array with no memory copying required.

I don't know if there is a Cython shorthand for a similar call.

-Chris

--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris....@noaa.gov

[Mark Melvin]

Thank you very much! This seems to work and I can copy the entire memory buffer and reduce/bitwise xor the copied arrays in realtime without crashes (for a reasonable sampling rate), which bodes well for the general feasibility of things.

Now to clean up the code a bit and implement a Queue and a proper data analyzer.

Thanks for all the help, guys. I appreciate you taking the time to answer my somewhat noob-ish questions.

Regards,
Mark

-----Original Message-----
From: Robert Bradshaw <robe...@math.washington.edu>

[Robert Bradshaw]

On Wed, Aug 3, 2011 at 3:20 PM, Christopher Barker
<Chris....@noaa.gov> wrote:
> On 8/3/11 2:32 PM, Robert Bradshaw wrote:
>>
>> Try creating an empty numpy array of the right size and then using
>> memcpy(ndarray.data, source, length). You can then deallocate the
>> source. (Yes, it is possible to avoid the copy, but that's a lot
>> messier and falls under what I would call "premature optimization.")
>
> indeed. But if you really want to, one numpy C API call to do it is:
>
> PyArray_SimpleNewFromData (PyObject*)(int nd npy_intp* dims, int typnum,
> void data*)
>
> data* is the pointer to your data. Numpy won't de-allocate that memory, and
> if you do de-allocate it when the numpy array is still around, ugly things
> will happen.

Yeah, that's the tricky part. Just out of curiosity, is there a way to
register a callback that gets invoked when the NumPy array is freed?

> but it will get you a numpy array with no memory copying required.
>
> I don't know if there is a Cython shorthand for a similar call.

I don't think there is one (yet, memory views might be a partial answer).

[Gregor Thalhammer]

Am 4.8.2011 um 00:26 schrieb Robert Bradshaw:

> On Wed, Aug 3, 2011 at 3:20 PM, Christopher Barker
> <Chris....@noaa.gov> wrote:
>> On 8/3/11 2:32 PM, Robert Bradshaw wrote:
>>>
>>> Try creating an empty numpy array of the right size and then using
>>> memcpy(ndarray.data, source, length). You can then deallocate the
>>> source. (Yes, it is possible to avoid the copy, but that's a lot
>>> messier and falls under what I would call "premature optimization.")
>>
>> indeed. But if you really want to, one numpy C API call to do it is:
>>
>> PyArray_SimpleNewFromData (PyObject*)(int nd npy_intp* dims, int typnum,
>> void data*)
>>
>> data* is the pointer to your data. Numpy won't de-allocate that memory, and
>> if you do de-allocate it when the numpy array is still around, ugly things
>> will happen.
>
> Yeah, that's the tricky part. Just out of curiosity, is there a way to
> register a callback that gets invoked when the NumPy array is freed?

Travis Oliphant describes how to do this (in plain C):
http://blog.enthought.com/python/numpy-arrays-with-pre-allocated-memory/
I would like to see this translated to cython, hope somebody else did/will do this ;-)

Gregor

Sorry, the frombuffer hint was incomplete, before I copied the data to a string in cython:
datastr = (<char *>ptr)[:length]

[Christopher Barker]

On 8/3/11 4:03 PM, Gregor Thalhammer wrote:
> Am 4.8.2011 um 00:26 schrieb Robert Bradshaw:
>> Yeah, that's the tricky part. Just out of curiosity, is there a way to
>> register a callback that gets invoked when the NumPy array is freed?
>
> Travis Oliphant describes how to do this (in plain C):
> http://blog.enthought.com/python/numpy-arrays-with-pre-allocated-memory/
> I would like to see this translated to cython, hope somebody else did/will do this ;-)

yep -- that would be nice -- and Travis proposed that there be a
pre-written version in the numpy C api too, that might be the place to
start.

thanks for the link.

I've actually been thinking about a numpy-compatible way to do nice
nd-arrays in C++, and have been thinking that it would build on two core
classes:

1) a "data block" class -- this would be a pretty simple wrapper around
a pointer to a block of memory. Pretty much all it would have is the
pointer and a reference count, and then ways to increment and decrement
the ref count, and delete the memory when the count goes to zero. It
would have no idea how the memory was formatted, or how it was used,
etc. (could this just be a "smart pointer")

2) an array class that would use a data block for the data itself, and
it would hold all the info about types, strides, etc. When one gets
created from scratch, it would create a data block instance, if one was
created by a view or slicing, etc operation, it would increment the data
block ref count. (and, of course, decrement when arrays are deleted)

It seems this would allow me to create nifty numpy-like operations in
C++, and have something easy to pass on to a numpy array object when I
wanted to interact with Python. Travis' blog post shows me how to
connect python/numpy ref counting with my "data block" class.

NOTE: this is all still just an idea -- who knows if I'll ever write the
code....

Oh -- maybe a class with a scheme like this already exists (boost
arrays?) I haven't looked hard enough yet, though want I've found
doesn't support numpy's slices-as-views approach well -- and I really
like that.

-Chris

[mark florisson]

With memoryviews you can cast any pointer to a cython.array. This
array exposes the buffer interface.

cython_array = <int[:10, :10]> mypointer

this indicates a C-contiguous block of size 10 * 10 * sizeof(int). The
array will not make a copy, and you can set a callback on the array
object that will be called to free the data (the default is to call
free()). You can then also get a typed memoryview of this array which
can be efficiently passed around, sliced (to be implemented) and
indexed (all in nogil mode). You can specify fortran contiguity with
int[:10:1, :10].

I suppose we could give memoryviews a numpy interface. It seems
numpy.frombuffer can only use the old buffer interface. But first I'll
implement slicing and I'm currently a little busy, so we may have to
wait a bit until I (or someone else who's willing) can find the time.

I think it would have to be a subclass of numpy.ndarray implementing
__array_finalize__? Then you can simply use PyArray_NewFromDescr().
I'm not sure what to do with indirect dimensions though... perhaps we
should require an explicit copy for that to a contiguous block (these
memoryviews have copy methods)?

[Dag Sverre Seljebotn]

Having np.asarray/frombuffer support PEP3118 is something that should
obviously be fixed in NumPy itself.

However, it wouldn't hurt to support current and earlier versions of
NumPy. All that is needed is to set the __array_interface__ attribute,
no subclassing:

http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

Dag Sverre

[mark florisson]

Hmm, it seems easier to have a custom object with __cinit__ and
__dealloc__. But these details are not important...

[mark florisson]

Oh very neat, thanks for the pointer!