On Thu, Jul 26, 2012 at 11:12 AM, Chris Barker <chris.bar...@noaa.gov> wrote:
> On Wed, Jul 25, 2012 at 9:41 PM, Robert Bradshaw <rober...@gmail.com> wrote:
>>>> The following code results in an IndexError:
>>>> print 'This is going to be out of bounds.'
>>>> cdef np.ndarray[long, ndim=2, mode="c"] two_d_arr = np.array([[],[]],
>>>> dtype=int)
>>>> print two_d_arr
>>>> printf("pointer to null data: %u\n", <unsigned long>two_d_arr.data)
On Thu, Jul 26, 2012 at 11:59 AM, Sturla Molden <sturlamol...@yahoo.no> wrote:
> Den 26.07.2012 20:12, skrev Chris Barker:
>> Which is why I suppose we really should have a canonical way to get
>> that pointer -- thus arr.data, but it has its problems, as well.
> That is what
> <dtype_t*> np.PyArray_DATA(arr)
> does.
But does that give you the address of the zeroth element? or the
address of the beginning of the data block? -- which I understand may
not be the case, say for an array that is a slice of another array.
i.e should I update that Wiki page with
<dtype_t*> np.PyArray_DATA(arr)
instead of
&arr[0]
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
> On Thu, Jul 26, 2012 at 11:59 AM, Sturla Molden <sturlamol...@yahoo.no> wrote:
>> Den 26.07.2012 20:12, skrev Chris Barker:
>>> Which is why I suppose we really should have a canonical way to get
>>> that pointer -- thus arr.data, but it has its problems, as well.
>> That is what
>> <dtype_t*> np.PyArray_DATA(arr)
>> does.
> But does that give you the address of the zeroth element? or the
> address of the beginning of the data block? -- which I understand may
> not be the case, say for an array that is a slice of another array.
Those will always be the same. When you slice an array that changes
the starting element in some dimension, the data pointer is moved for
the new view.
>> But does that give you the address of the zeroth element?
> Yes.
Strictly speaking it gives the address of the "zeroth byte of the zeroth element". I.e. it gives you a char*, and the strides along each axis are in number of characters, not number of elements. That is why we do pointer arithmetics on strides first, and then cast to a pointer of the correct type. If we have certain knowledge about the strides of the array, e.g. that it is contiguous in C order, we can avoid this character fiddling and use pointer arithmetics directly on elements instead. That is why specifying mode="c" and e.g. ndim=2 in the cdef of the ndarray will speed things up considerably.
Great; thanks for the response! This is my first time posting here so it's good to check back and see an involved discussion on the question I had-- namely, how to extract the buffer of a numpy array in Cython.
After reading through all the responses, it appears as though Sturla's solution:
<dtype_t*> np.PyArray_DATA(arr)
is the right way to go. It is superior to the &a[0] solution because it doesn't require bounds checking, and it is superior to the .data solution because 1) it won't break as numpy changes and 2) BONUS: it works even on arrays that haven't been statically casted to a Cython np.ndarray[...]. To fully understand what I mean by (2), take a look at the following example:
a = np.arange(10, dtype=int)
# This works even if a hasn't been casted yet. fprintf(stderr, "%lu %lu\n", <size_t>np.PyArray_DATA(a))
# These cause errors if the array is not casted because a[0] and a.data return # python objects: # cdef size_t broken_ptr1 = &a[0] # cdef size_t broken_ptr2 = a.data
cdef np.ndarray[long, ndim=1, mode="c"] x = a
# After casting, everything works and is consistent. cdef size_t buf_ptr1 = <size_t> np.PyArray_DATA(x) cdef size_t buf_ptr2 = <size_t> &x[0] cdef size_t buf_ptr3 = <size_t> x.data
# Output for an example run-- all addresses are the same, which is what # we want: # # 27354544 # 27354544 27354544 27354544
Not having to cast the array before extracting the buffer is useful if one knows the size of the elements one is dealing with, but not necessarily the type. In particular, I think it's useful if one wants to manipulate numpy arrays of strings, which Cython doesn't support-- though I'm not sure about this.
So I think Sturla's solution is what we should use in the future. It has the same semantics as .data, which I think a lot of people really like, and it won't be compromised if numpy changes.
Anyone have any other reasons why Sturla's solution might not be a good idea?
On Fri, Jul 27, 2012 at 10:45 AM, Nader Al-Naji <iamnotna...@gmail.com> wrote:
> <dtype_t*> np.PyArray_DATA(arr)
> is the right way to go. It is superior to the &a[0] solution because it
> doesn't require bounds checking, and it is superior to the .data solution
> 2) BONUS: it works even on
> arrays that haven't been statically casted to a Cython np.ndarray[...]. To
> fully understand what I mean by (2), take a look at the following example:
> a = np.arange(10, dtype=int)
> # This works even if a hasn't been casted yet.
> fprintf(stderr, "%lu %lu\n", <size_t>np.PyArray_DATA(a))
what happens if a is not a numpy array? If you create it in your
Cython code, you can be confident that it is, but if it's passed in,
who knows? and if you have to check, why not cast it?
> Not having to cast the array before extracting the buffer is useful if one
> knows the size of the elements one is dealing with, but not necessarily the
> type.
you can cast it to a ndarray with unspecified type -- I suspect arr[0]
will work there, too.
> So I think Sturla's solution is what we should use in the future. It has the
> same semantics as .data, which I think a lot of people really like, and it
> won't be compromised if numpy changes.
well, as Sturla points out, Cython essentially creates the same code
under the hood if you index the zeroth element anyway. My it's just
aesthetics, but I like the more pyhtonic looking: &arr[0]
-Chris
> Anyone have any other reasons why Sturla's solution might not be a good
> idea?
> Thanks,
> Nader
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
> 2) BONUS: it works even on
> arrays that haven't been statically casted to a Cython np.ndarray[...]. To
> fully understand what I mean by (2), take a look at the following example:
> a = np.arange(10, dtype=int)
> # This works even if a hasn't been casted yet.
> fprintf(stderr, "%lu %lu\n",<size_t>np.PyArray_DATA(a))
> what happens if a is not a numpy array?
np.PyArray_DATA is a C macro that does not check anything.
> well, as Sturla points out, Cython essentially creates the same code > under the hood if you index the zeroth element anyway. My it's just > aesthetics, but I like the more pyhtonic looking: &arr[0] -Chris
That's what Cython does, yes. You can see the code in Cython/Includes/numpy.pxd, i.e. the function __getbuffer__ in np.ndarray.
Two other benefits of &arr[0] are that the code will be easier to port to typed memoryviews and that it uses the same coding style as C++ code passing the buffer of a std::vector to C (or even Cython passing libcpp.vector.vector to C.) In all cases, &arr[0] is what we use. &arr[0] is also shorter to write than <dtype_t*>np.PyArray_DATA(arr). To get rid of the bounds checking (which hardly matters here), use @cython.boundscheck(False) and @cython.wraparound(False). Using &arr[0] will also provide better run-time error checking, such as ensuring the buffer is contiguous if that is required by the C code.
I do not prefer <dtype_t*>np.PyArray_DATA(arr) over &arr[0], but this is a matter of taste I guess. I just want to discourage "<dtype_t*>arr.data", which is soon due to break.
Sturla Molden <sturlamol...@yahoo.no> wrote:
>Den 27.07.2012 20:06, skrev Chris Barker:
>> 2) BONUS: it works even on
>> arrays that haven't been statically casted to a Cython
>np.ndarray[...]. To
>> fully understand what I mean by (2), take a look at the following
>example:
>> a = np.arange(10, dtype=int)
>> # This works even if a hasn't been casted yet.
>> fprintf(stderr, "%lu %lu\n",<size_t>np.PyArray_DATA(a))
>> what happens if a is not a numpy array?
>np.PyArray_DATA is a C macro that does not check anything.
>> well, as Sturla points out, Cython essentially creates the same code >> under the hood if you index the zeroth element anyway. My it's just >> aesthetics, but I like the more pyhtonic looking: &arr[0] -Chris
>That's what Cython does, yes. You can see the code in >Cython/Includes/numpy.pxd, i.e. the function __getbuffer__ in
>np.ndarray.
>Two other benefits of &arr[0] are that the code will be easier to port >to typed memoryviews and that it uses the same coding style as C++ code
>passing the buffer of a std::vector to C (or even Cython passing >libcpp.vector.vector to C.) In all cases, &arr[0] is what we use. >&arr[0] is also shorter to write than <dtype_t*>np.PyArray_DATA(arr).
>To >get rid of the bounds checking (which hardly matters here), use >@cython.boundscheck(False) and @cython.wraparound(False). Using
>&arr[0] >will also provide better run-time error checking, such as ensuring the >buffer is contiguous if that is required by the C code.
>I do not prefer <dtype_t*>np.PyArray_DATA(arr) over &arr[0], but this
>is >a matter of taste I guess. I just want to discourage >"<dtype_t*>arr.data", which is soon due to break.
(This isn't of consequence to the original poster and sort of goes off topic, I just thought Sturla may be interested:)
Actually there's a half-baked patch to turn arr.data into PyArray_DATA for np.ndarray. The lgoic is that arr.shape is so much used, and we don't want that to break, so we'll throw in some hacks to change that to using a macro, and then we may as well throw in 'data' too.
I think this is all pending a decision about how closely the new memoryviews should emulate the current ndarray/buffer syntax (i.e. can you access the underlying object of a memoryview transparently), and we don't seem to have the energy to discuss that yet.
Dag
>Sturla
-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
> Actually there's a half-baked patch to turn arr.data into PyArray_DATA for np.ndarray. The lgoic is that arr.shape is so much used, and we don't want that to break, so we'll throw in some hacks to change that to using a macro, and then we may as well throw in 'data' too.
That might prevent some code from breaking when NumPy completes the transition to PyArray_DATA...
> I think this is all pending a decision about how closely the new memoryviews should emulate the current ndarray/buffer syntax (i.e. can you access the underlying object of a memoryview transparently), and we don't seem to have the energy to discuss that yet.
I recently suggested using typed memoryviews in SciPy (scipy.spatial.cKDTree to begin with), but it got clubbed down for not being compatible with Python 2.4... We finally settled on using PyArray_DATA and plain old C style pointer arithmetics. But granted, the code would have looked much better with memoryviews and multidimensional arrays :(
On Fri, Jul 27, 2012 at 3:12 PM, Sturla Molden <sturlamol...@yahoo.no>wrote:
> I recently suggested using typed memoryviews in SciPy
> (scipy.spatial.cKDTree to begin with), but it got clubbed down for not
> being compatible with Python 2.4... We finally settled on using
> PyArray_DATA and plain old C style pointer arithmetics. But granted, the
> code would have looked much better with memoryviews and multidimensional
> arrays :(
> Sturla
I've been playing around with some similar code in scikit-learn which needs
to pass around views of arrays (the ball tree). Trying out some quick
benchmarks, it looks like memoryviews are pretty comparable to pointer
arithmetic for individual memory access, but lead to some significant
overhead when passing around slices as you'd need to in ckdtree. For my
application, I've settled on simply using pointers for the sake of speed.
I prepared some quick-and-dirty benchmarks of the behavior I need at
https://github.com/jakevdp/memview_benchmarks/ -- I'd be interested if
people more familiar with memory-views could take a look and let me know if
I'm missing anything there.
Jake
wrote:
> Ive been playing around with some similar code in scikit-learn which
> needs to pass around views of arrays (the ball tree). Trying out
> some quick benchmarks, it looks like memoryviews are pretty > comparable
> to pointer arithmetic for individual memory access, but lead to some
> significant overhead when passing around slices as youd need to in
> ckdtree. For my application, Ive settled on simply using pointers
> for the sake of speed.
> I prepared some quick-and-dirty benchmarks of the behavior I need at
> https://github.com/jakevdp/memview_benchmarks/ [2] -- Id be > interested
> if people more familiar with memory-views could take a look and let > me
> know if Im missing anything there.
Try to turn off bounds checking and wraparound for slice_func and see if this changes the timings.
On Fri, 27 Jul 2012 15:38:01 -0700, Jake Vanderplas <jake...@gmail.com>
wrote:
quick-and-dirty benchmarks of the behavior I need at
> https://github.com/jakevdp/memview_benchmarks/ [2] -- Id be > interested
> if people more familiar with memory-views could take a look and let > me
> know if Im missing anything there.
> Jake
You also forgot to declare M no.int_p in compute_distances, which makes it a Python object.
You might also want to turn on some compiler optimization in setup.py, e.g. -O2, as Cython often generates C code that needs the C compiler to optimize in order to be efficient.
>> Ive been playing around with some similar code in scikit-learn which
>> needs to pass around views of arrays (the ball tree). Trying out
>> some quick benchmarks, it looks like memoryviews are pretty comparable
>> to pointer arithmetic for individual memory access, but lead to some
>> significant overhead when passing around slices as youd need to in
>> ckdtree. For my application, Ive settled on simply using pointers
Indeed. You probably will also get much better performance by not
slicing, i.e. pass in the slices as 2D arrays and pass in the i and j
indices for the first dimension into the function, and perform full
indexing (an index for each dimension).
> On 28 July 2012 00:34, Sturla Molden <sturlamol...@yahoo.no> wrote:
>> On Fri, 27 Jul 2012 15:38:01 -0700, Jake Vanderplas <jake...@gmail.com>
>> wrote:
>>> Ive been playing around with some similar code in scikit-learn which
>>> needs to pass around views of arrays (the ball tree). Trying out
>>> some quick benchmarks, it looks like memoryviews are pretty comparable
>>> to pointer arithmetic for individual memory access, but lead to some
>>> significant overhead when passing around slices as youd need to in
>>> ckdtree. For my application, Ive settled on simply using pointers
> Indeed. You probably will also get much better performance by not
> slicing, i.e. pass in the slices as 2D arrays and pass in the i and j
> indices for the first dimension into the function, and perform full
> indexing (an index for each dimension).
(Basically slicing often involves function calls as well as
introducing a temporary memoryview slice, which is cleaned up for
every iteration (which happens in a thread-safe manner, using atomics
or locks)).
> I prepared some quick-and-dirty benchmarks of the behavior I need at > https://github.com/jakevdp/memview_benchmarks/ -- I'd be interested if > people more familiar with memory-views could take a look and let me > know if I'm missing anything there.
> Jake
I took the liberty to update your banchmarks (see attachment). For example I noticed that GCC was clever enough to optimize out all the loops in your pointer_arith.pyx...
Here are the timings I got from the updated version in the attachment. I think this gives the correct picture:
There is a table in the attached PDF that should be easier to read.
The overhead from the numpy versions comes from slicing the ndarray. In comparison, slicing the memoryview has a very small overhead. If we slice the ndarray in Cython, this is not much better than just using plain numpy in Python. But if we use memoryviews, slicing is just a little bit slower than using C style pointer arithmetics.
And consider this: Numerical code using array slicing in Fortran90 with gfortran is often 2x slower than the same code using pointer arithmetics in C with GCC. At least in my experience (Fortran 77 is another matter.)
If you wonder why using np.dot was faster than writing out the loop in Cython, that is due to Intel MKL in Enthought.
Conclusion:
Memoryviews are extremely fast, comparable to pointer arithmetics in C.
Now we need a real benchmark, e.g. some linear algebra solver or an FFT. Something like Scimark perhaps. Cython vs. C vs. Fortran 90.
I found another issue, the memoryview slices were not declared contiguous. This reduced the runtime from 1.86 to 1.83 seconds. That puts the overhead from using memoryview slices to 2.2% compared to raw C pointer arithmetics. The benchmark creates two million memoryview slices and computes one million dot products, each with vector lengths of 1000. I am more than willing to accept those 2.2 % to avoid those pesky pointers, but it remains to be seen how memoryviews perform on a more realistic problem.
>> I prepared some quick-and-dirty benchmarks of the behavior I need at >> https://github.com/jakevdp/memview_benchmarks/ -- I'd be interested >> if people more familiar with memory-views could take a look and let >> me know if I'm missing anything there.
>> Jake
> I took the liberty to update your banchmarks (see attachment). For > example I noticed that GCC was clever enough to optimize out all the > loops in your pointer_arith.pyx...
> Here are the timings I got from the updated version in the attachment. > I think this gives the correct picture:
> There is a table in the attached PDF that should be easier to read.
> The overhead from the numpy versions comes from slicing the ndarray. > In comparison, slicing the memoryview has a very small overhead. If we > slice the ndarray in Cython, this is not much better than just using > plain numpy in Python. But if we use memoryviews, slicing is just a > little bit slower than using C style pointer arithmetics.
> And consider this: Numerical code using array slicing in Fortran90 > with gfortran is often 2x slower than the same code using pointer > arithmetics in C with GCC. At least in my experience (Fortran 77 is > another matter.)
> If you wonder why using np.dot was faster than writing out the loop in > Cython, that is due to Intel MKL in Enthought.
> Conclusion:
> Memoryviews are extremely fast, comparable to pointer arithmetics in C.
> Now we need a real benchmark, e.g. some linear algebra solver or an > FFT. Something like Scimark perhaps. Cython vs. C vs. Fortran 90.
Sturla,
Thanks for looking at this. I'm still learning the details of optimizing
memviews - these are very impressive benchmarks! I've updated my github
repository with your changes:
https://github.com/jakevdp/memview_benchmarks Thanks
Jake
On Sat, Jul 28, 2012 at 8:18 AM, Sturla Molden <sturlamol...@yahoo.no>wrote:
> I found another issue, the memoryview slices were not declared contiguous.
> This reduced the runtime from 1.86 to 1.83 seconds. That puts the overhead
> from using memoryview slices to 2.2% compared to raw C pointer arithmetics.
> The benchmark creates two million memoryview slices and computes one
> million dot products, each with vector lengths of 1000. I am more than
> willing to accept those 2.2 % to avoid those pesky pointers, but it remains
> to be seen how memoryviews perform on a more realistic problem.
>> I took the liberty to update your banchmarks (see attachment). For
>> example I noticed that GCC was clever enough to optimize out all the loops
>> in your pointer_arith.pyx...
>> Here are the timings I got from the updated version in the attachment. I
>> think this gives the correct picture:
>> There is a table in the attached PDF that should be easier to read.
>> The overhead from the numpy versions comes from slicing the ndarray. In
>> comparison, slicing the memoryview has a very small overhead. If we slice
>> the ndarray in Cython, this is not much better than just using plain numpy
>> in Python. But if we use memoryviews, slicing is just a little bit slower
>> than using C style pointer arithmetics.
>> And consider this: Numerical code using array slicing in Fortran90 with
>> gfortran is often 2x slower than the same code using pointer arithmetics in
>> C with GCC. At least in my experience (Fortran 77 is another matter.)
>> If you wonder why using np.dot was faster than writing out the loop in
>> Cython, that is due to Intel MKL in Enthought.
>> Conclusion:
>> Memoryviews are extremely fast, comparable to pointer arithmetics in C.
>> Now we need a real benchmark, e.g. some linear algebra solver or an FFT.
>> Something like Scimark perhaps. Cython vs. C vs. Fortran 90.
You got a pull request. I'm not sure if you already updated your code.
I'm very happy with the speed of memoryviews too, particularly slicing. The slowness of slicing np.ndarray was the reason I never could use Cython+NumPy instead of Fortran 95.
I now want to see a more realistic benchmark. I'm not sure if porting Scimark will be too much work. I want preferably to compare these on a set of real-world problems:
Python
Python with NumPy
C
C++ using STL
Fortran 77
Fortran 95
Cython with memoryviews
Java (perhaps)
C#.NET (perhaps)
MATLAB (perhaps)
> Sturla,
> Thanks for looking at this. I'm still learning the details of > optimizing memviews - these are very impressive benchmarks! I've > updated my github repository with your changes:
> https://github.com/jakevdp/memview_benchmarks > Thanks
> Jake
> On Sat, Jul 28, 2012 at 8:18 AM, Sturla Molden <sturlamol...@yahoo.no > <mailto:sturlamol...@yahoo.no>> wrote:
> I found another issue, the memoryview slices were not declared
> contiguous. This reduced the runtime from 1.86 to 1.83 seconds.
> That puts the overhead from using memoryview slices to 2.2%
> compared to raw C pointer arithmetics. The benchmark creates two
> million memoryview slices and computes one million dot products,
> each with vector lengths of 1000. I am more than willing to accept
> those 2.2 % to avoid those pesky pointers, but it remains to be
> seen how memoryviews perform on a more realistic problem.
> Sturla
> Den 28.07.2012 16:40, skrev Sturla Molden:
> I prepared some quick-and-dirty benchmarks of the behavior
> I need at https://github.com/jakevdp/memview_benchmarks/ > -- I'd be interested if people more familiar with
> memory-views could take a look and let me know if I'm
> missing anything there.
> Jake
> I took the liberty to update your banchmarks (see attachment).
> For example I noticed that GCC was clever enough to optimize
> out all the loops in your pointer_arith.pyx...
> Here are the timings I got from the updated version in the
> attachment. I think this gives the correct picture:
> There is a table in the attached PDF that should be easier to
> read.
> The overhead from the numpy versions comes from slicing the
> ndarray. In comparison, slicing the memoryview has a very
> small overhead. If we slice the ndarray in Cython, this is not
> much better than just using plain numpy in Python. But if we
> use memoryviews, slicing is just a little bit slower than
> using C style pointer arithmetics.
> And consider this: Numerical code using array slicing in
> Fortran90 with gfortran is often 2x slower than the same code
> using pointer arithmetics in C with GCC. At least in my
> experience (Fortran 77 is another matter.)
> If you wonder why using np.dot was faster than writing out the
> loop in Cython, that is due to Intel MKL in Enthought.
> Conclusion:
> Memoryviews are extremely fast, comparable to pointer
> arithmetics in C.
> Now we need a real benchmark, e.g. some linear algebra solver
> or an FFT. Something like Scimark perhaps. Cython vs. C vs.
> Fortran 90.
On Friday, July 27, 2012 2:42:50 PM UTC-4, sturlamolden wrote:
> Den 27.07.2012 20:06, skrev Chris Barker: > > 2) BONUS: it works even on > > arrays that haven't been statically casted to a Cython np.ndarray[...]. > To > > fully understand what I mean by (2), take a look at the following > example:
> > a = np.arange(10, dtype=int)
> > # This works even if a hasn't been casted yet. > > fprintf(stderr, "%lu %lu\n",<size_t>np.PyArray_DATA(a)) > > what happens if a is not a numpy array?
> np.PyArray_DATA is a C macro that does not check anything.
> > well, as Sturla points out, Cython essentially creates the same code > > under the hood if you index the zeroth element anyway. My it's just > > aesthetics, but I like the more pyhtonic looking: &arr[0] -Chris
> That's what Cython does, yes. You can see the code in > Cython/Includes/numpy.pxd, i.e. the function __getbuffer__ in np.ndarray.
> Two other benefits of &arr[0] are that the code will be easier to port > to typed memoryviews and that it uses the same coding style as C++ code > passing the buffer of a std::vector to C (or even Cython passing > libcpp.vector.vector to C.) In all cases, &arr[0] is what we use. > &arr[0] is also shorter to write than <dtype_t*>np.PyArray_DATA(arr). To > get rid of the bounds checking (which hardly matters here), use > @cython.boundscheck(False) and @cython.wraparound(False). Using &arr[0] > will also provide better run-time error checking, such as ensuring the > buffer is contiguous if that is required by the C code.
> I do not prefer <dtype_t*>np.PyArray_DATA(arr) over &arr[0], but this is > a matter of taste I guess. I just want to discourage > "<dtype_t*>arr.data", which is soon due to break.
> Sturla
It seems like there are shortcomings to all the major approaches:
- arr.data - good: intent is clear (but the semantics don't actually match) - bad: will break at some point, doesn't handle offsets properly - &arr[0] - good: intent is mostly clear, follows a common C++ idiom - bad: requires completely disabling bounds checking for the entire function to handle 0-length arrays - PyArray_DATA(arr) - good: has proper semantics - bad: verbose and unpythonic
Would it make sense to enhance numpy.pxd to provide a new property that acts like PyArray_DATA(arr) but looks pythonic (we might call it zeroth_elem, first_elem, or something like that)?
Can somebody clarify what is meant by the semantics not matching? I apologize if this was covered earlier in the thread... I must have missed it.
Also, in Numpy 1.6, arr.data and PyArray_DATA are equivalent (up to a cast to the data type): #define PyArray_DATA(obj) ((void *)(((PyArrayObject *)(obj))->data))
I think the best approach is for Cython to just translate arr.data into PyArray_DATA(arr) in the generated C code.
> It seems like there are shortcomings to all the major approaches:
> - arr.data > - good: intent is clear (but the semantics don't actually match) > - bad: will break at some point, doesn't handle offsets properly > - &arr[0] > - good: intent is mostly clear, follows a common C++ idiom > - bad: requires completely disabling bounds checking for the entire > function to handle 0-length arrays > - PyArray_DATA(arr) > - good: has proper semantics > - bad: verbose and unpythonic
> Would it make sense to enhance numpy.pxd to provide a new property that > acts like PyArray_DATA(arr) but looks pythonic (we might call it > zeroth_elem, first_elem, or something like that)?
On Mon, Jul 30, 2012 at 11:33 AM, Bradley Froehle <brad.froe...@gmail.com>wrote:
> Can somebody clarify what is meant by the semantics not matching? I
> apologize if this was covered earlier in the thread... I must have missed
> it.
By mismatched semantics, I just meant that apparently arr.data doesn't
always refer to the first element of the array. Sometimes arr.data !=
&arr[0].
> Also, in Numpy 1.6, arr.data and PyArray_DATA are equivalent (up to a cast
> to the data type):
> #define PyArray_DATA(obj) ((void *)(((PyArrayObject *)(obj))->data))
> I think the best approach is for Cython to just translate arr.data into
> PyArray_DATA(arr) in the generated C code.
> Also, in Numpy 1.6, arr.data and PyArray_DATA are equivalent (up to a
> cast to the data type):
> #define PyArray_DATA(obj) ((void *)(((PyArrayObject *)(obj))->data))
That is due to change.
> I think the best approach is for Cython to just translate arr.data into
> PyArray_DATA(arr) in the generated C code.
> It seems like there are shortcomings to all the major approaches:
> * arr.data
> o good: intent is clear (but the semantics don't actually match)
> o bad: will break at some point, doesn't handle offsets properly
> * &arr[0]
> o good: intent is mostly clear, follows a common C++ idiom
> o bad: requires completely disabling bounds checking for the
> entire function to handle 0-length arrays
No, it does not require bounds checking to be turned off. But it will do a bounds check if you don't.
> * PyArray_DATA(arr)
> o good: has proper semantics
> o bad: verbose and unpythonic
> Would it make sense to enhance numpy.pxd to provide a new property
> that acts like PyArray_DATA(arr) but looks pythonic (we might call
> it zeroth_elem, first_elem, or something like that)?
In my opinion we should use memoryviews instead, unless compatibility with Python 2.4 is absolutely required. We are discussing how to use an older, half-broken (useless for anything but trivial code), and soon deprecated interface to solve a problem that memoryviews do right.
o Memoryviews are much faster than np.ndarray and expose all the PEP 3118 attributes. Particularly slicing and using them as function arguments are faster (like Fortran arrays). You will notice the difference if your code has array slicing or function calls in it. np.ndarray is only fast and useful if you can inline everything into one big function, and you never generate an array slice. So for everything except trivial use-cases, typed memoryviews is what we should use.
o Cython arrays (cython.view.array) have an attribute .data which will do what you want. They behave like NumPy arrays except they are much faster.
o Memoryviews have a shorter syntax.
o NumPy arrays can be converted to memoryviews and Cython arrays automatically.
o This will take build and runtime dependency on NumPy away. And yet we can use as much of NumPy's Python API as we want.
o np.ndarray might even be deprecated when Cython reach 1.0, so using it should be discouraged. I'd even go so far as to suggest all references to suggest all instructions on using np.ndarray removed from the Wiki and replaced with memoryviews.
o Memoryviews works with all PEP 3118 compliant buffers, including NumPy.
>> o good: intent is mostly clear, follows a common C++ idiom
>> o bad: requires completely disabling bounds checking for the
>> entire function to handle 0-length arrays
> No, it does not require bounds checking to be turned off. But it will do
> a bounds check if you don't.
Sorry, let me rephrase this:
Bounds checking will prevent you from passing a dangling data pointer to C if you have a 0-length array. I.e. you get an informative Python exception (IndexError) with a traceback instead of a segfault -- which can be nice I guess.
