The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Best Practices for passing numpy data pointer to C ?
 There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic. There was an error processing your request. Please try again. Standard view   View as tree
 Messages 51 - 59 of 59 < Older

From:
To:
Cc:
Followup To:
Subject:
 Validation: For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon.

More options Jul 30 2012, 4:37 pm
From: Sturla Molden <sturlamol...@yahoo.no>
Date: Mon, 30 Jul 2012 22:37:12 +0200
Local: Mon, Jul 30 2012 4:37 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?
Den 30.07.2012 19:16, skrev Sturla Molden:

Here is an example of using memoryviews:

The extra overhead here is just 2.2% compared to plain C pointer
arithmetics (using -O2) when a 1000 x 1000 array is passed to "compute
distances".

As we discussed in the same thread, doing this with np.ndarray is just
too slow:

Anything but trivial functions will need slicing and function calls. You
can see here how memoryviews allows us to code in "Fortran style" with
arrays instead of using C style pointer arithmetics.

Pointer arithmetics is still not too bad in this simple case, but it can
be much more complicated (but why bother?):

So I suggest one of these idioms for calling C from Cython:

cdef double[:,::1] array = numpy_array
cdef Py_ssize_t m, n
m = array.shape[0]
n = array.shape[1]
foobar(m, n, &array[0,0])

cdef view.array array = <double[:,::1]> numpy_array
cdef Py_ssize_t m, n
m = array.shape[0]
n = array.shape[1]
foobar(m, n, <double*> array.data)

Sturla

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 6 2012, 1:28 pm
From: Chris Barker <chris.bar...@noaa.gov>
Date: Mon, 6 Aug 2012 10:28:10 -0700
Local: Mon, Aug 6 2012 1:28 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?

On Mon, Jul 30, 2012 at 1:37 PM, Sturla Molden <sturlamol...@yahoo.no> wrote:
> So I suggest one of these idioms for calling C from Cython:
> cdef double[:,::1] array = numpy_array
> cdef Py_ssize_t m, n
> m = array.shape[0]
> n = array.shape[1]
> foobar(m, n, &array[0,0])

> cdef view.array array = <double[:,::1]> numpy_array
> cdef Py_ssize_t m, n
> m = array.shape[0]
> n = array.shape[1]
> foobar(m, n, <double*> array.data)

The only difference is that first line: will they generate the same C
code?  -- why would one choose on or the other?

(there should be one obvious way to do it...)

I'll probably go update the wiki page, but to make sure I'm clear:

Using a view.array will build a Cython "memoryview", using the
extended buffer interface -- so you can pass in any Python object that
supports PEP 3118 buffers (does  anything other than numpy suport that
now??)

This is a lightweight way to both get a pointer to the data, and to
have a way, in Cython, to slice and dice the data.

Will this check for C-contiguous and raise an error if not?

-Chris

> Sturla

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Bar...@noaa.gov

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 6 2012, 5:12 pm
From: mark florisson <markflorisso...@gmail.com>
Date: Mon, 6 Aug 2012 22:12:01 +0100
Local: Mon, Aug 6 2012 5:12 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?
On 6 August 2012 18:28, Chris Barker <chris.bar...@noaa.gov> wrote:

Not entirely. A cython.view.array is an array itself, that has memory
itself. In other words, it's a (lousy) replacement for numpy. It's
useful for users when they have memory allocated from somewhere and
want to turn it into a numpy array or memoryview. The array is more of
an intermediate thing, usually created implicitly by casting a
pointer: <double[:10:1, :10]> p creates a Fortran contiguous cython
array view on the memory of shape (10, 10).

> This is a lightweight way to both get a pointer to the data, and to
> have a way, in Cython, to slice and dice the data.

> Will this check for C-contiguous and raise an error if not?

Yes, buffers are always validated when they are assigned to by objects
at runtime. Assigning memoryviews to each other with wrong types is an
error, e.g.

cdef double[:, ::1] array = numpy_array # runtime check
cdef double[:, ::1] array2 = array # compile-time check

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 6 2012, 8:03 pm
From: Sturla Molden <sturlamol...@yahoo.no>
Date: Tue, 07 Aug 2012 02:03:21 +0200
Local: Mon, Aug 6 2012 8:03 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?
Den 06.08.2012 19:28, skrev Chris Barker:

>> cdef double[:,::1] array = numpy_array
>> cdef Py_ssize_t m, n
>> m = array.shape[0]
>> n = array.shape[1]
>> foobar(m, n, &array[0,0])

>> cdef view.array array = <double[:,::1]> numpy_array
>> cdef Py_ssize_t m, n
>> m = array.shape[0]
>> n = array.shape[1]
>> foobar(m, n, <double*> array.data)

> The only difference is that first line: will they generate the same C
> code?  -- why would one choose on or the other?

The last line is also different ;-)

The first creates a typed memoryview of the NumPy array. That is, using
the NumPy array as an PEP 3118 buffer directly from Cython. No auxillary
Python object is created.

The second created an auxillary Cython array that shares memory with the
NumPy array. The Cython array is a Python object with a PEP 3118 buffer.
Unlike np.ndarray, it has a valid .data attribute that will not break.

If you dislike &array[0,0] syntax, pick the latter. If you don't want an
auxillary Python object, pick the former.

> (there should be one obvious way to do it...)

> I'll probably go update the wiki page, but to make sure I'm clear:

> Using a view.array will build a Cython "memoryview", using the
> extended buffer interface -- so you can pass in any Python object that
> supports PEP 3118 buffers (does  anything other than numpy suport that
> now??)

Typed memoryviews will give you convinient and fast data access and fast
array slicing in Cython. cython.view.array can e.g. be used if you need
to send a C pointer as a Python pep 3118 object to some Python routine.

Thus they are not exactly the same, but "almost".

Typical use cases:

double[:,:] or double[:,::1]:
- You have a C pointer, you want fast "NumPy style" indexing and slicing.
- You have a NumPy array, you want fast indexing and fast slicing.
- You are considering to use Fortran because of its array syntax.

double[::1,:]:
- You need to interface with Fortran code.
- You are porting old Fortran code line-by-line.
- You are working with OpenGL.

cython.view.array:
- You have a C pointer and need a Python PEP 3118 object.
- You want a safe replacement for malloc (one that is faster than NumPy).
- You are porting old NumPy/Cython code that uses the .data attribute everywhere.

Also beware of this:

Common contra-indications for using double[::1,:] orcython.view.array:
- You need to interface with BLAS. Use cblas instead.
- You need to interface with LAPACK. Use LAPACKE instead.
- You need to allocate WORK arrays for LAPACK. Use LAPACKE instead.

Sturla

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 7 2012, 3:01 pm
From: Chris Barker <chris.bar...@noaa.gov>
Date: Tue, 7 Aug 2012 12:01:55 -0700
Local: Tues, Aug 7 2012 3:01 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?
On Mon, Aug 6, 2012 at 5:03 PM, Sturla Molden

>>> cdef double[:,::1] array = numpy_array
>>> cdef Py_ssize_t m, n
>>> m = array.shape[0]
>>> n = array.shape[1]
>>> foobar(m, n, &array[0,0])

>>> cdef view.array array = <double[:,::1]> numpy_array
>>> cdef Py_ssize_t m, n
>>> m = array.shape[0]
>>> n = array.shape[1]
>>> foobar(m, n, <double*> array.data)
> The first creates a typed memoryview of the NumPy array. That is, using the
> NumPy array as an PEP 3118 buffer directly from Cython. No auxillary Python
> object is created.

OK -- the original start of this post was how best to essentially pass
though a numpy data buffer to C/C++ code (or Fortran, I suppose).

So it looks like:

cdef double[:,::1] array = numpy_array

is very lightweight, so may be a good way to go, and would support
other PEP 3118 buffer objects, so seems a good idea.

Should I change the recommendation in the Wiki?

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Bar...@noaa.gov

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 11 2012, 4:12 am
From: Prashant <animator...@gmail.com>
Date: Sat, 11 Aug 2012 01:12:08 -0700 (PDT)
Local: Sat, Aug 11 2012 4:12 am
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?

Hi,

Python 2.6.6
GCC 3.4.5
Cython 0.15.1
XP 32

I tried building the extension using code but it seems multiply.pyx is
having some missing code.

@cython.boundscheck(False)
^
multiply.pyx:18:0: undeclared name not builtin: cython

This is because you are not importing cython. After adding "*cimport cython*",
next error is in generating *"multiply.c*" file.

multiply.o:multiply.c:(.text+0x5c3): undefined reference to
`_imp__c_multiply'

Any pointers? I am not much into cython lust started digging.

Regards

Prashant

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 20 2012, 7:02 pm
From: Chris Barker <chris.bar...@noaa.gov>
Date: Mon, 20 Aug 2012 16:02:09 -0700
Local: Mon, Aug 20 2012 7:02 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?

On Sat, Aug 11, 2012 at 1:12 AM, Prashant <animator...@gmail.com> wrote:
> Python 2.6.6
> GCC 3.4.5
> Cython 0.15.1

I'm running Cython 0.16 -- that may  be the issue

> XP 32

I haven't tested this code with gcc on Windows -- are you successfully
compiling other stuff?

> http://wiki.cython.org/tutorials/NumpyPointerToC
> multiply.pyx:18:0: undeclared name not builtin: cython

> This is because you are not importing cython. After adding "cimport cython",

sorry about that -- I"ve updated the wiki -- I thought I cut&pasted
from a working version...

> next error is in generating "multiply.c" file.

> multiply.o:multiply.c:(.text+0x5c3): undefined reference to
> `_imp__c_multiply'

> Any pointers? I am not much into cython lust started digging.

try the latest Cython -- then report back.

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Bar...@noaa.gov

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 25 2012, 1:26 am
From: Prashant <animator...@gmail.com>
Date: Fri, 24 Aug 2012 22:26:56 -0700 (PDT)
Local: Sat, Aug 25 2012 1:26 am
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?

This is with cython 0.16

running build_ext
cythoning multiply.pyx to multiply.c
building 'multiply' extension
D:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall
-IC:\Python26\lib\site-packages\numpy\core\include -IC:\Python26\include
-IC:\Python26\PC -c multiply.c -o build\temp.win32-2.6\Release\multiply.o
multiply.c: In function `__Pyx_GetBuffer':
multiply.c:4353: warning: unused variable `getbuffer_cobj'
multiply.c: In function `__Pyx_ReleaseBuffer':
multiply.c:4393: warning: unused variable `releasebuffer_cobj'
multiply.c: At top level:
C:/Python26/lib/site-packages/numpy/core/include/numpy/__multiarray_api.h:1 533:
warning: '_import_array' defined but not used
C:/Python26/lib/site-packages/numpy/core/include/numpy/__ufunc_api.h:227:
warning: '_import_umath' defined but not used
D:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall
-IC:\Python26\lib\site-packages\numpy\core\include -IC:\Python26\include
-IC:\Python26\PC -c c_multiply.c -o
build\temp.win32-2.6\Release\c_multiply.o
c_multiply.c:22:2: warning: no newline at end of file
writing build\temp.win32-2.6\Release\multiply.def
D:\MinGW\bin\gcc.exe -mno-cygwin -shared -s
build\temp.win32-2.6\Release\multiply.o
build\temp.win32-2.6\Release\c_multiply.o
build\temp.win32-2.6\Release\multiply.def -LC:\Python26\libs
-LC:\Python26\PCbuild -lpython26 -lmsvcr90 -o
D:\Projects\cython\multiply.pyd
build\temp.win32-2.6\Release\multiply.o:multiply.c:(.text+0x6ff): undefined
reference to `_imp__c_multiply'
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

To post a message you must first join this group.
You do not have the permission required to post.
More options Aug 27 2012, 3:54 pm
From: Chris Barker <chris.bar...@noaa.gov>
Date: Mon, 27 Aug 2012 12:54:59 -0700
Local: Mon, Aug 27 2012 3:54 pm
Subject: Re: [cython-users] Re: Best Practices for passing numpy data pointer to C ?

Something weird is going on.

I've enclosed my copy of the code, sertup.py and test file, so we at
least make sure we're working with the same code.

Below is what I get on my Mac -- but it works OK for me on Windows, too:

Python 2.7 Cython 0.16, MSVC 2008 compiler

It looks like you're using MinGW, but I"d expect that to work -- what
else is different?

Python 3 maybe?

-Chris

## My results:

\$ python setup.py build_ext --inplace
running build_ext
cythoning multiply.pyx to multiply.c
building 'multiply' extension
creating build
creating build/temp.macosx-10.3-fat-2.7
gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -g -O2 -DNDEBUG
-g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-pack ages/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c multiply.c -o build/temp.macosx-10.3-fat-2.7/multiply.o
gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -arch ppc -arch i386 -g -O2 -DNDEBUG
-g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-pack ages/numpy/core/include
-I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c c_multiply.c -o build/temp.macosx-10.3-fat-2.7/c_multiply.o
gcc-4.0 -bundle -undefined dynamic_lookup -arch ppc -arch i386
-isysroot /Developer/SDKs/MacOSX10.4u.sdk -isysroot
/Developer/SDKs/MacOSX10.4u.sdk -g
build/temp.macosx-10.3-fat-2.7/multiply.o
build/temp.macosx-10.3-fat-2.7/c_multiply.o -o
/Users/chris.barker/HAZMAT/GNOME-dev/GNOME-GIT/experiments/Cython-numpy_tes t/multiply.so
orrw-lm-1689049:Cython-numpy_test chris.barker\$ py.test test_multiply.py
========================================================== test
session starts ===========================================================
platform darwin -- Python 2.7.3 -- pytest-2.2.3
collected 4 items

test_multiply.py ....

======================================================== 4 passed in
0.16 seconds ========================================================
orrw-lm-1689049:Cython-numpy_test chris.barker\$

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Bar...@noaa.gov