first time using cython to pass numpy array to C++ and return an iterator or new array to python

[Krishna Bhogaonker]

up votedown votefavorite

This is my first time really using Cython, so I was hoping someone could just validate my approach here. I am working on using a C++ computational topology library to calculate some topological summaries of a dataset of n-dimensional points. Here is the basic gist of what I am trying to do. I am trying to pass a Numpy array from python to a C++ computational topology library. I have a 2-d numpy array of float data that represents n-dimensional points. I have to pass the numpy array as a pointer or something to the C++ function. The C++ function converts the data to a std::vector<std::vector<float>>, processes the data and then returns either a C++ boost::transform_iterator or a std::iterator. I want to return either this iterator or some array based upon this iterator back to python. So once again, Numpy 2-d array --> C++ --> C++ 2-d iterator/array --> new python numpy array. Note that the returned information is an entirely new array or iterator, and not the original numpy array. I was reading over Kurt Smith's book on Cython, and just wanted to make sure I was doing this correctly.  So to pass the numpy array to C++ I could use a `typed memoryview.` That takes care of the first part. For the second part--returning a new array or iterator, I was not sure how to do this? Basically I have to pass some sort of pointer from C++ back to python? I was not sure how to do this second part of returning an array from C++ to python. Sorry I don't have any code yet. Since this is my first time really using Cython, I just wanted to make sure I was on the right track before writing a bunch of code on the wrong track. So once again, I was hoping someone could just indicate (1) whether this was the correct technical approach, meaning memoryview, and (2) indicate what the right way return an array from C++ to python is. Thanks. Krishna

[Chris Barker]

up votedown votefavorite

This is my first time really using Cython, so I was hoping someone could just validate my approach here.

Do look at the docs and Wiki for everything you can find about working with numpy arrays, but...

> So once again, Numpy 2-d array --> C++ --> C++ 2-d iterator/array --> new python numpy array. Note that the returned information is an entirely new array or iterator, and not the original numpy array.

numpy arrays are a wrapper around a "regular" C array -- that is a pointer to a block of memory with the numbers in it. If your C/C++ code works with such pointers (as Classic) C code generally does, then you can wrap and unwrap numpy arrays without data copying.

However, this is limited because you have to be very careful about what code is responseible for freeing the memory -- if you get that wrong you will either get crashes or memory leaks.

And, more to the point, C++ code may well work with the data structured in different ways, so you can't jsut pass a pointer.

So, for your "C++ 2-d iterator/array", unless you can confirm that the 2D array has the right layout in memory, you probably want to make a copy to makeit a numpy array.

in which case, you can create the numpy array in Cython the usual way:

arr = np.empty(shape, dtype)

and then loop through the C++ iterator to copy the values over. If you declare the array to be a ndarray in cython, then this should be pretty fast and efficient.

not as efficient as re-using the array without copying data -- but more robust and easier to do.

-CHB

 

I was reading over Kurt Smith's book on Cython, and just wanted to make sure I was doing this correctly.  So to pass the numpy array to C++ I could use a `typed memoryview.` That takes care of the first part. For the second part--returning a new array or iterator, I was not sure how to do this? Basically I have to pass some sort of pointer from C++ back to python? I was not sure how to do this second part of returning an array from C++ to python. Sorry I don't have any code yet. Since this is my first time really using Cython, I just wanted to make sure I was on the right track before writing a bunch of code on the wrong track. So once again, I was hoping someone could just indicate (1) whether this was the correct technical approach, meaning memoryview, and (2) indicate what the right way return an array from C++ to python is. Thanks. Krishna

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Krishna Bhogaonker]

Thanks so much Chris. I was kinda just overwhelmed with information, but you really helped me get a concrete plan together. I really appreciate that. Say, I was reading about writing "wrappers" on this group and saw some comments by you about writing thick wrappers in python. Do you know of any decent blog posts or guides on writing thick wrappers? I was hoping to find some sort of best practices guide or something. I can always write this stuff on my own, but it is always useful to hear about the challenges that others faced.

Krishna

On Tuesday, April 4, 2017 at 11:43:16 AM UTC-7, Chris Barker wrote:

up votedown votefavorite

This is my first time really using Cython, so I was hoping someone could just validate my approach here.

Do look at the docs and Wiki for everything you can find about working with numpy arrays, but...

> So once again, Numpy 2-d array --> C++ --> C++ 2-d iterator/array --> new python numpy array. Note that the returned information is an entirely new array or iterator, and not the original numpy array.

numpy arrays are a wrapper around a "regular" C array -- that is a pointer to a block of memory with the numbers in it. If your C/C++ code works with such pointers (as Classic) C code generally does, then you can wrap and unwrap numpy arrays without data copying.

However, this is limited because you have to be very careful about what code is responseible for freeing the memory -- if you get that wrong you will either get crashes or memory leaks.

And, more to the point, C++ code may well work with the data structured in different ways, so you can't jsut pass a pointer.

So, for your "C++ 2-d iterator/array", unless you can confirm that the 2D array has the right layout in memory, you probably want to make a copy to makeit a numpy array.

in which case, you can create the numpy array in Cython the usual way:

arr = np.empty(shape, dtype)

and then loop through the C++ iterator to copy the values over. If you declare the array to be a ndarray in cython, then this should be pretty fast and efficient.

not as efficient as re-using the array without copying data -- but more robust and easier to do.

-CHB

 

I was reading over Kurt Smith's book on Cython, and just wanted to make sure I was doing this correctly.  So to pass the numpy array to C++ I could use a `typed memoryview.` That takes care of the first part. For the second part--returning a new array or iterator, I was not sure how to do this? Basically I have to pass some sort of pointer from C++ back to python? I was not sure how to do this second part of returning an array from C++ to python. Sorry I don't have any code yet. Since this is my first time really using Cython, I just wanted to make sure I was on the right track before writing a bunch of code on the wrong track. So once again, I was hoping someone could just indicate (1) whether this was the correct technical approach, meaning memoryview, and (2) indicate what the right way return an array from C++ to python is. Thanks. Krishna

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Robert Bradshaw]

On Wed, Apr 5, 2017 at 7:22 AM, Krishna Bhogaonker <cyclo...@gmail.com> wrote:

Thanks so much Chris. I was kinda just overwhelmed with information, but you really helped me get a concrete plan together. I really appreciate that. Say, I was reading about writing "wrappers" on this group and saw some comments by you about writing thick wrappers in python. Do you know of any decent blog posts or guides on writing thick wrappers? I was hoping to find some sort of best practices guide or something. I can always write this stuff on my own, but it is always useful to hear about the challenges that others faced.

I'd say the key to writing "thick" wrappers is to sketch out what a Pythonic API would be of the given functionality (which may or may not follow what the C/C++ interface looks like), and then simply implement this API using the underlying library. 

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.

[Chris Barker]

On Wed, Apr 5, 2017 at 7:22 AM, Krishna Bhogaonker <cyclo...@gmail.com> wrote:

Thanks so much Chris. I was kinda just overwhelmed with information, but you really helped me get a concrete plan together. I really appreciate that. Say, I was reading about writing "wrappers" on this group and saw some comments by you about writing thick wrappers in python. Do you know of any decent blog posts or guides on writing thick wrappers?

I don't think I've even seen guidelines like that, no.

I'd look for examples. Here's one (one of mine :-) ):

https://github.com/NOAA-ORR-ERD/py_gd

it's a wrapper around the old and venerable libgd graphics lib. old-style C code, rather than modern C++, but maybe you'll get some ideas....

And shapely has a pretty nice API:

https://github.com/Toblerity/Shapely

-CHB

To unsubscribe from this group and stop receiving emails from it, send an email to cython-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.