ctypes version of pypar

[Tyler Wilson]

Good day,

I was looking for some work the elance.com site, and ran across what I
thought looked like an interesting project - porting pypar to work on
Windows 7 x64 with Python 2.7 x64. Though I put in a message to the
requester, it appears the project was given to somebody else or has
been closed. I had done some work on this, and so as not to waste it,
I thought I would put it out here.

My preferred approach to something like this is to keep it simple. So
instead of setting up a whole C/C++ toolchain to build the MPI
libraries and the mpiext.c/dll/so what I did was to use the existing
binary builds of OpenMPI and MPICH2 and then re-wrote the mpiext.c as
a pure Python mpiext.py.

I got as far as getting the test_pypar.py running successfully (minus
all the complex data types [the numpy ctypeslib module does not handle
this type properly]).

If there is any interest in this, I can post my version here or on
Google code, or I can put it inline here (the files option of Google
groups went away I think).

Let me know.

- Tyler

[Ole Nielsen]

Hi Tyler

Many thanks for your interest and contribution to pypar.

Although I don't use Windows myself, I know many are so anything that will make pypar easier to use in the Windows environment. Your idea of using Ctypes is great and I and very happy that you were able to verify that (most of) the test suite passes. I definitely think your project should be hosted here within the pypar site on Google code and will be happy to add you as project contributor. 

Can you provide me with your google id or whatever it takes to grant the necessary permissions?

Cheers

Ole

[Tyler W. Wilson]

Ole,

Thank you for the response. This list seems kind of quiet. Perhaps when we get the test suite working with this we can announce on the OpenMPI list too?

I am attaching the files I have to this email for your perusal. (After I am on the commiters list, I will send a proper diff). The Python code is based very closely on your C version, in hopes that it would be easier for you and others in the know to move around. Remember this is a pretty minimal version, implementing just enough to match the test_pypar module.

Also note this is designed to work on any platform that supports an MPI in shared library form. This includes Linux and OSX (I tested an earlier version on Ubuntu, and it appeared to work after installing the OpenMPI developer packages there).

Lastly, I would like to get an email over to the NumPy mailing list to ask about them potentially supporting the ctypeslib for a complex array typer. Would make this code as clean as it is now. Meanwhile, I need to figure out how to assemble/disassemble the complex arrays as float arrays like you did in the C version.

And I think this email - twi...@pulse-robotics.com - ought to work for the Google Code stuff.

Thank you,
Tyler

[Ole Nielsen]

Hi again

>Thank you for the response. This list seems kind of quiet.

Yeah, most activity was on the sourceforge list before we moved. Now, I guess, we are all just using pypar for our projects without many needs.

>Perhaps when we get the test suite working with this we can announce on the OpenMPI list too?

Yes please, that'd be great.

I am attaching the files I have to this email for your perusal. (After I am on the commiters list, I will send a proper diff).

Actually, would  you mind sending me diffs on pypar.py and mpiext.c?

 

The Python code is based very closely on your C version, in hopes that it would be easier for you and others in the know to move around. Remember this is a pretty minimal version, implementing just enough to match the test_pypar module.

That sounds good. Main thing for me is that all tests pass on the platforms we target. I

 

Also note this is designed to work on any platform that supports an MPI in shared library form. This includes Linux and OSX (I tested an earlier version on Ubuntu, and it appeared to work after installing the OpenMPI developer packages there).

Sounds good  - did the examples (in particular the mandelbrot demos) work?

When you have a release candidate, I'd be happy to verify that that tests and demos work on Linux. 

 

Lastly, I would like to get an email over to the NumPy mailing list to ask about them potentially supporting the ctypeslib for a complex array typer. Would make this code as clean as it is now.

Yes please - that is a good idea.

 

Meanwhile, I need to figure out how to assemble/disassemble the complex arrays as float arrays like you did in the C version.

That wasn't so bad (from memory)  as they are really just two float arrays.

 

And I think this email - twi...@pulse-robotics.com - ought to work for the Google Code stuff.

I have added you to the project (with commit rights).

Enjoy

Cheers

Ole

PS - I am insanely busy with a bunch of other things, so please forgive me if I don't get back to you quickly.

PPS - I'll be going to PyCon 2011 in Singapore next week. One of the pypar contributors Prabhu is giving a keynote.