combining multiprocessing and MPIMigration

[Santosh Suram]

I am running pyevolve0.6rc1 version on a cluster with python2.7.

I am using genetic programming and in order to speed up my evaluation I have been using multiprocessing on cores within a compute node and it has worked very well for me.

I have also used MPIMigration for generating multiple GP evolutions that exchange individuals at regular intervals and this has also worked very well.

I have tried to combine both the multiprocessing and MPIMigration by generating one MPI task per node and allowing these tasks to participate in MPIMigration, whereas, within a node I attempted to use multiprocessing for faster evaluations.

Unfortunately, this has resulted in the script hanging up at the multiprocessing evolution step after an MPIMigration occurs. In addition, if I set the migration rate (generations interval at which MPIMigration is invoked) =mr; the hanging up problem occurs at i*mr generations and not necessarily at the same i each time.

Can anyone shed some light on how to proceed?

My background is not computer science but am curious to get better at parallel programming in python and would appreciate any answer that helps me understand the specifics of what is going on.

I tried going through the multiprocessing module but just could not find a place to start digging this problem.

[Ben Elliston]

Hi Santosh

I use this approach myself and it works fine for me. I have a number of
16 core machines and a larger number of 4 core machines in my cluster.
I tell Pyevolve to use 4 CPUs per run and then distribute the work with
MPI using four slots on the 16 core machines and one slot on the 4 core
machines. Perhaps drop me a reply in private email and we can thrash
out the problem? There are some things we can do with OpenMPI to
troubleshoot it.

Cheers, Ben