ANN: mpi4py_map brings a parallel map() implementation to mpi4py
Thomas Wiecki
About
MPI (Message Passing Interface) is a library that allows processes
running on different hosts to communicate and exchange information. It
is ideally suited for parallel computations on a cluster. mpi4py
provides a convenient python wrapper for this library. However, it
still a pretty bare exposition of MPI functions which requires you to
code most of the interprocess-communication by hand. Other parallel
python libraries such as the multiprocessing package (which only
allows process management on a single host) have more simple
interfaces. For example, you can create a worker pool and use the
map() function to easily parallelize evaluations of your function over
a given sequence (e.g. pool.map(lambda x: x**2, range(500)) will
square the list in parallel).
mpi4py_map brings this simple but powerful map() functionality to
mpi4py. It takes care of assigning the jobs to the workers, queueing
if all workers are busy and cleanly shutting down all workers after
the job is complete.