mpi4py vs multiprocessing

[lty_...@126.com]

I'm trying to convert my 3d model data from .dae to .gltf,  and my teacher want me to make it faster, and advise me to try parallel processing.I found 2 ways to solve the problem, the multiprocessing module and the mpi4py module.I tried both of them and the time is similar,but I think the mpi4py should be better. My computer is Dell precision,8-core cpu. And I tried a 1.2G dataset. So what is the reason do you think? Should mpi4py be faster than multiprocessing? or mpi4py can be faster in clusters?

[Lisandro Dalcin]

On 26 October 2016 at 04:08, <lty_...@126.com> wrote:

I'm trying to convert my 3d model data from .dae to .gltf,  and my teacher want me to make it faster, and advise me to try parallel processing.I found 2 ways to solve the problem, the multiprocessing module and the mpi4py module.I tried both of them and the time is similar,but I think the mpi4py should be better. My computer is Dell precision,8-core cpu. And I tried a 1.2G dataset. So what is the reason do you think? Should mpi4py be faster than multiprocessing?

That is expected, you should not expect too much differences. 

 

or mpi4py can be faster in clusters?

It is not really about being faster, it is about actually using these packages in clusters. It is trivial to use mpi4py in a cluster. I don't know how to make multiprocessing work across compute nodes. Do you?

PS: Take a look at Python 3 concurrent.futures package (Python 2 backport available on PyPI, just "pip install futures") and the new mpi4py.futures implementation in mpi4py git repo. These are much nicer APIs than using raw multiprocessing (or MPI) for the kind of tasks you are doing.

--

Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459