ImportError: no module named mpi4py

[Ahmad Ridwan]

hello everyone, i am new in this project.
I created a cluster system using ubuntu 14.04 with 2 nodes in a virtual machine. I took the example of the youtube channel 'eugene' but it only came to the installation of mpich2. then I took an example from the link https://technicalustad.com/build-your-own-super-computer-with-raspberry-pi-3-cluster/ for mpi4py installation.
when I type the command "mpiexec -n 5 python demo / helloworld.py", the result runs correctly in the master node.
But when I type the command for the cluster that is "mpiexec -f machinefile -n 4 python /mirror/mpi4py-2.0.0/demo/helloworld.py", it generates an error
"from mpi4py import MPI"
"importError: no module named mpi4py"

sorry for my english.
thank a lot.

[Lisandro Dalcin]

It seems that somehow mpi4py is not properly installed in the compute
nodes. Or maybe PYTHONPATH is not set correctly or not propagated to
the compute nodes, maybe you can try:

mpiexec -f machinefile -genvlist PYTHONPATH -n 4 python
/mirror/mpi4py-2.0.0/demo/helloworld.py

--
Lisandro Dalcin
============
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Extreme Computing Research Center (ECRC)
King Abdullah University of Science and Technology (KAUST)
http://ecrc.kaust.edu.sa/

4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 0109
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa

Office Phone: +966 12 808-0459

[Ahmad Ridwan]

the command does not work.
I may have to repeat everything from scratch.
where can I get complete tutorial from installation to test that really work?

Im research student from departement computer engineering, Universitas Sriwijaya, Palembang, Indonesia.

[Ahmad Ridwan]

Maybe I have another question. I followed the tutorial at https://help.ubuntu.com/community/MpichCluster and got to the test stage using C program.

In which directory should I install mpi4py? is it in the synced mirror folder to the slave node?
Thank you very much.

[Lisandro Dalcin]

On 12 April 2018 at 13:29, Ahmad Ridwan <ahmadrid...@gmail.com> wrote:
> Maybe I have another question. I followed the tutorial at https://help.ubuntu.com/community/MpichCluster and got to the test stage using C program.
>
> In which directory should I install mpi4py? is it in the synced mirror folder to the slave node?

1) You should install mpi4py in a shared folder that all compute nodes
have access to, that means you should install it what you called
"synced mirror folder".

2) Python has to found mpi4py in that special place, so if you
installed mpi4py in prefix "/path/to/synced/folder" (that is, mpi4py
stuff is inside "/path/to/synced/folder/mpi4py"), then you have to
manage to set the environment variable

export PYTHONPATH=/path/to/synced/folder:$PYTHONPATH

in EVERY compute node (how to do this depends on how other things are setup).

Alternatively, start your Python script with this like:

import sys
sys.path.append("/path/to/synced/folder")
from mpi4py import MPI

[Hossein Talebi]

I would like to ask another question related to this. In case the code uses mpi4py and the user runs in a system with no MPI and mpi4py installed, is Stub functions available somewhere?  This way the output of the functions will be when there is 1 process.
This is the case for C and Fortran for MPI.

[Lisandro Dalcin]

On 12 April 2018 at 14:40, Hossein Talebi <talebi....@gmail.com> wrote:
>
> I would like to ask another question related to this. In case the code uses
> mpi4py and the user runs in a system with no MPI and mpi4py installed, is
> Stub functions available somewhere? This way the output of the functions
> will be when there is 1 process.

No, stubs functions for the no-MPI-available case are not available.
And before you ask, I'm not interested at all in adding support for
that in mpi4py.

> This is the case for C and Fortran for MPI.
>

Is that a question or an assertion? Do you know any C stub functions
that REALLY work? For example, MPI allows send-to-self.

> --
> You received this message because you are subscribed to the Google Groups
> "mpi4py" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mpi4py+un...@googlegroups.com.
> To post to this group, send email to mpi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/mpi4py.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mpi4py/8f406126-db36-4774-95aa-cd987ebbf18b%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Lisandro Dalcin]

On 12 April 2018 at 15:20, Lisandro Dalcin <dal...@gmail.com> wrote:
> On 12 April 2018 at 14:40, Hossein Talebi <talebi....@gmail.com> wrote:
>>
>> I would like to ask another question related to this. In case the code uses
>> mpi4py and the user runs in a system with no MPI and mpi4py installed, is
>> Stub functions available somewhere? This way the output of the functions
>> will be when there is 1 process.
>
> No, stubs functions for the no-MPI-available case are not available.
> And before you ask, I'm not interested at all in adding support for
> that in mpi4py.
>
>> This is the case for C and Fortran for MPI.
>>
>
> Is that a question or an assertion? Do you know any C stub functions
> that REALLY work? For example, MPI allows send-to-self.
>

For example, PETSc do have a rather good set of stubs functions, but
PETSc is coded with enough care as to never send-to-self. So these
stubs are useful in the context of PETSc, but they are not really a
truly working sequential-MPI implementation.

[talebi....@gmail.com]

I see. It was an assertion. For both C and Fortran I am using the MPI Stubs for years. LAMMPS is also another code beside PETSc.


I will not ask why, but I explain my problem. mpi4py needs the MPI library installed even when running serial (right?). In Linux it is no problem. But in Windows for many users it is a headache to install a MPI library. In my special situation packaging the code and supporting MPI for all users is a big problem.

I can try to open a case for every MPI call I make for the serial mode, or write the Stubs myself.

[Ahmad Ridwan]

thanks. finally i can do it.

I have another question, how do I measure program execution time for this cluster system?

my task here is to compare the speed and efficiency of program execution on n nodes.

thank you very much, this forum is very helpful.

[Lisandro Dalcin]

On 12 April 2018 at 15:31, <talebi....@gmail.com> wrote:
>
>
> I see. It was an assertion. For both C and Fortran I am using the MPI Stubs
> for years. LAMMPS is also another code beside PETSc.
>

Are you talking about this MPI stubs?
http://people.sc.fsu.edu/~jburkardt/c_src/mpi_stubs/mpi_stubs.html


Or maybe something else? Could you provide a link?

>
> I will not ask why, but I explain my problem. mpi4py needs the MPI library
> installed even when running serial (right?).

Yes, of course.

> In Linux it is no problem. But
> in Windows for many users it is a headache to install a MPI library.

Why? Windows users can just use the Microsoft MPI implementation! As
far as I can tell, it is a free download. Even if running on a desktop
and not a cluster, you can benefit from multiple processor cores
through MPI.

> In my
> special situation packaging the code and supporting MPI for all users is a
> big problem.
>

> I can try to open a case for every MPI call I make for the serial mode, or
> write the Stubs myself.
>

If you build your own (or third-party) MPI stub library, then mpi4py
should build just fine, setup.py will check for availability of EVERY
symbol in MPI and should endup linking just fine. At runtime, any
missing MPI function in the stubs will raise NotImplementedError if
called.

[mahf...@gmail.com]

pip install mpi4py