Multiplie mpi instances of code

[RAJAT ARORA]

Hello all,

I am working on a 3D solid mechanics problem using deal.ii.

To run the code on n  processes, I used to run the command

mpirun -np <n> ./<code-name>

But, now, when I run it using this command, it runs n programs with n mpi processes each.

It is like I have executed mpirun -np <n> ./<code-name> command n times and the compiler is then running n^2 mpi processes.

I figured this out as printing from rank 0 is also printing n times.

I am unable to figure out anything. I did not install any new library.

Can anyone help me figuring out, what is wrong ?

Thanks a lot.

[Wolfgang Bangerth]

On 10/28/2016 09:34 AM, RAJAT ARORA wrote:
>
> To run the code on n processes, I used to run the command
>
> mpirun -np <n> ./<code-name>
>
> But, now, when I run it using this command, it runs n programs with n
> mpi processes each.
>
> It is like I have executed mpirun -np <n> ./<code-name> command n times
> and the compiler is then running n^2 mpi processes.
>
> I figured this out as printing from rank 0 is also printing n times.
>
> I am unable to figure out anything. I did not install any new library.
>
> Can anyone help me figuring out, what is wrong ?

I bet you are actually only running n copies of the program, each of
which has only one rank. What happens if you output from rank 1? Do you
get any output at all?

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[RAJAT ARORA]

Respected Prof. Bangerth,

Yes, you are right :).

When running with 4 mpi processes, all threads print their rank as 0.

Why is it broken all of a sudden. What is this problem called ? How do I correct it ?

[Wolfgang Bangerth]

On 10/28/2016 09:47 AM, RAJAT ARORA wrote:
>
> Yes, you are right :).
> When running with 4 mpi processes, all threads print their rank as 0.
>
> Why is it broken all of a sudden. What is this problem called ? How do I
> correct it ?

I don't know. Are you calling MPI_Init, for example?

[RAJAT ARORA]

Respected Prof. Bangerth,

I am just calling  dealii::Utilities::MPI::MPI_InitFinalize mpi_initialization(argc, argv, 1) once.

I am not making an explicit call to MPI_Init anywhere.

Should I reinstall my mpi library ? Does deal.ii will also need to be recompiled then ?

[Wolfgang Bangerth]

On 10/28/2016 09:55 AM, RAJAT ARORA wrote:
>
> I am just calling dealii::Utilities::MPI::MPI_InitFinalize
> mpi_initialization(argc, argv, 1) once.
> I am not making an explicit call to MPI_Init anywhere.

Yes, calling MPI_InitFinalize is enough.

> Should I reinstall my mpi library ? Does deal.ii will also need to be
> recompiled then ?

I don't know. You say that it suddenly stopped working. Did it work
correctly before? What has changed in the meantime?

Best

[Timo Heister]

>> Should I reinstall my mpi library ? Does deal.ii will also need to be
>> recompiled then ?
>
> I don't know. You say that it suddenly stopped working. Did it work
> correctly before? What has changed in the meantime?

This can happen if you have conflicting mpi libraries installed (for
example linking to mpich but running with mpirun coming from openmpi).
Make sure you only have one implementation installed and that all your
dependencies are configured with it. One common mistake is to install
petsc with --download-mpich for example.

--
Timo Heister
http://www.math.clemson.edu/~heister/

[RAJAT ARORA]

I am surprised why it is not working. I can't recall what has changed.

I don't remember installing any new libraries. It was working till Monday.

Also, I have installed petsc with --download mpich tag but donot have any other mpi installation. 

and more importantly, it was working until monday.

How should I know, if suddenly other mpi is detected and where is it installed ?

Thanks.

[RAJAT ARORA]

Hello Professors,

I just now found out something that Timo suggested.

I think that openmpi and Mpich are clashing now. (I didn't even know that both were installed upto now).

I think, if I can disable openmpi somehow, I will be in good shape. This is because, I compiled deal.ii with mpich library which was installed by petsc.

Can you help me how to disable openmpi ?

I ran the command

mpichversion

and got this output

MPICH Version:     3.1.3
MPICH Release date: Wed Oct  8 09:37:19 CDT 2014
MPICH Device:     ch3:sock
MPICH configure: --prefix=/home/rajat/Documents/Code-Libraries/petsc/petsc-3.6.4/arch-linux2-c-debug MAKE=/usr/bin/make --libdir=/home/rajat/Documents/Code-Libraries/petsc/petsc-3.6.4/arch-linux2-c-debug/lib CC=gcc CFLAGS=-fPIC -g3 -O0 AR=/usr/bin/ar ARFLAGS=cr CXX=g++ CXXFLAGS=-g -O0 -fPIC F77=gfortran FFLAGS=-fPIC -ffree-line-length-0 -g -O0 FC=gfortran FCFLAGS=-fPIC -ffree-line-length-0 -g -O0 --enable-shared --with-device=ch3:sock --with-pm=hydra --enable-g=meminit --enable-fast
MPICH CC: gcc -fPIC -g3 -O0   -DNDEBUG -DNVALGRIND -O2
MPICH CXX: g++ -g -O0 -fPIC  -DNDEBUG -DNVALGRIND -O2
MPICH F77: gfortran -fPIC -ffree-line-length-0 -g -O0  -O2
MPICH FC: gfortran -fPIC -ffree-line-length-0 -g -O0  -O2

And when I did  mpirun --version, I got this:

mpirun (Open MPI) 1.6.5

and doing 

ompi_info

gives:

 Package: Open MPI buildd@allspice Distribution
                Open MPI: 1.6.5
   Open MPI SVN revision: r28673
   Open MPI release date: Jun 26, 2013
                Open RTE: 1.6.5
   Open RTE SVN revision: r28673
   Open RTE release date: Jun 26, 2013
                    OPAL: 1.6.5
       OPAL SVN revision: r28673
       OPAL release date: Jun 26, 2013
                 MPI API: 2.1
            Ident string: 1.6.5
                  Prefix: /usr
 Configured architecture: x86_64-pc-linux-gnu
          Configure host: allspice
           Configured by: buildd
           Configured on: Sat Dec 28 23:38:31 UTC 2013
          Configure host: allspice
                Built by: buildd
                Built on: Sat Dec 28 23:41:47 UTC 2013
              Built host: allspice
              C bindings: yes
            C++ bindings: yes
      Fortran77 bindings: yes (all)
      Fortran90 bindings: yes
 Fortran90 bindings size: small
              C compiler: gcc
     C compiler absolute: /usr/bin/gcc
  C compiler family name: GNU
      C compiler version: 4.8.2
            C++ compiler: g++
   C++ compiler absolute: /usr/bin/g++
      Fortran77 compiler: gfortran
  Fortran77 compiler abs: /usr/bin/gfortran
      Fortran90 compiler: gfortran
  Fortran90 compiler abs: /usr/bin/gfortran
             C profiling: yes
           C++ profiling: yes
     Fortran77 profiling: yes
     Fortran90 profiling: yes
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
           Sparse Groups: no
  Internal debug support: no
  MPI interface warnings: no
     MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
         libltdl support: yes
   Heterogeneous support: yes
 mpirun default --prefix: no
         MPI I/O support: yes
       MPI_WTIME support: gettimeofday
     Symbol vis. support: yes
   Host topology support: yes
          MPI extensions: affinity example
   FT Checkpoint support: yes (checkpoint thread: yes)
     VampirTrace support: no
  MPI_MAX_PROCESSOR_NAME: 256
    MPI_MAX_ERROR_STRING: 256
     MPI_MAX_OBJECT_NAME: 64
        MPI_MAX_INFO_KEY: 36
        MPI_MAX_INFO_VAL: 256
       MPI_MAX_PORT_NAME: 1024
  MPI_MAX_DATAREP_STRING: 128
           MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.5)
              MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.5)
           MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.5)
               MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.5)
               MCA carto: file (MCA v2.0, API v2.0, Component v1.6.5)
               MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.5)
               MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.5)
               MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.5)
           MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.5)
           MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.5)
               MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.5)
         MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.5)
         MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.5)
             MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.5)
               MCA hwloc: external (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA crs: none (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.5)
              MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.5)
           MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.5)
           MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: self (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.5)
                MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.5)
                  MCA io: romio (MCA v2.0, API v2.0, Component v1.6.5)
               MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.5)
               MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.5)
               MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA pml: crcpw (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA pml: v (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.5)
              MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA btl: ofud (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA btl: self (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.5)
                MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.5)
                MCA crcp: bkmrk (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.5)
                MCA odls: default (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ras: tm (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.5)
               MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA rml: ftrm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.5)
              MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA plm: tm (MCA v2.0, API v2.0, Component v1.6.5)
               MCA snapc: full (MCA v2.0, API v2.0, Component v1.6.5)
               MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.5)
              MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: env (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: tm (MCA v2.0, API v2.0, Component v1.6.5)
                 MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.5)
             MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.5)
             MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.5)
             MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.5)
            MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.5)
            MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.5)

[Wolfgang Bangerth]

On 10/28/2016 11:49 AM, RAJAT ARORA wrote:
>
> I think that openmpi and Mpich are clashing now. (I didn't even know
> that both were installed upto now).
>
> I think, if I can disable openmpi somehow, I will be in good shape. This
> is because, I compiled deal.ii with mpich library which was installed by
> petsc.
>
> Can you help me how to disable openmpi ?

It is almost always a bad idea to install PETSc via --download-mpich.
Throw away your PETSc installation and re-install it without
--download-mpich. It should just find the OpenMPI installation you have
on your system.

[RAJAT ARORA]

Respected Professor,

Yes, I remember this discussion regarding not using --download mpich tag while installing Petsc happening on github as well.

Yes, I think I should now do it. Although it still remains a mystery to me, how and why openmpi got installed and is now clashing with mpich.

Anyways, for the time being, I have edited my PATH, and using Petsc mpich as default. 

I must reinstall Petsc and deal.ii again. Is there a need to reinstall P4est as well ? 

Thanks a lot Prof. Bangerth and Timo for your help and suggestions.

[Timo Heister]

> I must reinstall Petsc and deal.ii again. Is there a need to reinstall P4est
> as well ?

p4est also uses MPI so yes.

By the way: after you install deal.II, you should run "make test" in
the build directory. I am pretty sure it would have detected this
problem.