Installing trilinos with MPI

Juan Rodríguez

unread,

Apr 2, 2015, 3:03:51 PM4/2/15

to dea...@googlegroups.com

Hello all,

I am trying to install Trilinos before compiling deal.II. I downloaded Trilinos 11.12.1 and I am following the instructions in the deal.II webpage:

cmake -D Trilinos_ENABLE_Sacado=ON \
-D Trilinos_ENABLE_Stratimikos=ON \
-D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_CXX_FLAGS="-g -O3" \
-D CMAKE_C_FLAGS="-g -O3" \
-D CMAKE_FORTRAN_FLAGS="-g -O5" \
-D Trilinos_EXTRA_LINK_FLAGS="-lgfortran" \
-D CMAKE_VERBOSE_MAKEFILE=FALSE \
-D Trilinos_VERBOSE_CONFIGURE=FALSE \
-D TPL_ENABLE_MPI=ON \
-D BUILD_SHARED_LIBS=ON \
-D CMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \
..

During the installation of Trilinos, I get the following error:

[ 0%] Building CXX object packages/teuchos/core/src/CMakeFiles/teuchoscore.dir/Teuchos_UnitTestRepository.cpp.o
In file included from /home/jrglez/Downloads/trilinos-11.12.1-Source/packages/teuchos/core/src/Teuchos_UnitTestRepository.cpp:51:0:
/home/jrglez/Downloads/trilinos-11.12.1-Source/packages/teuchos/core/src/Teuchos_Time.hpp:56:17: fatal error: mpi.h: No such file or directory
#include "mpi.h"
^
compilation terminated.
make[2]: *** [packages/teuchos/core/src/CMakeFiles/teuchoscore.dir/Teuchos_UnitTestRepository.cpp.o] Error 1
make[1]: *** [packages/teuchos/core/src/CMakeFiles/teuchoscore.dir/all] Error 2
make: *** [all] Error 2

I can avoid this error if I set -D TPL_ENABLE_MPI=OFF, but I would like to install it with MPI. Do you have any advice?

I have open MPI v.1.6.5. Just in case it's useful, when this is what I get when I run ompi_info

Package: Open MPI buildd@allspice Distribution
Open MPI: 1.6.5
Open MPI SVN revision: r28673
Open MPI release date: Jun 26, 2013
Open RTE: 1.6.5
Open RTE SVN revision: r28673
Open RTE release date: Jun 26, 2013
OPAL: 1.6.5
OPAL SVN revision: r28673
OPAL release date: Jun 26, 2013
MPI API: 2.1
Ident string: 1.6.5
Prefix: /usr
Configured architecture: x86_64-pc-linux-gnu
Configure host: allspice
Configured by: buildd
Configured on: Sat Dec 28 23:38:31 UTC 2013
Configure host: allspice
Built by: buildd
Built on: Sat Dec 28 23:41:47 UTC 2013
Built host: allspice
C bindings: yes
C++ bindings: yes
Fortran77 bindings: yes (all)
Fortran90 bindings: yes
Fortran90 bindings size: small
C compiler: gcc
C compiler absolute: /usr/bin/gcc
C compiler family name: GNU
C compiler version: 4.8.2
C++ compiler: g++
C++ compiler absolute: /usr/bin/g++
Fortran77 compiler: gfortran
Fortran77 compiler abs: /usr/bin/gfortran
Fortran90 compiler: gfortran
Fortran90 compiler abs: /usr/bin/gfortran
C profiling: yes
C++ profiling: yes
Fortran77 profiling: yes
Fortran90 profiling: yes
C++ exceptions: no
Thread support: posix (MPI_THREAD_MULTIPLE: no, progress: no)
Sparse Groups: no
Internal debug support: no
MPI interface warnings: no
MPI parameter check: runtime
Memory profiling support: no
Memory debugging support: no
libltdl support: yes
Heterogeneous support: yes
mpirun default --prefix: no
MPI I/O support: yes
MPI_WTIME support: gettimeofday
Symbol vis. support: yes
Host topology support: yes
MPI extensions: affinity example
FT Checkpoint support: yes (checkpoint thread: yes)
VampirTrace support: no
MPI_MAX_PROCESSOR_NAME: 256
MPI_MAX_ERROR_STRING: 256
MPI_MAX_OBJECT_NAME: 64
MPI_MAX_INFO_KEY: 36
MPI_MAX_INFO_VAL: 256
MPI_MAX_PORT_NAME: 1024
MPI_MAX_DATAREP_STRING: 128
MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.6.5)
MCA memory: linux (MCA v2.0, API v2.0, Component v1.6.5)
MCA paffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.5)
MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.6.5)
MCA carto: file (MCA v2.0, API v2.0, Component v1.6.5)
MCA shmem: mmap (MCA v2.0, API v2.0, Component v1.6.5)
MCA shmem: posix (MCA v2.0, API v2.0, Component v1.6.5)
MCA shmem: sysv (MCA v2.0, API v2.0, Component v1.6.5)
MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.6.5)
MCA maffinity: hwloc (MCA v2.0, API v2.0, Component v1.6.5)
MCA timer: linux (MCA v2.0, API v2.0, Component v1.6.5)
MCA installdirs: env (MCA v2.0, API v2.0, Component v1.6.5)
MCA installdirs: config (MCA v2.0, API v2.0, Component v1.6.5)
MCA sysinfo: linux (MCA v2.0, API v2.0, Component v1.6.5)
MCA hwloc: external (MCA v2.0, API v2.0, Component v1.6.5)
MCA crs: none (MCA v2.0, API v2.0, Component v1.6.5)
MCA dpm: orte (MCA v2.0, API v2.0, Component v1.6.5)
MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.6.5)
MCA allocator: basic (MCA v2.0, API v2.0, Component v1.6.5)
MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: basic (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: inter (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: self (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: sm (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: sync (MCA v2.0, API v2.0, Component v1.6.5)
MCA coll: tuned (MCA v2.0, API v2.0, Component v1.6.5)
MCA io: romio (MCA v2.0, API v2.0, Component v1.6.5)
MCA mpool: fake (MCA v2.0, API v2.0, Component v1.6.5)
MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.6.5)
MCA mpool: sm (MCA v2.0, API v2.0, Component v1.6.5)
MCA pml: bfo (MCA v2.0, API v2.0, Component v1.6.5)
MCA pml: crcpw (MCA v2.0, API v2.0, Component v1.6.5)
MCA pml: csum (MCA v2.0, API v2.0, Component v1.6.5)
MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.6.5)
MCA pml: v (MCA v2.0, API v2.0, Component v1.6.5)
MCA bml: r2 (MCA v2.0, API v2.0, Component v1.6.5)
MCA rcache: vma (MCA v2.0, API v2.0, Component v1.6.5)
MCA btl: ofud (MCA v2.0, API v2.0, Component v1.6.5)
MCA btl: openib (MCA v2.0, API v2.0, Component v1.6.5)
MCA btl: self (MCA v2.0, API v2.0, Component v1.6.5)
MCA btl: sm (MCA v2.0, API v2.0, Component v1.6.5)
MCA btl: tcp (MCA v2.0, API v2.0, Component v1.6.5)
MCA topo: unity (MCA v2.0, API v2.0, Component v1.6.5)
MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.6.5)
MCA osc: rdma (MCA v2.0, API v2.0, Component v1.6.5)
MCA crcp: bkmrk (MCA v2.0, API v2.0, Component v1.6.5)
MCA iof: hnp (MCA v2.0, API v2.0, Component v1.6.5)
MCA iof: orted (MCA v2.0, API v2.0, Component v1.6.5)
MCA iof: tool (MCA v2.0, API v2.0, Component v1.6.5)
MCA oob: tcp (MCA v2.0, API v2.0, Component v1.6.5)
MCA odls: default (MCA v2.0, API v2.0, Component v1.6.5)
MCA ras: cm (MCA v2.0, API v2.0, Component v1.6.5)
MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.5)
MCA ras: loadleveler (MCA v2.0, API v2.0, Component v1.6.5)
MCA ras: slurm (MCA v2.0, API v2.0, Component v1.6.5)
MCA ras: tm (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: resilient (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.6.5)
MCA rmaps: topo (MCA v2.0, API v2.0, Component v1.6.5)
MCA rml: ftrm (MCA v2.0, API v2.0, Component v1.6.5)
MCA rml: oob (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: binomial (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: cm (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: direct (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: linear (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: radix (MCA v2.0, API v2.0, Component v1.6.5)
MCA routed: slave (MCA v2.0, API v2.0, Component v1.6.5)
MCA plm: rsh (MCA v2.0, API v2.0, Component v1.6.5)
MCA plm: slurm (MCA v2.0, API v2.0, Component v1.6.5)
MCA plm: tm (MCA v2.0, API v2.0, Component v1.6.5)
MCA snapc: full (MCA v2.0, API v2.0, Component v1.6.5)
MCA filem: rsh (MCA v2.0, API v2.0, Component v1.6.5)
MCA errmgr: default (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: env (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: hnp (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: singleton (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: slave (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: slurm (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: slurmd (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: tm (MCA v2.0, API v2.0, Component v1.6.5)
MCA ess: tool (MCA v2.0, API v2.0, Component v1.6.5)
MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.6.5)
MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.6.5)
MCA grpcomm: hier (MCA v2.0, API v2.0, Component v1.6.5)
MCA notifier: command (MCA v2.0, API v1.0, Component v1.6.5)
MCA notifier: syslog (MCA v2.0, API v1.0, Component v1.6.5)

Thank you in advance,

Juan

Bruno Turcksin

unread,

Apr 2, 2015, 3:44:40 PM4/2/15

to dea...@googlegroups.com

Juan,

the problem is that Trilinos cannot find MPI. Where is MPI installed ? If it is not in a standard place (for example you installed MPI yourself instead of using the package manager), you can set yourself (http://trilinos.org/oldsite/Trilinos10CMakeQuickstart.txt).

Best,

Bruno

Juan Rodríguez

unread,

Apr 2, 2015, 8:11:56 PM4/2/15

to dea...@googlegroups.com

Thanks for the help,

I think that openmpi was not properly installed, because changing that didn't work. I installed v.1.8 in a new location (/home/$USER/openmp) and added the options:

-D MPI_BASE_DIR:PATH="/home/$USER/openmpi" \

-D MPI_BIN_DIR:PATH="/home/$USER/openmpi/bin" \
-D MPI_EXEC:FILEPATH="/home/$USER/openmpi/bin/mpirun" \

This time it worked just fine.

Thanks again

Juan Rodríguez

unread,

Apr 3, 2015, 8:41:43 PM4/3/15

to dea...@googlegroups.com

Hello again,

The configuration went well (or so I thought), but when I continued the installation I got an error message. I am trying to install deal.II with UMFPACK and Trilinos, so I used the following options during configuration:

cmake -DCMAKE_INSTALL_PREFIX=/home/jrglez/dealII ../dealii-8.2.1/ \

-DDEAL_II_WITH_LAPACK=ON \
-DDEAL_II_WITH_UMFPACK=ON \
-DDEAL_II_WITH_MPI=ON\
-DMPI_DIR=~/src/openmpi\
-DCMAKE_C_COMPILER="~/src/openmpi/bin/mpicc" \
-DCMAKE_CXX_COMPILER="~/src/openmpi/bin/mpicxx" \
-DCMAKE_Fortran_COMPILER="~/src/openmpi/bin/mpif90" \
-DDEAL_II_WITH_TRILINOS=ON \
-DTRILINOS_DIR=~/share/trilinos/ \
-DDEAL_II_WITH_PETSC=OFF

The error I got is pretty long and I cannot figure out what the problem is, so I have attached a file with the error I got (dealIIinstall.txt). I also attached the configuration log (detailed.log).

I tried installing without trilinos (but with mpi), but keeping the rest of the options as I described and it worked, passed all the tests and run at least step-1. However, I got a lot of warnings, so I have attached them (dealII_install_NO_trilinos.txt, the error appear at 1%, 74% and 96%) and the log (detailed_NO_trilinos.log).

Thanks for the help.

Juan

detailed.log

dealII_install.txt

dealII_install_NO_trilinos.txt

detailed_NO_trilinos.log

Wolfgang Bangerth

unread,

Apr 3, 2015, 9:54:30 PM4/3/15

to dea...@googlegroups.com

> The error I got is pretty long and I cannot figure out what the problem is, so
> I have attached a file with the error I got (dealIIinstall.txt). I also
> attached the configuration log (detailed.log).

The errors are all of the kind

In file included from /home/jrglez/share/trilinos/include/ml_epetra_utils.h:42:0,
from
/home/jrglez/Downloads/dealii-8.2.1/source/lac/trilinos_sparse_matrix.cc:30:
/home/jrglez/share/trilinos/include/mpi.h:27:0: warning: "MPI_VERSION"
redefined [enabled by default]
#define MPI_VERSION 1
^
In file included from
/home/jrglez/Downloads/build/include/deal.II/base/config.h:575:0,
from
/home/jrglez/Downloads/dealii-8.2.1/include/deal.II/lac/trilinos_sparse_matrix.h:20,
from
/home/jrglez/Downloads/dealii-8.2.1/source/lac/trilinos_sparse_matrix.cc:16:
/home/jrglez/src/openmpi/include/mpi.h:239:0: note: this is the location of
the previous definition
#define MPI_VERSION 3

Notice how it references two different mpi.h files, in
/home/jrglez/share/trilinos/include/mpi.h
/home/jrglez/src/openmpi/include/mpi.h

This can't work. You need to figure out which one you want to use. (It may be
that the first one was installed by Trilinos because Trilinos couldn't find an
MPI installation.)

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@math.tamu.edu
www: http://www.math.tamu.edu/~bangerth/

Juan Rodríguez

unread,

Apr 7, 2015, 1:28:49 PM4/7/15

to dea...@googlegroups.com

Thanks for the answer. I had to remove both mpi and trilinos and re-install everything again. I installed openmpi in /usr/share to make sure that trilinos would find that installation.

This time it worked (I run "make tests", step-1 and step-31 to make sure that everything works fine).

However, now I am having problems with lapack and umpack, wich was working before. But I will deal with that later.

Thanks for the help.

Juan

Ivan Cherkashin

unread,

Dec 22, 2015, 7:11:52 PM12/22/15

to deal.II User Group

Hello everyone!

I realize this discussion ended a while ago and that my message is not directly related to Juan's problem. Nevertheless, I would like to share my experience in hopes of it being useful to beginners trying to use Trilinos with MPI.

I ran into a similar problem when I compiled my program using Trilinos (Trilinos's installation went fine, however, in contrast with Juan's case). Namely, I got the error

fatal error: mpi.h: No such file or directory

whenever I compiled my code.

I was looking for a solution over the Internet in vain until I read that MPI programs should be compiled with a compiler wrapper. In CMake (example attached), this is achieved simply by finding the MPI package and linking MPI libraries with the code. This resolved the problem and all calls to Trilinos began to work.

This may be obvious to experienced users, but I personally was not aware of the need to compile the code using Trilinos with an MPI compiler wrapper: I assumed compiling Trilinos itself with an MPI wrapper during installation was sufficient.

Hopefully, this may save time of a beginner user since this discussion is a second link provided by Google if you search "trilinos mpi.h not found" (that's how I found this discussion).

Ivan

CMakeLists.txt

Ivan Cherkashin

unread,

Dec 22, 2015, 7:12:20 PM12/22/15

to deal.II User Group

Hello everyone!

I realize this discussion ended a while ago and that my message is not directly related to Juan's problem. Nevertheless, I would like to share my experience in hopes of it being useful to beginners trying to use Trilinos with MPI.

I ran into a similar problem when I compiled my program using Trilinos (Trilinos's installation went fine, however, in contrast with Juan's case). Namely, I got the error