Combination of Trilinos and deal.II fails at compilation

[Maxi Miller]

When compiling deal.II without Trilinos, the compilation works. When including Trilinos, I get a lot of errors, beginning with:

In file included from /usr/local/include/ml_epetra_utils.h:42:0,
                 from /usr/local/include/ml_MultiLevelPreconditioner.h:86,
                 from ~/Downloads/dealii/source/lac/trilinos_precondition_ml.cc:33:
/usr/local/include/mpi.h:157:16: error: redefinition of ‘struct MPI_Status’
 typedef struct MPI_Status {
                ^~~~~~~~~~
In file included from ~/Downloads/dealii/build/include/deal.II/base/config.h:328:0,
                 from ~/Downloads/dealii/include/deal.II/lac/trilinos_index_access.h:19,
                 from ~/Downloads/dealii/source/lac/trilinos_precondition_ml.cc:16:
/opt/intel/compilers_and_libraries/linux/mpi/include64/mpi.h:679:16: note: previous definition of ‘struct MPI_Status’
 typedef struct MPI_Status {
                ^~~~~~~~~~
In file included from /usr/local/include/ml_epetra_utils.h:42:0,
                 from /usr/local/include/ml_MultiLevelPreconditioner.h:86,
                 from ~/Downloads/dealii/source/lac/trilinos_precondition_ml.cc:33:
/usr/local/include/mpi.h:162:3: error: conflicting declaration ‘typedef int MPI_Status’
 } MPI_Status;

Compilation is done using mpicc/mpicxx from the Intel MPI package. Configuration and project generation using cmake works without any errors.
How can I fix that? Did I forgot anything?

[Bruno Turcksin]

Hi,

do you have the same problem with openmpi or mpich? If I remember correctly, we had problems in the past with Intel MPI

Best,

Bruno

[Maxi Miller]

Not tested yet. The compilation of both libraries separated works fine with the Intel MPI toolkit, but the combination does not.
Should I test that (i.e. recompile all libraries using OpenMPI), or are there other possible fixes?

[Bruno Turcksin]

2017-07-08 16:40 GMT-04:00 'Maxi Miller' via deal.II User Group
<dea...@googlegroups.com>:

> Should I test that (i.e. recompile all libraries using OpenMPI), or are
> there other possible fixes?

Yeah, I would try using OpenMPI. Honestly, I have never seen this
error before, so I don't know what else to do.

Best,

Bruno

[Wolfgang Bangerth]

The problem is thisL

> In file included from /usr/local/include/ml_epetra_utils.h:42:0,
> from /usr/local/include/ml_MultiLevelPreconditioner.h:86,
> from
> ~/Downloads/dealii/source/lac/trilinos_precondition_ml.cc:33:
> /usr/local/include/mpi.h:157:16: error: redefinition of ‘struct MPI_Status’

^^^^^^^^^^^^^^^^^^^^^^^^

> typedef struct MPI_Status {
> ^~~~~~~~~~
> In file included from
> ~/Downloads/dealii/build/include/deal.II/base/config.h:328:0,
> from
> ~/Downloads/dealii/include/deal.II/lac/trilinos_index_access.h:19,
> from
> ~/Downloads/dealii/source/lac/trilinos_precondition_ml.cc:16:
> /opt/intel/compilers_and_libraries/linux/mpi/include64/mpi.h:679:16: note:

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Note how there are two mpi.h files that get into each other's way.

I suspect that you really want to use the latter because you use the intel
compilers, right? Looking into the Trilinos file ml_epetra_utils.h, it has
this unfortunate piece of code:

#ifdef ML_MPI
#ifndef EPETRA_MPI
#define EPETRA_MPI
#endif
#include "mpi.h"
#endif
#include "ml_include.h"
#include <iostream>

The code is unfortunate because it uses double quotes around "mpi.h", which in
your case happens to pick up /usr/local/include/mpi.h because the file is
installed in /usr/local/include/ml_epetra_utils.h when you really wanted the
mpi.h that comes with the compiler. You can fix this by replacing this by
#include <mpi.h>
in ml_epetra_utils.h. You may have to do the same in ml_smoother.h,
MLAPI_Workspace.h and Epetra_Time.h.

This is, at its core, a bug in Trilinos. I'll open a report about it.

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Wolfgang Bangerth]

On 07/08/2017 11:55 AM, 'Maxi Miller' via deal.II User Group wrote:
> When compiling deal.II without Trilinos, the compilation works. When including
> Trilinos, I get a lot of errors, beginning with:

See
https://github.com/trilinos/Trilinos/issues/1480

[Maxi Miller]

Bug also confirmed with OpenMPI. Is there a way to "uninstall" the conflicting files, such that I can install them in their own paths?

[Wolfgang Bangerth]

On 07/09/2017 12:56 PM, Maxi Miller wrote:
> Bug also confirmed with OpenMPI. Is there a way to "uninstall" the conflicting
> files, such that I can install them in their own paths?

That depends on how they got there :-) Did you install them yourself? Or did a
sysadmin do that?

As I said, one solution would also be to edit the four files in
/usr/local/include.

[Maxi Miller]

Installed it via

make install

. Now I get the error message:

[ 61%] Building CXX object source/numerics/CMakeFiles/
obj_numerics_debug.dir/matrix_creator_inst3.cc.o
In file included from /usr/local/include/Teuchos_Assert.hpp:46:0,
                 from /usr/local/include/Teuchos_any.hpp:51,
                 from /usr/local/include/Teuchos_ParameterEntry.hpp:51,
                 from /usr/local/include/Teuchos_ParameterList.hpp:51,
                 from ~/Downloads/dealii/include/deal.II/lac/trilinos_precondition.h:39,
                 from ~/Downloads/dealii/source/lac/trilinos_precondition_muelu.cc:17:
/usr/local/include/Teuchos_DefaultMpiComm.hpp: In instantiation of ‘Teuchos::MpiComm<Ordinal>::MpiComm(MPI_Comm) [with Ordinal = int; MPI_Comm = ompi_communicator_t*]’:
/usr/local/include/Ifpack2_LocalFilter_def.hpp:110:20:   required from ‘Ifpack2::LocalFilter<MatrixType>::LocalFilter(const Teuchos::RCP<const Tpetra::RowMatrix<typename MatrixType::scalar_type, typename MatrixType::local_ordinal_type, typename MatrixType::global_ordinal_type, typename MatrixType::node_type> >&) [with MatrixType = Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >; typename MatrixType::node_type = Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>; typename MatrixType::global_ordinal_type = int; typename MatrixType::local_ordinal_type = int; typename MatrixType::scalar_type = double]’
/usr/local/include/Ifpack2_Details_DenseSolver_def.hpp:245:23:   required from ‘void Ifpack2::Details::DenseSolver<MatrixType, false>::initialize() [with MatrixType = Tpetra::RowMatrix<double, int, int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> >]’
~/Downloads/dealii/source/lac/trilinos_precondition_muelu.cc:302:1:   required from here
/usr/local/include/Teuchos_DefaultMpiComm.hpp:835:3: error: ISO C++ forbids comparison between pointer and integer [-fpermissive]
   TEUCHOS_TEST_FOR_EXCEPTION(rawMpiComm == MPI_COMM_NULL,

[Wolfgang Bangerth]

On 07/09/2017 01:48 PM, Maxi Miller wrote:
> Installed it via
> |
> make install
> |

It's possible that, assuming you still have the build directory, you can
uninstall things by saying
make uninstall

But I don't know whether Trilinos supports that.


> |
> [61%]BuildingCXX objectsource/numerics/CMakeFiles/
> obj_numerics_debug.dir/matrix_creator_inst3.cc.o
> Infile included from/usr/local/include/Teuchos_Assert.hpp:46:0,
> from/usr/local/include/Teuchos_any.hpp:51,
> from/usr/local/include/Teuchos_ParameterEntry.hpp:51,
> from/usr/local/include/Teuchos_ParameterList.hpp:51,
> from~/Downloads/dealii/include/deal.II/lac/trilinos_precondition.h:39,
> from~/Downloads/dealii/source/lac/trilinos_precondition_muelu.cc:17:
> /usr/local/include/Teuchos_DefaultMpiComm.hpp:Ininstantiation of

> ‘Teuchos::MpiComm<Ordinal>::MpiComm(MPI_Comm)[withOrdinal=int;MPI_Comm
> =ompi_communicator_t*]’:
> /usr/local/include/Ifpack2_LocalFilter_def.hpp:110:20: required
> from‘Ifpack2::LocalFilter<MatrixType>::LocalFilter(constTeuchos::RCP<constTpetra::RowMatrix<typenameMatrixType::scalar_type,typenameMatrixType::local_ordinal_type,typenameMatrixType::global_ordinal_type,typenameMatrixType::node_type>>&)[withMatrixType=Tpetra::RowMatrix<double,int,int,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>>;typenameMatrixType::node_type
> =Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>;typenameMatrixType::global_ordinal_type
> =int;typenameMatrixType::local_ordinal_type
> =int;typenameMatrixType::scalar_type =double]’
> /usr/local/include/Ifpack2_Details_DenseSolver_def.hpp:245:23: required

> from‘voidIfpack2::Details::DenseSolver<MatrixType,false>::initialize()[withMatrixType=Tpetra::RowMatrix<double,int,int,Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP>>]’

> ~/Downloads/dealii/source/lac/trilinos_precondition_muelu.cc:302:1: required
> fromhere
> /usr/local/include/Teuchos_DefaultMpiComm.hpp:835:3:error:ISO C++forbids
> comparison between pointer andinteger [-fpermissive]
> TEUCHOS_TEST_FOR_EXCEPTION(rawMpiComm ==MPI_COMM_NULL,
> |

That looks like another bug in Trilinos, though I don't know how to fix it
other than uncommenting line 835-837. It's also possible that you could change
this to
TEUCHOS_TEST_FOR_EXCEPTION(rawMpiComm == (MPI_Comm)MPI_COMM_NULL,
but I'm only partially sure this will work.

Please do report this bug to the Trilinos folks the same way as I reported the
previous issue.

[Maxi Miller]

Neither did Trilinos provide such a method, nor did the fix help, instead it invoked multiple errors in addition. Is there a temporary fix I can apply to continue work while this problem is fixed by the Trilinos-team?