Are there known issue between Trilinos 13.0.1 and deal.II?

[blais...@gmail.com]

Dear All,

I hope you are well.

I am trying to compile deal.II on a new cluster in Canada called Narval (named after another cute whale, we love those up north). It uses the following versions of the core library that we need:

gcc/9.3.0

openmpi/4.0.3

trilinos/13.0.1

parmetis/4.0.3

p4est/2.2

However, whenever I run CMAKE using deal.II AND enabling Trilinos, I get the following error:

Configuration error: Cannot compile a test program with the final set of

    compiler and linker flags:

      CXX flags (DEBUG): -pedantic -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd -Wno-parentheses -Wno-unused-local-typedefs -O0 -ggdb -Wa,--compress-debug-sections

I'll spare you the rest of the Blabla.

This is clearly something I need to fix with my cluster admin. However, before i delve into this, have any of you faced such issue? Is it an issue with Trilinos 130.1, or is this  a cluster-related issue?

Thank  you so much

Bruno

[Wolfgang Bangerth]

On 10/28/21 8:02 PM, blais...@gmail.com wrote:
>
> However, whenever I run CMAKE using deal.II AND enabling Trilinos, I get the
> following error:
> Configuration error: Cannot compile a test program with the final set of
>     compiler and linker flags:
>     CXX flags (DEBUG): -pedantic -fPIC -Wall -Wextra -Wmissing-braces
> -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override
> -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new
> -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi
> -Wno-class-memaccess -fopenmp-simd -Wno-parentheses -Wno-unused-local-typedefs
> -O0 -ggdb -Wa,--compress-debug-sections

Somewhere, in some file that is probably located under CMakeFiles in your
build directory, it will provide you with whatever error messages cmake got
when it tried to compile a test program with these flags. If you know what
that error is, you're probably 75% there towards identifying what the real
underlying problem is.

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[Praveen C]

I have compiled dealii@9.3 with tril...@13.0.1 on various systems with gcc 8/9/10 via spack.

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/87dc8659-1d0c-420d-99c2-4d98764f78f8n%40googlegroups.com.

[blais...@gmail.com]

Wolfgang, you are a genius.

The exact error I am getting is related to AVX instructions not being available. I think this is normal because this is an AMD Rome based cluster and not an intel one.

Run Build Command(s):/cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin/gmake cmTC_be568/fast && /cvmfs/soft.computecanada.ca/gentoo/2020/usr/bin/gmake -f CMakeFiles/cmTC_be568.dir/build.make CMakeFiles/cmTC_be568.dir/build
gmake[1]: Entering directory '/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_be568.dir/src.cxx.o
/cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/gcccore/9.3.0/bin/c++    -DDEAL_II_HAVE_AVX   -o CMakeFiles/cmTC_be568.dir/src.cxx.o -c /home/blaisbru/dealii/build/CMakeFiles/CMakeTmp/src.cxx
/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp/src.cxx:3:6: error: #error "__AVX__ flag not set, no support for AVX"
    3 |     #error "__AVX__ flag not set, no support for AVX"
      |      ^~~~~
/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp/src.cxx: In function ‘int main()’:
/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp/src.cxx:35:9: warning: AVX vector return without AVX enabled changes the ABI [-Wpsabi]
   35 |       b = _mm256_set1_pd (static_cast<volatile double>(2.25));
      |       ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gmake[1]: *** [CMakeFiles/cmTC_be568.dir/build.make:66: CMakeFiles/cmTC_be568.dir/src.cxx.o] Error 1
gmake[1]: Leaving directory '/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:121: cmTC_be568/fast] Error 2

Hopefully I find a solution from there :)!

[Martin Kronbichler]

Dear Bruno,

This is a regular error and not related to the actual problem I believe. This is simply the message that we use to detect what SIMD extensions we can enable, and in your case the compilation does not support AVX. But I think further down in the CMakeFiles/CMakeError.log file you should find the culprit.

As a side remark, the AMD Rome architecture does indeed support AVX2, as long as you pass "-mavx2 -mfma" or "-march=znver2" or simply "-march=native" to the compile flags ("CMAKE_CXX_FLAGS"). However, AMD does not (yet) support AVX-512, which might be what you're referring to. In other words, the compiler needs to be told what kind of architecture extensions are supported on the target CPU. I most often use "-march=native" unless I'm cross-compiling on a different CPU compared to where the code is eventually executed.

Best,
Martin

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/fb910de8-53f0-4eca-90a7-14a54e3069a1n%40googlegroups.com.

[blais...@gmail.com]

Dear Martin,

I guess I was being overly enthusiastic. I understand now, every "failed test" (e.g.  DEAL_II_HAVE_CXX20_FEATURES) end up in this file.

The majority of the "errors" I get are thus regular configuration "failure"

The final failure I get is related to : Performing C++ SOURCE FILE Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG failed with the following output:
Change Dir: /home/blaisbru/dealii/build/CMakeFiles/CMakeTmp

It's a massive error message, but the key line appear to be these one to me:

/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/trilinos/13.0.1/lib/libzadelus.so: error: undefined reference to 'Adelus::nprocs_col'
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/trilinos/13.0.1/lib/libzadelus.so: error: undefined reference to 'Adelus::me'
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/trilinos/13.0.1/lib/libzadelus.so: error: undefined reference to 'Adelus::nprocs_row'
collect2: error: ld returned 1 exit status
gmake[1]: *** [CMakeFiles/cmTC_c4ee5.dir/build.make:266: cmTC_c4ee5] Error 1

gmake[1]: Leaving directory '/home/blaisbru/dealii/build/CMakeFiles/CMakeTmp'

gmake: *** [Makefile:121: cmTC_c4ee5/fast] Error 2

Source file was:
int main(){ return 0; }

Is Adelus a Trilinos component that is mandatory? In my case, this is clearly the issue, if this component is not there, then that's why I am getting this error.

Thank you very much for the help! This community is always so amazing.

[Wolfgang Bangerth]

On 10/29/21 5:10 AM, blais...@gmail.com wrote:
>
> Is Adelus a Trilinos component that is mandatory? In my case, this is clearly
> the issue, if this component is not there, then that's why I am getting this
> error.

I have never heard of that component. You can probably disable it.

A good question is why the error happens in the first place. I believe
(Matthias might be able to confirm) that we import the list of libraries to
link with from Trilinos. Apparently, this list did not, but should have,
included the Adelus library. That might be a Trilinos bug, but that is no
consolation.

[blais...@gmail.com]

Is there a way to disable the usage of Zadelus from deal.II?

I don't see any CMAKE options that I could pass to disable Zadelus.

I tried googling the component and I can't figure out what it is...

:)

I'll ask my cluster administrator to recompile Trilinos with Zadelus, I just hope it's not an old component that was deprecated in Trilinos 13.0.1

Thanks for the help! I really appreciate it.

Best

Bruno

[Wolfgang Bangerth]

On 10/29/21 7:10 AM, blais...@gmail.com wrote:
> Is there a way to disable the usage of Zadelus from deal.II?
> I don't see any CMAKE options that I could pass to disable Zadelus.
> I tried googling the component and I can't figure out what it is...
> :)
> I'll ask my cluster administrator to recompile Trilinos with Zadelus, I just
> hope it's not an old component that was deprecated in Trilinos 13.0.1

I think you will have to recompile Trilinos without that specific package.
When you compile Trilinos, you pass a list of cmake flags of the form
TPL_ENABLE_... for the package you want, and you probably want to omit (or
explicitly disable) the one that is the problem.

[Bruno Turcksin]

Bruno,

It looks like you are not the only one to have this problem:

https://github.com/trilinos/Trilinos/issues/8299

https://github.com/trilinos/Trilinos/issues/9426

Possible solution from the second issue:  use `-DBUILD_SHARED_LIBS:BOOL=OFF` or add -undefined dynamic_lookup to CMAKE_CXX_FLAGS

Best,

Bruno

[blais...@gmail.com]

Dear all,

After discussion with my cluster admin, he suggested the use of the following flag:

-DTrilinos_FIND_COMPONENTS="Pike;PikeImplicit;PikeBlackBox;TrilinosCouplings;Panzer;PanzerMiniEM;PanzerAdaptersSTK;PanzerDiscFE;PanzerDofMgr;PanzerCore;Piro;ROL;Stokhos;Tempus;Rythmos;ShyLU;ShyLU_DD;ShyLU_DDCommon;ShyLU_DDFROSch;ShyLU_DDBDDC;Zoltan2;Zoltan2Sphynx;MueLu;Moertel;NOX;Phalanx;Percept;STK;STKExprEval;STKDoc_tests;STKUnit_tests;STKBalance;STKTools;STKTransfer;STKSearchUtil;STKSearch;STKUnit_test_utils;STKNGP_TEST;STKIO;STKMesh;STKTopology;STKSimd;STKUtil;STKMath;Compadre;Intrepid2;Intrepid;Teko;FEI;Stratimikos;Ifpack2;Anasazi;Komplex;SEACAS;SEACASEx2ex1v2;SEACASTxtexo;SEACASNumbers;SEACASNemspread;SEACASNemslice;SEACASMat2exo;SEACASMapvar-kd;SEACASMapvar;SEACASMapvarlib;SEACASExplore;SEACASGrepos;SEACASGenshell;SEACASGen3D;SEACASGjoin;SEACASFastq;SEACASEx1ex2v2;SEACASExo_format;SEACASExotxt;SEACASExomatlab;SEACASExodiff;SEACASExo2mat;SEACASEpu;SEACASEjoin;SEACASConjoin;SEACASBlot;SEACASAprepro;SEACASAlgebra;SEACASPLT;SEACASSVDI;SEACASSuplibCpp;SEACASSuplibC;SEACASSuplib;SEACASSupes;SEACASAprepro_lib;SEACASChaco;SEACASIoss;SEACASNemesis;SEACASExoIIv2for32;SEACASExodus_for;SEACASExodus;Amesos2;ShyLU_Node;ShyLU_NodeTacho;ShyLU_NodeHTS;Belos;ML;Ifpack;Zoltan2Core;Pamgen;Amesos;Galeri;AztecOO;Pliris;Isorropia;Xpetra;Thyra;ThyraTpetraAdapters;ThyraEpetraExtAdapters;ThyraEpetraAdapters;ThyraCore;Domi;TrilinosSS;Tpetra;TpetraCore;TpetraTSQR;TpetraClassic;EpetraExt;Triutils;Shards;Zoltan;Epetra;MiniTensor;Sacado;RTOp;KokkosKernels;Teuchos;TeuchosKokkosComm;TeuchosKokkosCompat;TeuchosRemainder;TeuchosNumerics;TeuchosComm;TeuchosParameterList;TeuchosParser;TeuchosCore;Kokkos;KokkosAlgorithms;KokkosContainers;KokkosCore;Gtest;TrilinosATDMConfigTests;TrilinosFrameworkTests"

I guess this enables us to choose which Trilinos components we include and which we don't. Interesting to say the least, I will have to test more to see if this has consequences.

Thank you for your help and your time. This is always greatly appreciated. I hope this may help others in the future.

Best

Bruno