Compilation fails for deal.II when compiling with PETSc+Kokkos

[develo...@googlemail.com]

Hei,

I recently tried to re-install deal.II on a new server. For testing purposes I installed PETSc with Kokkos (as it supports the direct integration) and CUDA-support.

Unfortunately, compilation does now not work at all, with the error

/usr/bin/cmake -S/home/local/Downloads/git-files/dealii -B/home/local/Downloads/git-files/dealii/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/bin/cmake -E cmake_progress_start /home/local/Downloads/git-files/dealii/build/CMakeFiles /home/local/Downloads/git-files/dealii/build//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/local/Downloads/git-files/dealii/build'
make  -f cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build.make cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/depend
make[2]: Entering directory '/home/local/Downloads/git-files/dealii/build'
cd /home/local/Downloads/git-files/dealii/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/local/Downloads/git-files/dealii /home/local/Downloads/git-files/dealii/cmake/scripts /home/local/Downloads/git-files/dealii/build /home/local/Downloads/git-files/dealii/build/cmake/scripts /home/local/Downloads/git-files/dealii/build/cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/local/Downloads/git-files/dealii/build'
make  -f cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build.make cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build
make[2]: Entering directory '/home/local/Downloads/git-files/dealii/build'
[  0%] Building CXX object cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o
cd /home/local/Downloads/git-files/dealii/build/cmake/scripts && /opt/petsc/bin/kokkos_launch_compiler /opt/petsc/bin/nvcc_wrapper /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx -DBOOST_NO_AUTO_PTR  -fPIC -march=native -mavx2 -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd -Wno-parentheses -Wno-unused-local-typedefs -expt-extended-lambda;-Wext-lambda-captures-this;-lineinfo;-arch=sm_60;-Xcompiler;-fopenmp -MD -MT cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o -MF CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o.d -o CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o -c /home/local/Downloads/git-files/dealii/cmake/scripts/expand_instantiations.cc
/usr/bin/ld: warning: cannot find entry symbol xpt-extended-lambda; defaulting to 0000000000001040
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o: in function `_start':
(.text+0x1b): undefined reference to `main'
collect2: error: ld returned 1 exit status
/bin/sh: 1: -Wext-lambda-captures-this: not found
/bin/sh: 1: -lineinfo: not found
/bin/sh: 1: -arch=sm_60: not found
/bin/sh: 1: -Xcompiler: not found
/bin/sh: 1: -fopenmp: not found
make[2]: *** [cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build.make:76: cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o] Error 127
make[2]: Leaving directory '/home/local/Downloads/git-files/dealii/build'
make[1]: *** [CMakeFiles/Makefile2:1758: cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/all] Error 2
make[1]: Leaving directory '/home/local/Downloads/git-files/dealii/build'
make: *** [Makefile:136: all] Error 2

Is there a way I can fix/circumvent that bug? And where is it coming from?

Thanks!

[Daniel Arndt]

Somewhere the compiler flag -DKOKKOS_DEPENDENCE got lost, you might try if adding it fixes your problem.

Kokkos allows specifying an arbitrary CMake C++ compiler to be used as host compiler for nvcc under the hood. As long as the compiler is not clang++ (which can compile CUDA code on its own),

everything that depends on Kokkos (all targets that use -DKOKKOS_DEPENDENCE for compilation) is compiled by nvcc with the specified CMake C++ as host compiler.

If the target doesn't depend on Kokkos, the host compiler is invoked directly but it of course doesn't know about CUDA compiler flags like -arch=sm_60.

Best,

Daniel

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/9671187b-de9e-44bb-a096-c78cb81e882bn%40googlegroups.com.

[develo...@googlemail.com]

Thanks for the idea! I tested it, and got the following output:

/opt/petsc/bin/kokkos_launch_compiler /opt/petsc/bin/nvcc_wrapper /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx -DBOOST_NO_AUTO_PTR  -fPIC -march=native -mavx2 -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd -DKOKKOS_DEPENDENCE -fPIC -march=native -mavx2 -Wno-parentheses -Wno-unused-local-typedefs -expt-extended-lambda;-Wext-lambda-captures-this;-lineinfo;-arch=sm_60;-Xcompiler;-fopenmp -MD -MT cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o -MF CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o.d -o CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o -c /home/local/Downloads/git-files/dealii/cmake/scripts/expand_instantiations.cc
nvcc fatal   : No input files specified; use option --help for more information

/bin/sh: 1: -Wext-lambda-captures-this: not found
/bin/sh: 1: -lineinfo: not found
/bin/sh: 1: -arch=sm_60: not found
/bin/sh: 1: -Xcompiler: not found
/bin/sh: 1: -fopenmp: not found
make[2]: *** [cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build.make:76: cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o] Error 127

Are there other flags which might help?

Thanks,

regards,

Roland

[Wolfgang Bangerth]

On 6/16/22 01:49, 'develo...@googlemail.com' via deal.II User Group wrote:
> **

>
> Thanks for the idea! I tested it, and got the following output:

> //opt/petsc/bin/kokkos_launch_compiler /opt/petsc/bin/nvcc_wrapper

> /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx
> /opt/intel/oneapi/mpi/2021.6.0/bin/mpicxx -DBOOST_NO_AUTO_PTR  -fPIC
> -march=native -mavx2 -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual
> -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth
> -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations
> -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd
> -DKOKKOS_DEPENDENCE -fPIC -march=native -mavx2 -Wno-parentheses
> -Wno-unused-local-typedefs
> -expt-extended-lambda;-Wext-lambda-captures-this;-lineinfo;-arch=sm_60;-Xcompiler;-fopenmp
> -MD -MT
> cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o
> -MF CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o.d -o
> CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o -c
> /home/local/Downloads/git-files/dealii/cmake/scripts/expand_instantiations.cc
> nvcc fatal   : No input files specified; use option --help for more information
> /bin/sh: 1: -Wext-lambda-captures-this: not found
> /bin/sh: 1: -lineinfo: not found
> /bin/sh: 1: -arch=sm_60: not found
> /bin/sh: 1: -Xcompiler: not found
> /bin/sh: 1: -fopenmp: not found
> make[2]: ***
> [cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/build.make:76:
> cmake/scripts/CMakeFiles/expand_instantiations_exe.dir/expand_instantiations.cc.o]

> Error 127/

> Are there other flags which might help?

The underlying issue here is that you are trying to compile a tool that is
supposed to be run during compile time on the host (namely, the
'expand_instantiations' program) with the *target* compiler (namely, nvcc).
The error messages indicate that that doesn't work, and indeed it shouldn't
work because the executable is not meant to be executed on the GPU but on the CPU.

The bigger question is *why* that happens. I don't really know the answer
because I've never tried to compile for GPUs, so others might have to help out
with that, but I wonder how it comes that you are trying to compile with
/opt/petsc/bin/kokkos_launch_compiler. Did you set the CXX environment
variable to that, or passed this as a flag to cmake?

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[develo...@googlemail.com]

I did not set any of those settings by hand, they were set automatically during the configuration/generation step, I suppose.

Regards,

Roland

[Daniel Arndt]

Roland,

Instead of debugging where thighs go wring in choosing the correct compiler when using the launch_compiler, you could either use Kokkos' nvcc_wrapper (in the Kokkos installation directory in the bin directory) or use clang++.

The former one forces all compilation to go through nvcc which you likely want anyway since everything you are compiling now implicitly depends on Kokkos (which considers all files as CUDA files if you compile with the CUDA backend).

The same applies to using clang++ that can deal with CUDA files out-of-the-box on its own.

Best,

Daniel

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/313e44cd-de0a-4c6a-9642-a6c912248c92n%40googlegroups.com.