Installation problem with CUDA

[Severin]

Dear dealii developers,

I've followed readme.html to try installing deal.ii with Trilinos 16.0 and the internal Kokkos is configured with -DKokkos_ENABLE_CUDA=ON. And I've exported OMPI_CXX=/path/to/kokkos's/nvcc_wrapper when compiling Trilinos and deal.ii.

Now problems are that 

1. the bundled version of boost library has problem with my nvcc (12.4), this issue has been reported here, then I compiled another version of boost (1.89.0) by myself.

2. after replacing to a newer version of boost, I still met a problem related to boost:

/home/fengshw/Downloads/dealii-9.7.0/source/non_matching/quadrature_generator.cc(581): error: namespace "boost::math::tools" has no member "toms748_solve"
                boost::math::tools::toms748_solve(lambda,
                                    ^

1 error detected in the compilation of "/home/fengshw/Downloads/dealii-9.7.0/source/non_matching/quadrature_generator.cc". 

It seems that using the bundled version is a better choice

3. and a new problem took place in matrix-free module (only for debug version, release version can be built):

Building CXX object source/numerics/CMakeFiles/object_numerics_debug.dir/vector_tools_project_qpmf.cc.o
/home/fengshw/Downloads/dealii-9.7.0/include/deal.II/matrix_free/tools.h: In member function ‘void dealii::MatrixFreeTools::internal::ComputeDiagonalHelper<dim, VectorizedArrayType, is_face>::reinit(unsigned int)’:
/home/fengshw/Downloads/dealii-9.7.0/include/deal.II/matrix_free/tools.h:1003:924: error: ‘__T0’ was not declared in this scope; did you mean ‘__y0’?
 1003 |                           AssertIndexRange(constraint_position[std::get<1>(hn)],
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ^  
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            __y0

/home/fengshw/Downloads/dealii-9.7.0/include/deal.II/matrix_free/tools.h:1003:979: error: template argument 1 is invalid
 1003 |                           AssertIndexRange(constraint_position[std::get<1>(hn)],

I wonder the last one is caused by nvcc or not since I succeed in compiling deal.ii with gcc (without setting MPI, Trilinos). If it is the case, which version of nvcc should I choose? My device is RTX 4060 Ti.

log.txt is the output during make.

Thank you

Severin

[Bruno Turcksin]

Hello Severin,

The last error is a compiler issue. One thing that is strange in your detailed.log is that your CMAKE_CXX_COMPILER is set to mpicxx. I always set it to the nvcc_wrapper. Normally any version of nvcc should work. I use nvcc 12.6 but 12.4 should be fine too.

Best,

Bruno

[Severin]

Well, after weeks attempt, I've succeeded in compiling (and running of course) deal.ii with CUDA through built-in Kokkos in Trilinos with the great advice from Bruno. I leaves my configuration information of both Trilinos and deal.ii here for reference by those who need it.

Trilinos configuration:

cmake -DTrilinos_ENABLE_Amesos=ON -DTrilinos_ENABLE_Epetra=ON -DTrilinos_ENABLE_EpetraExt=ON -DTrilinos_ENABLE_Ifpack=ON -DTrilinos_ENABLE_AztecOO=ON -DTrilinos_ENABLE_Sacado=ON -DTrilinos_ENABLE_SEACAS=OFF -DTrilinos_ENABLE_Teuchos=ON -DTrilinos_ENABLE_MueLu=ON -DTrilinos_ENABLE_ML=ON -DTrilinos_ENABLE_NOX=ON -DTrilinos_ENABLE_ROL=ON -DTrilinos_ENABLE_Tpetra=ON -DTrilinos_ENABLE_FLOAT=ON -DTrilinos_ENABLE_Zoltan=ON -DTrilinos_VERBOSE_CONFIGURE=OFF -DTPL_ENABLE_MPI=ON -DMPI_BASE_DIR=/home/fengshw/libraries/openmpi -DBUILD_SHARED_LIBS=ON -DCMAKE_VERBOSE_MAKEFILE=OFF -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/home/fengshw/libraries/trilinos -DTrilinos_ENABLE_Kokkos=ON -DKokkos_ENABLE_CUDA=ON -DKokkos_ENABLE_CUDA_CONSTEXPR=ON -DTpetra_INST_SERIAL:BOOL=ON -DMueLu_ENABLE_Tutorial:BOOL=OFF -DTrilinos_SHOW_DEPRECATED_WARNINGS=OFF -DShyLU_NodeTacho_ENABLE_CUSOLVER=ON -DShyLU_NodeTacho_ENABLE_CUBLAS=ON ..

deal.ii configuration:

cmake -DCMAKE_CXX_COMPILER=/home/fengshw/libraries/trilinos/bin/nvcc_wrapper -DCMAKE_INSTALL_PREFIX=/home/fengshw/libraries/dealii -DDEAL_II_WITH_MPI=ON -DMPI_DIR=/home/fengshw/libraries/openmpi -DDEAL_II_MPI_WITH_DEVICE_SUPPORT=ON -DDEAL_II_WITH_64BIT_INDICES=ON -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR=/home/fengshw/libraries/trilinos -DDEAL_II_WITH_TBB=ON -DTBB_DIR=/home/fengshw/libraries/oneTBB -DDEAL_II_WITH_P4EST=ON -DP4EST_DIR=/home/fengshw/libraries/p4est ..

Notes:

1. Compare to Trilinos configuration example provided in deal.ii README, I cancel option -DTrilinos_ENABLE_SEACAS because I don't have NetCDF and cancel option -DTrilinos_ENABLE_COMPLEX since with this option CMake will give warning that requires -DCMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS:BOOL turns to ON. However, turn on the latter flag will result in no input file for nvcc.
2. One have to firstly export OMPI_CXX to nvcc_wrapper that comes with Kokkos if MPI and CUDA work together, details see this site.
3. The additional flags -DShyLU_NodeTacho_ENABLE_CUSOLVER=ON and -DShyLU_NodeTacho_ENABLE_CUBLAS=ON are necessary if use CUDA, otherwise deal.ii will complain about no cusolver-related libraries can be found even when running tutorial step-1.
4. Another additional flag  -DTrilinos_SHOW_DEPRECATED_WARNINGS=OFF just turns off deprecated warning therefore Trilinos stays quiet during the compile phase.
5. deal.ii with configurations above is able to pass the tests within the release package and I've run successfully the tutorial step 1, step 9 and step 64.
6. oneTBB is built with default setting and p4est is built with the script provided by deal.ii website. 
7. Finally, my gcc, g++ and gfortran version is 12.3; nvcc version, as the advice by Bruno, is 12.6; MPI is OpenMPI 5.0.8, oneTBB is of version 2022.2.0 and p4est is of version 2.8.7. OS: Ubuntu 22.04.

Best,

Severin

[Severin]

An additional information is that with configurations above, code can be compiled and run but if I build the project in VSCode with CMakeTools, the intelliSense will complain about cuda.h and cuda_runtime.h cannot be found (maybe other headers related to CUDA). The building is, however, successful and the executable file can run normally.

I wonder if this the problem of CMakeTools extension or the generated cmake files by deal.ii? I leaves CMake output below:

[cmake] Not searching for unused variables given on the command line.
[cmake] -- Using the deal.II-9.7.0 installation found at /home/fengshw/libraries/dealii
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_add_test.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_initialize_cached_variables.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_invoke_autopilot.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_pickup_tests.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_query_git_information.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_deal_ii_setup_target.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_shell_escape_option_groups.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_target_compile_flags.cmake
[cmake] -- Include macro /home/fengshw/libraries/dealii/share/deal.II/macros/macro_target_link_flags.cmake
[cmake] -- The C compiler identification is GNU 12.3.0
[cmake] -- The CXX compiler identification is GNU 12.3.0
[cmake] -- Detecting C compiler ABI info
[cmake] -- Detecting C compiler ABI info - done
[cmake] -- Check for working C compiler: /usr/bin/cc - skipped
[cmake] -- Detecting C compile features
[cmake] -- Detecting C compile features - done
[cmake] -- Detecting CXX compiler ABI info
[cmake] -- Detecting CXX compiler ABI info - done
[cmake] -- Check for working CXX compiler: /home/fengshw/libraries/trilinos/bin/nvcc_wrapper - skipped
[cmake] -- Detecting CXX compile features
[cmake] -- Detecting CXX compile features - done
[cmake] -- Autopilot invoked
[cmake] ###
[cmake] #
[cmake] #  Project  step-1  set up with  deal.II-9.7.0  found at
[cmake] #      /home/fengshw/libraries/dealii
[cmake] #
[cmake] #  CMAKE_BUILD_TYPE:          Debug
[cmake] #
[cmake] #  You can now run
[cmake] #       $ make                - to compile and link the program
[cmake] #       $ make run            - to (compile, link and) run the program
[cmake] #
[cmake] #       $ make debug          - to switch the build type to 'Debug'
[cmake] #       $ make release        - to switch the build type to 'Release'
[cmake] #
[cmake] #       $ make edit_cache     - to change (cached) configuration variables
[cmake] #                               and rerun the configure and generate phases of CMake
[cmake] #
[cmake] #       $ make strip_comments - to strip the source files in this
[cmake] #                               directory off their comments; this is irreversible
[cmake] #       $ make clean          - to remove the generated executable as well as
[cmake] #                               all intermediate compilation files
[cmake] #       $ make runclean       - to remove all output generated by the program
[cmake] #       $ make distclean      - to clean the directory from _all_ generated
[cmake] #                               files (includes clean, runclean and the removal
[cmake] #                               of the generated build system)
[cmake] #       $ make info           - to view this message again
[cmake] #
[cmake] #  Have a nice day!
[cmake] #
[cmake] ###
[cmake] -- Configuring done (1.8s)
[cmake] -- Generating done (0.0s)
[cmake] -- Build files have been written to: /home/fengshw/program/dealiiTutorialStep1GPU/build

Anyone who use VSCode for development met this problem?