CUDA facilities

[giuseppe orlando]

Hello everyone, 

I am writing this post to ask a couple of questions about the CUDA facilities in deal.II, after a look on step-64 tutorial.

1. Searching for the documentation, I noticed that routines such as CUDAWrappers::FEFaceEvalution or CUDAWrappers::MatrixFree::loop are not defined. Am I correct? Does this mean, for instance, that a Discontinuous Galerkin extension of step-64 tutorial with a matrix free formulation is not feasible at the moment?

2. I noticed instead that a CUDAWrappers::SparseMatrix class is defined. However, it seems to me that it can be initialized only with SparseMatrix class. Does it mean that it cannot be combined with MPI sparse matrices (i.e. PETSc or Trilinos)? Moreover, is there somewhere an example of use of such class, so as to check e.g. the proper use Utilities::CUDA::Handle for the 'reinit' function?

Thanks in advance.

Best,

Giuseppe

[Bruno Turcksin]

Giuseppe,

I am not sure if google sent my previous message, so I will send it again. I apologize if you get it twice.

We are currently replacing the CUDA code with Kokkos. These will allow us to support Nvidia, AMD, and Intel GPUs. It will also make it easier to add new functionalities. Unfortunately there will be a very limited number of non-backward compatible changes.

On Monday, May 8, 2023 at 5:50:22 AM UTC-4 gius...@gmail.com wrote:

1. Searching for the documentation, I noticed that routines such as CUDAWrappers::FEFaceEvalution or CUDAWrappers::MatrixFree::loop are not defined. Am I correct? Does this mean, for instance, that a Discontinuous Galerkin extension of step-64 tutorial with a matrix free formulation is not feasible at the moment?

That's correct. Currently we only support continuous finite elements with a single component. Support for multiple components is on my todo list.

 

2. I noticed instead that a CUDAWrappers::SparseMatrix class is defined. However, it seems to me that it can be initialized only with SparseMatrix class. Does it mean that it cannot be combined with MPI sparse matrices (i.e. PETSc or Trilinos)? Moreover, is there somewhere an example of use of such class, so as to check e.g. the proper use Utilities::CUDA::Handle for the 'reinit' function?

CUDAWrappers::SparseMatrix is the CUDA version of the SparseMatrix class and so it has the same limitations. If you want MPI sparse matrices, you need to use PETSc or Trilinos. They support GPU through Kokkos.

For examples using CUDAWrappers::SparseMatrix look at the solver, sparse_matrix, and precondition tests here.

Best,

Bruno

[Giuseppe Orlando]

Dear Bruno,

thanks for the kind response. I will certainly have a look to the files that you suggested me about the CUDAWrappers::SparseMatrix (and also the other tests included therein).

Just as curiosity (if I am not inappropriate and you can answer me): are these new GPU facilities with Kokkos already and usable ailable anin the development repository? Because, for instance, the step-64 tutorial source code has been converted from ‘step-64.cu’ to ‘step-64.cc’, the same holds for the CUDA tests folder that you linked me and, finally, I could find some ‘Kokkos keywords’, but I cannot figure out from a rapid glance if it can be already executed with a version of deal.II installed with ‘DEAL_II_WITH_KOKKOS’ flag on or one has to wait the new release.

 

Best

 

Giuseppe

 

Da: Bruno Turcksin
Inviato: lunedì 8 maggio 2023 16:11
A: deal.II User Group
Oggetto: [deal.II] Re: CUDA facilities

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "deal.II User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dealii/Fgm4g2TjOrQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/6ea874b3-8758-47f5-b6a5-a225ddbddd00n%40googlegroups.com.

 

[Bruno Turcksin]

Giuseppe,

The move to Kokkos is still in progress, so right now we have a mix of Kokkos and CUDA. This works because under the hood Kokkos uses CUDA to run on Nvidia GPUs. So far we have changed the build system and the distributed vectors. I have a branch where the MatrixFree infrastructure uses Kokkos. I still need to clean up some of the code before I create the PR. The SparseMatrix code won't be moved to Kokkos for a while but you can mix CUDA and Kokkos so it shouldn't be a problem.

Best,

Bruno

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/0F7FB6EA-24F4-4146-B1BA-FC856DD4BC18%40hxcore.ol.