error encountered while using matrix free in GPU with periodic BCs

[vishal...@gmail.com]

Hello, 

I am facing an error while I am using matrix-free in GPU with periodic boundary conditions. I am attaching a minimal example that illustrates the issue I am facing. I am using deal.II -9.3.0-pre.

The minimal example is derived from step-64 of the tutorials. In this code, 

1) Create a single element using the hypercube function.

2) Create HEX27 finite element based dof_handler and also create the constraint matrices

3) Create matrix-free objects on the host and GPU. 

4) Create a host input vector compatible with the constraints ( I set the values at the unconstrained nodes to be its global Id).

5) Send the input vector from the host to the GPU

6) Perform a single vmult operation with the Laplace operator on the host and the GPU.

7) Send the output from the GPU to the host

8) Compare the two outputs

When I ran the code in debug mode on a single MPI task and compared the two outputs, the values at the unconstrained nodes do not seem to match. 

To ensure there are no bugs in my minimal example, I have a periodicBC flag. When The periodicBC is set to true, periodic + homogeneous Dirichlet boundary condition are imposed. If it is set to false, an homogeneous Dirichlet BC is imposed at the interior node. In this case, the output values do match. This flag affects how the constraint matrix is created and nothing else. 

I would be very grateful if someone can tell me what mistake I am making. 

Any help is greatly appreciated. 

thanks and regards,

Vishal Subramanian

[Bruno Turcksin]

Vishal,

I don't think anyone has ever tried to use periodic boundary conditions with GPU. The way we apply constraints on the GPU is very different than what is done on the CPU. So, I am not surprised that it doesn't work. I'll add that to my todo list but I have no idea when I will be able to look at this. If you need this capability any time soon, you will probably need to implement it yourself. If you choose to work on this, I can help you.

Best,

Bruno

[Sambit Das]

Hi Bruno,

Thank you for your reply.

Vishal and me are collaborating on implementing hybrid functionals in the DFT-FE code, with Vishal taking the lead on this project. Our exploration of the GPU ported matrix-free poisson solve in GPU is in that context. I did take a look at the "include/deal.II/matrix_free/cuda_hanging_nodes_internal.h" but it seems to me that detailed understanding the inner workings of the matrix-free implementation would be required to extend to periodic BCs. Since we don't have an urgent need for this capability, we will wait for this capability to be implemented in the future whenever you get a chance.

Best,

Sambit