Solving Step-74 by MPI

[chong liu]

Dear community,

I have a problem in the assembly part when solving Step-74 (DG method) by MPI. Generally speaking, we can first determine whether one cell is locally owned or not by cell->is_locally_owned(), and then assemble matrices and residuals by hands (e.g. Step-40 for CG and “Distributed LDG Method” for DG in code gallery). However, Step-74 using Meshworker and FEInterfaceValues classes automatically loops and assembles all cells, faces, and interfaces. My question is how to identify one cell which is locally owned or not in Step-74. Is there any deal.II code combining discontinuous Galerkin method (which uses FEInterfaceValues class, e.g. Step-12 and Step-74) with MPI (either PETSc or Trilinos)? Any help is going to be appreciated.

Thanks and regards,

Chong

[Timo Heister]

Hi Chong,

MeshWorker does work without much effort in MPI parallel code and is
made to help with exactly this (who assembles what is non-trivial if
you have hanging nodes and processor boundaries). The only thing you
have to watch out for is supplying the right flags that determine the
cells and faces that will be assembled. I don't think we have a
step-12 or -74 with MPI, but there is the error estimator in step-50:
https://www.dealii.org/current/doxygen/deal.II/step_50.html#Theerrorestimator

You probably want the flags
MeshWorker::assemble_own_cells |
MeshWorker::assemble_ghost_faces_once |
MeshWorker::assemble_own_interior_faces_once

(or _both for both of them if you want to assemble each face twice).

> --
> The deal.II project is located at http://www.dealii.org/
> For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/eed5b3fb-2052-44a3-8840-1be2f1484c7a%40Spark.



--
Timo Heister
http://www.math.clemson.edu/~heister/

[chong liu]

Hello Timo Heister,

Thank you for your reply. The link you shared is extremely helpful. I will try to extend Step-74 based on the ideas in Step-50. 

Best,

Chong

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/CAMRj59HahTQf0%3D0h3aLXo64DkXeoeW3RYv3pHmo8aqb91_-zYQ%40mail.gmail.com.

[Wolfgang Bangerth]

On 8/13/22 20:36, chong liu wrote:
>
> Thank you for your reply. The link you shared is extremely helpful. I will try
> to extend Step-74 based on the ideas in Step-50.

If you make that work, it would actually be quite nice to have that as a code
gallery program! Feel free to submit it as such (and/or talk to me about the
process if you're unsure)!

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[chong liu]

Hi, Wolfgang and Timo,

Sounds great!

I modified Step-74 based on the error_estimation part of Step-50. I found it can work for the attached step-74-mpi, while it cannot work for the attached step-74-mpi-error. The only difference is the location of the output command as the attached figure 1 shows. This is weird. The error information states that one vector is out of bounds as shown in figure 2.

In addition, there are three points I would like to ask

1. The direct solver cannot work for the modified MPI program. I changed it to the iterative solver (solver_cg same as Step-40) since I am not familiar with the MPI direct solver. Could you give me some suggestions on the direct solver for MPI?
2. Does not ConvergenceTable support the parallel data output? I found that the first parameter for convergencetable.write_text() is std::out. How can I modify it to pcout for MPI?
3. I guess the l1_norm() calculation for a vector should be modified. For example, the code std::sqrt(energy_norm_square_per_cell.l1_norm()) should be modified to std::sqrt(Utilities::MPI::sum(estimated_error_square_per_cell.l1_norm(), mpi_communicator)).

Looking forward to your reply. Please let me know if you have any questions. 

Thanks and regards,

Chong

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/3e1cfd2e-2b71-fd5c-8308-c6f1ea5282d5%40colostate.edu.

[Wolfgang Bangerth]

On 8/17/22 03:10, chong liu wrote:
>
> I modified Step-74 based on the error_estimation part of Step-50. I found it
> can work for the attached step-74-mpi, while it cannot work for the attached
> step-74-mpi-error. The only difference is the location of the output command
> as the attached figure 1 shows. This is weird. The error information states
> that one vector is out of bounds as shown in figure 2.

The location of the print message isn't the problem -- it's just that because
you don't end the
std::cout << "...";
call with std::endl, the text is put into a buffer but never printed to
screen. You should run the program in a debugger and check where the erroneous
access comes from. I bet that if the two programs are otherwise the same, they
should both fail.

> In addition, there are three points I would like to ask
>

> 1. The direct solver cannot work for the modified MPI program. I changed it

> to the iterative solver (solver_cg same as Step-40) since I am not
> familiar with the MPI direct solver. Could you give me some suggestions on
> the direct solver for MPI?

There isn't a good option. There are some parallel direct solvers in both the
PETSc and Trilinos (for which you can probably find information by searching
the mailing list archives), but at the end of the day, if the problem becomes
sufficiently large, even parallel direct solvers cannot compete.


> 2. Does not ConvergenceTable support the parallel data output? I found that

> the first parameter for convergencetable.write_text() is std::out. How can
> I modify it to pcout for MPI?

I'm not sure the class supports this, but you can always put a
if (Utilities::MPI::this_mpi_process(...) == 0)
in front of the place where you generate output.


> 3. I guess the l1_norm() calculation for a vector should be modified. For

> example, the code std::sqrt(energy_norm_square_per_cell.l1_norm()) should
> be modified to std::sqrt

> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dealii.org%2Fcurrent%2Fdoxygen%2Fdeal.II%2Fnamespacestd.html%23aa2def4dc0bd5b6d07af7aff83feba9b2&data=05%7C01%7CWolfgang.Bangerth%40colostate.edu%7C8676b13756db4734d87408da803061ae%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637963243499566483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=G7i0Xg9AQIQkzIUAodl745glNO9sRfgnok1piSbkatM%3D&reserved=0>(Utilities::MPI::sum
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.dealii.org%2Fcurrent%2Fdoxygen%2Fdeal.II%2FnamespaceUtilities_1_1MPI.html%23ab544a3bf3301a6dd3e705ee352c5551b&data=05%7C01%7CWolfgang.Bangerth%40colostate.edu%7C8676b13756db4734d87408da803061ae%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637963243499566483%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=dZzc08aCivKuCOOuxnaeEr0ZBSnMB3qRzuVUNDTShvY%3D&reserved=0>(estimated_error_square_per_cell.l1_norm(),
> mpi_communicator)).

Yes, something like this.

[Timo Heister]

For error computations using cellwise errors you can use
VectorTools::compute_global_error(), which does the MPI communication
for you:
https://www.dealii.org/developer/doxygen/deal.II/namespaceVectorTools.html#a21eb62d70953182dcc2b15c4e14dd533

See step-55 for example.

> > <https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.dealii.org*2Fcurrent*2Fdoxygen*2Fdeal.II*2Fnamespacestd.html*23aa2def4dc0bd5b6d07af7aff83feba9b2&data=05*7C01*7CWolfgang.Bangerth*40colostate.edu*7C8676b13756db4734d87408da803061ae*7Cafb58802ff7a4bb1ab21367ff2ecfc8b*7C0*7C0*7C637963243499566483*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000*7C*7C*7C&sdata=G7i0Xg9AQIQkzIUAodl745glNO9sRfgnok1piSbkatM*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!PTd7Sdtyuw!QO7eTW5kBhW7q3_5N54yZsv-uBFuFmPb-34csveLecKNqiHWY-Dvroos6-RjWwuJhwHp3T_urOOoL15wN9um0II$ >(Utilities::MPI::sum
> > <https://urldefense.com/v3/__https://nam10.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.dealii.org*2Fcurrent*2Fdoxygen*2Fdeal.II*2FnamespaceUtilities_1_1MPI.html*23ab544a3bf3301a6dd3e705ee352c5551b&data=05*7C01*7CWolfgang.Bangerth*40colostate.edu*7C8676b13756db4734d87408da803061ae*7Cafb58802ff7a4bb1ab21367ff2ecfc8b*7C0*7C0*7C637963243499566483*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C2000*7C*7C*7C&sdata=dZzc08aCivKuCOOuxnaeEr0ZBSnMB3qRzuVUNDTShvY*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!PTd7Sdtyuw!QO7eTW5kBhW7q3_5N54yZsv-uBFuFmPb-34csveLecKNqiHWY-Dvroos6-RjWwuJhwHp3T_urOOoL15wBuy3vhk$ >(estimated_error_square_per_cell.l1_norm(),

> > mpi_communicator)).
>
> Yes, something like this.
>
> Best
> W.
>
>
> --
> ------------------------------------------------------------------------
> Wolfgang Bangerth email: bang...@colostate.edu

> www: https://urldefense.com/v3/__http://www.math.colostate.edu/*bangerth/__;fg!!PTd7Sdtyuw!QO7eTW5kBhW7q3_5N54yZsv-uBFuFmPb-34csveLecKNqiHWY-Dvroos6-RjWwuJhwHp3T_urOOoL15w0hl1NSs$