Multigrid in 3D questions

[Lucas Myers]

Hi folks,

I'm trying to solve a 3D problem in parallel using an AMG preconditioner, but the performance is bad. I'm wondering if I can get some advice from someone who has experience choosing and tuning multigrid preconditioners, particularly for problems in 3D.

For context:

• The problem is vector-valued (5 components) and elliptic(ish)
  
• I use GMRES w/BoomerAMG preconditioner for asymmetric matrices, everything is default
• In 2D, scaling is good even down to 5,000 DoFs per processor, and number of iterations is independent of problem size
  
• In 3D the scaling is bad. Adding more processors after about 50,000 DoFs/processor actually slows the program down and sometimes gives memory errors.
• Taking away the preconditioner in 3D gives a ~20x speedup at 16 processors, and strong scaling is linear. However, the number of iterations increases with problem size.
  

Given all that, I have some questions:

1. Why might it be that the memory and wall-time scaling is so bad in 3D?
2. Are there any examples lying around deal.II of folks using multigrid for 3D problems in parallel? All the tutorials that I looked at were in 2D, including the 2 examples used in the distributed computing paper.
   
3. Might there be an easy way to fix it while still using BoomerAMG? I know the Hypre documentation gives some parameter recommendations, but I'm not so sure (1) how to set those options via deal.II (I think they are not available via the AdditionalData interface), or (2) whether those will work. Does anyone have experience with this?
4. Might the Trilinos AMG preconditioner work any better for this problem by default? And if not, is there a systematic way to tune it (particularly using a deal.II interface) to work better for a 3D problem?
5. Might the in-house GMG methods work better? And if so, do the matrix-free methods stand a chance of performing better even if I have to use some complicated functions in the matrix assembly (for my weak form I have a 5-component nonlinear function of my solution vector which has to be inverted via Newton's method across my domain).

If more context on the particular problem would be helpful I can certainly give details. Mostly I'm looking for intuition, general suggestions, or pointers to good references. Any help is much appreciated.

Kind regards,

Lucas

[Wolfgang Bangerth]

Lucas,
it's hard to tell why this might be so slow. In general, I *think* what
you are doing is apply the AMG to the entire, coupled system? That is
unusual, though maybe not strictly wrong. In practice, however, we
almost always apply multigrid preconditioners only to the elliptic
blocks of coupled systems. You can see that in the preconditioner that
is discussed in the "Possibilities for extensions" section of step-22,
where we need to solve an elliptic problem as part of a preconditioner
for Stokes, and this is the approach that is also taken in a number of
other related programs (31, 32, 56). The experience of the community is
that applying multigrid (geometric or algebraic) to the entire coupled
problem is just fraught with so many problems that it is hard to get
right and harder yet to efficiently implement.

Best
W.


On 3/3/22 10:22, Lucas Myers wrote:
> *** Caution: EXTERNAL Sender ***

>
> Hi folks,
>
> I'm trying to solve a 3D problem in parallel using an AMG
> preconditioner, but the performance is bad. I'm wondering if I can get
> some advice from someone who has experience choosing and tuning
> multigrid preconditioners, particularly for problems in 3D.
>
> For context:
>

> * The problem is vector-valued (5 components) and elliptic(ish)
> * I use GMRES w/BoomerAMG preconditioner for asymmetric matrices,
> everything is default
> * In 2D, scaling is good even down to 5,000 DoFs per processor, and

> number of iterations is independent of problem size

> * In 3D the scaling is bad. Adding more processors after about 50,000

> DoFs/processor actually slows the program down and sometimes gives
> memory errors.

> * Taking away the preconditioner in 3D gives a ~20x speedup at 16

> processors, and strong scaling is linear. However, the number of
> iterations increases with problem size.
>
> Given all that, I have some questions:
>

> 1. Why might it be that the memory and wall-time scaling is so bad in 3D?
> 2. Are there any examples lying around deal.II of folks using multigrid

> for 3D problems in parallel? All the tutorials that I looked at were
> in 2D, including the 2 examples used in the distributed computing paper.

> 3. Might there be an easy way to fix it while still using BoomerAMG? I
> know the Hypre documentation
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fhypre.readthedocs.io%2Fen%2Flatest%2Fsolvers-boomeramg.html&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=P8OcXGNcXuwWJtLFRCdDsxvIJpKKpkA2p5%2BQSxdkDKE%3D&reserved=0>

> gives some parameter recommendations, but I'm not so sure (1) how to
> set those options via deal.II (I think they are not available via
> the AdditionalData interface), or (2) whether those will work. Does
> anyone have experience with this?

> 4. Might the Trilinos AMG preconditioner work any better for this

> problem by default? And if not, is there a systematic way to tune it
> (particularly using a deal.II interface) to work better for a 3D
> problem?

> 5. Might the in-house GMG methods work better? And if so, do the

> matrix-free methods stand a chance of performing better even if I
> have to use some complicated functions in the matrix assembly (for
> my weak form I have a 5-component nonlinear function of my solution
> vector which has to be inverted via Newton's method across my domain).
>
> If more context on the particular problem would be helpful I can
> certainly give details. Mostly I'm looking for intuition, general
> suggestions, or pointers to good references. Any help is much appreciated.
>
> Kind regards,
> Lucas
>

> --
> The deal.II project is located at http://www.dealii.org/
> <https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dealii.org%2F&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=NJZnFeCaXecCwK6xyRuBwI033BIPlGrkD5E%2F%2Fy4%2B6ew%3D&reserved=0>
> For mailing list/forum options, see
> https://groups.google.com/d/forum/dealii?hl=en
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fforum%2Fdealii%3Fhl%3Den&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CGYWQ0dRtj%2Fhxr6m7t4DdR0BcU59lM25AuMwIrB4Ns4%3D&reserved=0>
> ---
> You received this message because you are subscribed to the Google
> Groups "deal.II User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to dealii+un...@googlegroups.com
> <mailto:dealii+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/dealii/7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%40googlegroups.com
> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fdealii%2F7bcfdab4-ea5e-47ea-82f5-c78bc9e3ffd8n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7CWolfgang.Bangerth%40colostate.edu%7Cacf6fbe3988a4f6fad0c08d9fd3a5c3d%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637819249379691458%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xPHuLbXYG8SbVTOQJ0PLCz4JqiaFJC7y%2F3AmI5V9N34%3D&reserved=0>.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[blais...@gmail.com]

Maybe I can add something there.
Although it's clearly not ideal to use the AMG preconditioner on the whole matrix, it can work quite well sometimes.

Have you played around with the smoother and the coarsener you use or are you using the default values? I have found that you need to tune the AMG preconditioner parameters a bit to get a decent. My experience is limited to the Trilinos AMG preconditioner (and not the MueLU one), but if you define your constant modes correctly and use adequate smoother and coarsener, it's really not all that bad.

Feel free to reach out if you feel I could help you with this.
Best

Bruno

[Lucas Myers]

Hi Bruno,

No, unfortunately I have not played with the parameter values. Hypre has a set of recommended parameter values for 3D problems, but I think some are not exposed to the deal.II interface so I have avoided digging into the source.

I chose PETSc somewhat arbitrarily, so if you are more familiar with the Trilinos equivalents I can switch to that. Glancing at the documentation it appears that all of the Trilinos parameters are manipulable via the deal.II interface using the Teuchos::ParameterList object -- do you know if this is true? If, so it might be a better option.

With regards to actually choosing parameters: Is there some (reasonably) systematic way to tune them? It looks like the ML documentation gives some tools to get started on profiling with the different parameters (section 6.6) but I'm unsure whether the utility methods are necessarily accessible via the deal.II interface. Additionally, it looks like there are quite a few parameter values to choose. Do you just futz with each one individually and look at the speed? Or is there some particular order or hierarchy to choosing the parameters? Is there some manner in which I can analyze my actual problem to optimize some choices? And finally, do you have recommendations for 3D problems rather than 2D?

Thanks so much for the help

- Lucas

[blais...@gmail.com]

Hey Lucas,

Sorry for the delay in my answer.

All Trilinos AMG parameter are manipulable through the deal.II interface. Jean-Paul Pelterer added that (at my request I think) something like 2 years ago. It was nice of him :)

It's a bit difficult to explain all we tried in terms of AMG. If you want, you can reach out to me by e-mail (bruno.blais at polymtl.ca) and we can set-up a small zoom to discuss this. I'll be glad to try and help.

Sorry again for the later answer

Best

Bruno