Scaling Tests in Zapdos

[Shane Keniley]

Hey all,

I'm looking to do some preliminary scaling tests with Zapdos and CRANE. At the moment I'm just using a desktop with 8 cores to scope out a good problem and solver options, but hopefully I'll move it to a cluster to do some proper scaling soon. As a first test I'm just using the mean_en.i input file (using CRANE instead of the Zapdos reaction kernels), but in order to make sure I could scale the grid by exactly factors of 2, I included a GeneratedMesh with a ParsedSubdomainMeshModifier to add the gas and liquid subblocks. That way the mesh is uniform and I can control the size exactly. That seems to work fine, but at this point I have a couple questions related to running Zapdos in parallel: 

1. I'm relatively new to parallel computing, so does anyone have any good suggestions for preconditioner options for this type of problem? I've read that the LU preconditioner is not scalable so I'll need to change it out for something better. (That's for the best, as apparently the LU preconditioner is rather memory intensive -- I ran out of my measly 8 GB RAM with a problem size of only 140,000 elements running with 8 processors.)  I'm getting something working with -pc_type = bjacobi and -sub_pc_type = ilu with ksp_norm = none, but I'm not sure if this is really the best choice. Here's the full options I have running: 

   petsc_options_iname = '-pc_type -sub_pc_type -pc_factor_shift_type -pc_factor_shift_amount -snes_linesearch_minlambda'

   petsc_options_value = 'bjacobi ilu NONZERO 1e-10 1e-3' 

    
2. Would field splitting be a good approach here? 
   
   
3. And finally, just to be sure: has anyone done strong and/or weak scaling tests on Zapdos before? 

Thanks,

Shane

[Alexander Lindsay]

If it's running bjacobi, then you can definitely use that. If you see that you are taking a lot of linear iterations, then you can look at using ASM and playing around with level of overlap. The options `-pc_factor_shift_type` and `-pc_factor_shift_amount` aren't doing anything for you here. If you actually want them to be in effect, then you you need to prepend a "sub", e.g. `-sub_pc_factor_shift_type`. But since you're not using them and it's running, then maybe you can drop those options.

I'm very curious about the ksp_norm = none selection. Is that necessary? It seems wrong to me...I would hope you don't have to set anything for the ksp_norm and use our default. If the linear solve isn't converging, then the preconditioner is not working.

If you're using the newest MOOSE, try setting Executioner/automatic_scaling=true. It should improve the condition number of your matrix and improve the performance of any preconditioner.

1.  
2. Would field splitting be a good approach here? 
   

Hmm, what fields would you want to split?

1. 
   
2. And finally, just to be sure: has anyone done strong and/or weak scaling tests on Zapdos before? 

Thanks,

Shane

--
You received this message because you are subscribed to the Google Groups "zapdos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.
To post to this group, send email to zapdos...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/2345d662-d9a8-4650-aff1-452099793a4e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Lindsay]

Sent too early.

To answer your final question, no there have not been any scaling studies.

Make sure that you don't exceed 20k dofs per process for your scaling study, otherwise the scaling will be disrupted by communication overhead.

Alex

[Shane Keniley]

I'm very curious about the ksp_norm = none selection. Is that necessary? It seems wrong to me...I would hope you don't have to set anything for the ksp_norm and use our default. If the linear solve isn't converging, then the preconditioner is not working.

Upon trying with it turned off...It's definitely not necessary. The problem seemed to work better (e.g. it converged as opposed to failing at the first timestep) with one set of PETSc options I tried and I just left it on after that. But yes, you're right - that set of options was just not working. 

If you're using the newest MOOSE, try setting Executioner/automatic_scaling=true. It should improve the condition number of your matrix and improve the performance of any preconditioner.

This option is interesting. It definitely seems to decrease the number of linear iterations, but it also takes a long time to run the first timestep when I turn this option on. When I run with 17000 nodes the terminal output gets to "0 Nonlinear |R| = ..." and just stays there for a while, so the simulation takes 2-3 times as long just because of that first timestep. (The time spent on that first timestep is much shorter for smaller meshes, obviously.) I tried this out on a pretty large mesh for 1D, 140,000 nodes, and the simulation took almost 30 minutes to run even though I only ran for 10 timesteps and each step took ~10 seconds to complete.

Hmm, what fields would you want to split? 

To be fair I've never used field splitting, but I have seen people mention splitting up the Poisson-like equation and transport equations in problems like this. For example, this comment: https://groups.google.com/d/msg/moose-users/sYhWlvr1rQ0/WtYh7R6cAwAJ

So I suppose in this case the species variables and energy would be split from the Poisson equation(s). 

Thanks for the feedback! It looks like this preconditioner is working but I'll give ASM as shot as well.

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos...@googlegroups.com.

[Alexander Lindsay]

On Tue, Jun 25, 2019 at 1:31 PM Shane Keniley <smkfi...@gmail.com> wrote:

I'm very curious about the ksp_norm = none selection. Is that necessary? It seems wrong to me...I would hope you don't have to set anything for the ksp_norm and use our default. If the linear solve isn't converging, then the preconditioner is not working.

Upon trying with it turned off...It's definitely not necessary. The problem seemed to work better (e.g. it converged as opposed to failing at the first timestep) with one set of PETSc options I tried and I just left it on after that. But yes, you're right - that set of options was just not working. 

If you're using the newest MOOSE, try setting Executioner/automatic_scaling=true. It should improve the condition number of your matrix and improve the performance of any preconditioner.

This option is interesting. It definitely seems to decrease the number of linear iterations, but it also takes a long time to run the first timestep when I turn this option on. When I run with 17000 nodes the terminal output gets to "0 Nonlinear |R| = ..." and just stays there for a while, so the simulation takes 2-3 times as long just because of that first timestep. (The time spent on that first timestep is much shorter for smaller meshes, obviously.) I tried this out on a pretty large mesh for 1D, 140,000 nodes, and the simulation took almost 30 minutes to run even though I only ran for 10 timesteps and each step took ~10 seconds to complete.

Is that initial "hang" longer than the time it takes to do a normal jacobian evaluation? If so, there's definitely something weird going on. For computing the scaling, MOOSE only calls through to the diagonal Jacobian pieces of kernels.

Hmm, what fields would you want to split? 

To be fair I've never used field splitting, but I have seen people mention splitting up the Poisson-like equation and transport equations in problems like this. For example, this comment: https://groups.google.com/d/msg/moose-users/sYhWlvr1rQ0/WtYh7R6cAwAJ

So I suppose in this case the species variables and energy would be split from the Poisson equation(s). 

Yea, that's interesting. That's what I would've guessed. I think it would be very neat to try at some point. Fande says that Shohei Ogawa on the moose-users mailing list has done something kind of like this. I would say let's start with some "standard" preconditioners and then we can get more exotic as we go.

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.

To post to this group, send email to zapdos...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/e106c772-8f98-4a21-8c84-768feb29f537%40googlegroups.com.

[Shane Keniley]

Is that initial "hang" longer than the time it takes to do a normal jacobian evaluation? If so, there's definitely something weird going on. For computing the scaling, MOOSE only calls through to the diagonal Jacobian pieces of kernels.

I believe so, as it's the only part of the simulation that takes so long. It's much, much less noticeable for smaller simulations (O(1e3) nodes)), but when I start getting into the 10k range that initial "hang" takes about as long as 10 full timesteps. 

Not sure if it's the cause, but I did notice when I updated and compiled MOOSE that I got a message I hadn't seen before: 

"/usr/bin/ld: warning: libgfortran.so.4, needed by //usr/lib/x86_64-linux-gnu/liblapack.so.3, may conflict with libgfortran.so.5"

This was on a fresh installation of Ubuntu 18.04; not sure if that's related. Otherwise I'm using the latest version of MOOSE and libmesh is up to date. 

[Alexander Lindsay]

No that should be unrelated. If you could make a branch and tell me an input file that demonstrates the behavior, that would be great. I will profile it when I get the chance (which could be a week or a month haha).

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.

To post to this group, send email to zapdos...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/13c45bed-3c12-4917-8859-906fe2c8450d%40googlegroups.com.

[Shane Keniley]

Sure! It's the "plasma_liquid_test.i" file in my multiple_blocks branch: 

https://github.com/keniley1/zapdos/tree/multiple_blocks

Note that this uses a later version of CRANE than the one that is lumped together with Zapdos at the moment. I modified the Action a little bit so multiple different reaction blocks could be included (one in gas and one in liquid, in this example). 

[Alexander Lindsay]

Did you check-in the crane update for your branch? I just tried running the input file and i got the error:

*** ERROR ***
plasma_liquid_test.i:1255: section 'Reactions' does not have an associated "Action".
Did you misspell it?
plasma_liquid_test.i:1256: section 'Reactions/gas_phase_reactions' does not have an associated "Action".
Did you misspell it?
plasma_liquid_test.i:1386: section 'Reactions/liquid_phase_reactions' does not have an associated "Action".
Did you misspell it?

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.

To post to this group, send email to zapdos...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/7748d917-ba9c-48a9-a58c-5ded1de4a4a9%40googlegroups.com.

[Shane Keniley]

Wow, sorry for the late response...this just slipped my mind. Nope, I did not check in the update. I didn't think about it since it always uses my local version of CRANE, so the submodule is ignored. Sorry about that. Looks like you got it working regardless!

Apparently I was a little overly ambitious (aka wrong) when I said bjacobi was working with mpi. The problems certainly run and converge, but after a very short amount of time I noticed that it was solving in 0 nonlinear iterations. That seemed odd. I looked at the results and sure enough, it seemed to reach some kind of nonphysical steady state solution very early. I'm also seeing very substantial discontinuities where I assume the problem is split between cores, and the discontinuities grow and sometimes cause the problem to not converge when increasing the number of cores. I'm guessing these are both symptoms of the preconditioner being a poor fit. 

I've been having more success with asm with an overlap of 2 (though it seems to work better with an overlap of 4 when I'm running with 8 cores) and a sub_pc_type of lu. I think that's a good baseline for running in Zapdos in parallel. 

[Alexander Lindsay]

Shane, I believe we've fixed this in MOOSE, so go ahead and try out the latest MOOSE master or devel revision.

To unsubscribe from this group and stop receiving emails from it, send an email to zapdos-users...@googlegroups.com.

To post to this group, send email to zapdos...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/zapdos-users/658c141f-279a-4c0c-a8b1-6140c525756b%40googlegroups.com.

[Shane Keniley]

Alex, thanks for the heads up. The program no longer hangs up with automatic scaling turned on so it looks like it's fixed!