Solve does not converge

362 views
Skip to first unread message

at...@mst.edu

unread,
Jan 16, 2017, 11:26:41 PM1/16/17
to moose-users
Dear users,

I am trying to develop a new code based on  previously developed code by my  friend. In the old model I had two concentrations  for Iron and Oxygen (c_fe, c_o)  and three non conserved parameter for three phases of Iron, Oxide and Air (eta_fe, eta_o, eta_air). Now, for the newer model, I basically tried to add one more concentration of Nitrogen (c_ni) to the air phase. I added required equations to the Kernel section (lines 305-322) and tried to update other parts of the code accordingly based on the old given code. However, my solution does not converge. Since it does not give an error message, I believe the syntax is right but I must have missed sth or messed up other parts of the code. I know it is hard to debug these kind of stuff but I would very much appreciate any help or idea. 

Please let me know If more data is required.

I have attached the old (FeO_rand_mod7.i) and the new (FeO_Nitrogen_VM.i) input files.

Bests,
Alireza 
FeO_Nitrogen_VM.i
FeO_rand_mod7.i

Kong, Fande

unread,
Jan 17, 2017, 10:10:12 AM1/17/17
to moose...@googlegroups.com
Hi Alireza,

Make your problem small and run with "-pc_type lu". The first thing we want to check is if you have the right Jacobian because most of divergence cases are caused by an incorrect or inaccurate Jacobian.

The nonlinear solver does not converge or the linear one does not converge? Could you share the residual history with us?


Fande,

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/24dead82-a0eb-4f43-a77c-0180857a04e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Alireza Toghraee

unread,
Jan 17, 2017, 3:34:25 PM1/17/17
to moose...@googlegroups.com
Thank you Fande for the help. Should I add the "-pc_type lu" at the end of the running command in terminal ( I think that's what you meant)? or should I add it in the preconditioning section of input file? 

Anyway I add that at the end of my command line in terminal in the following format:

[moose][~]> mpiexec -np 16 ./projects/moose/modules/combined/modules-opt -i projects/moose/Alireza/FeO_Nit/FeO_Nitrogen_VM.i -pc_type lu

and below is the result: 

*** Warning ***
This simulation does not actually need the third derivatives of DerivativeFunctionMaterialBase elastic_free_energy_face
at /home/moose/projects/moose/modules/phase_field/src/materials/DerivativeFunctionMaterialBase.C, line 112


Time Step  0, time = 0
                dt = 0

Postprocessor Values:
+----------------+----------------+----------------------+-------------------+----------------+----------------+
| time           | total_Ni       | total_elastic_energy | total_free_energy | total_o        | total_solute   |
+----------------+----------------+----------------------+-------------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |         0.000000e+00 |      0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
+----------------+----------------+----------------------+-------------------+----------------+----------------+

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: MatSolverPackage petsc does not support matrix type mpiaij
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 
[0]PETSC ERROR: ./projects/moose/modules/combined/modules-opt on a arch-linux2-c-opt named moose-VM by moose Tue Jan 17 14:27:44 2017
[0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/openmpi_petsc-3.6.3/gcc-opt-superlu --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 CC=mpicc CXX=mpicxx FC=mpif90 F77=mpif77 F90=mpif90 CFLAGS="-fPIC -fopenmp" CXXFLAGS="-fPIC -fopenmp" FFLAGS="-fPIC -fopenmp" FCFLAGS="-fPIC -fopenmp" F90FLAGS="-fPIC -fopenmp" F77FLAGS="-fPIC -fopenmp"
[0]PETSC ERROR: #1 MatGetFactor() line 4158 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCSetUp_LU() line 125 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/ksp/pc/impls/factor/lu/lu.c
[0]PETSC ERROR: #3 PCSetUp() line 983 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 332 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 546 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #6 SNESSolve_NEWTONLS() line 233 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #7 SNESSolve() line 3906 in /tmp/src_temp_petsc-3.6.3-openmpi-gcc.FKWRdr/petsc-3.6.3/src/snes/interface/snes.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------


Alireza 



--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd2Mcqf32Gpgynq%2BSTOftSv8v0m%2BRmUQvv2Px5m_pvbYEQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


Jesse Carter

unread,
Jan 17, 2017, 3:40:57 PM1/17/17
to moose-users
The lu solver in PETSc only works in serial. Remove the "mpiexec -np 16" from your command line and run again.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

Kong, Fande

unread,
Jan 17, 2017, 3:41:25 PM1/17/17
to moose...@googlegroups.com
On Tue, Jan 17, 2017 at 1:34 PM, Alireza Toghraee <at...@mst.edu> wrote:

Thank you Fande for the help. Should I add the "-pc_type lu" at the end of the running command in terminal ( I think that's what you meant)? or should I add it in the preconditioning section of input file?
 

You could do either way. You need to use superlu_dist because you are running in parallel. Please use the options "-pc_type lu -pc_factor_mat_solver_package superlu_dist", and report back the residual history.

Thanks,

Fande,

Kong, Fande

unread,
Jan 17, 2017, 3:47:37 PM1/17/17
to moose...@googlegroups.com
On Tue, Jan 17, 2017 at 1:40 PM, Jesse Carter <jesse....@gmail.com> wrote:

The lu solver in PETSc only works in serial. Remove the "mpiexec -np 16" from your command line and run again.


The native LU implementation in PETSc is serial, but we could use the parallel version through other packages such as supderlu_dist, and MUMPS.

Fande,
 

Alireza Toghraee

unread,
Jan 17, 2017, 3:54:53 PM1/17/17
to moose...@googlegroups.com
Thank you again guys. I added -pc_type lu -pc_factor_mat_solver_package superlu_dist  and it is converging right now. Is it possible to explain here what was the problem and how can I learn how to run in different situations? I am asking this because I have to run this code on the HPC cluster after debugging and I wanted to know if I should make other changes for running the code in HPC.

Thanks again,

Alireza,

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd12L3q54VU9kiAjYHG60KGS3YAcOKkrYNyBLh%2B8RHbHnA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Kong, Fande

unread,
Jan 17, 2017, 3:59:29 PM1/17/17
to moose...@googlegroups.com
On Tue, Jan 17, 2017 at 1:54 PM, Alireza Toghraee <at...@mst.edu> wrote:

Thank you again guys. I added -pc_type lu -pc_factor_mat_solver_package superlu_dist  and it is converging right now.

Please report the residual back because we want to know if the Jacobian is formed correctly.

Thanks.

Alireza Toghraee

unread,
Jan 17, 2017, 4:00:01 PM1/17/17
to moose...@googlegroups.com
Hi Jesse,

Thanks for explaining. I am a little confused here because my code was running OK previously with the old  "  "mpiexec -np 16 ... " command and I just added another concentration "c_ni" to my model and it caused this problem. Is it possible that adding a new variable prevents the code from running in parallel? 

Thanks 

Alireza,

To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/09ee5624-14e8-4a43-ab33-a77677c27c77%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alireza Toghraee

unread,
Jan 17, 2017, 4:02:53 PM1/17/17
to moose...@googlegroups.com
Dear Fende,

Here is the output I got:

*** Warning ***
This simulation does not actually need the third derivatives of DerivativeFunctionMaterialBase elastic_free_energy_face
at /home/moose/projects/moose/modules/phase_field/src/materials/DerivativeFunctionMaterialBase.C, line 112


Time Step  0, time = 0
                dt = 0

Postprocessor Values:
+----------------+----------------+----------------------+-------------------+----------------+----------------+
| time           | total_Ni       | total_elastic_energy | total_free_energy | total_o        | total_solute   |
+----------------+----------------+----------------------+-------------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |         0.000000e+00 |      0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
+----------------+----------------+----------------------+-------------------+----------------+----------------+

 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!
 Solve Converged!

Time Step 25, time = 2.60382e-07
                dt = 4.7282e-08
    |residual|_2 of individual variables:
                    c_o:     4.62902e-14
                    w_o:     11602.8
                    c_fe:    8.88373e-14
                    w_fe:    1639.78
                    c_ni:    1.75324e-15
                    w_ni:    283.486
                    eta_Fe:  473781
                    eta_O:   2.46936e-08
                    eta_air: 398553
                    disp_x:  1.46092e-23
                    disp_y:  1.34285e-23

 0 Nonlinear |R| = 6.192339e+05
    |residual|_2 of individual variables:
                    c_o:     0.0052639
                    w_o:     0.0045791
                    c_fe:    0.0112921
                    w_fe:    0.000647148
                    c_ni:    8.11213e-05
                    w_ni:    0.000111879
                    eta_Fe:  1259.25
                    eta_O:   0.00331524
                    eta_air: 1898.65
                    disp_x:  8.67087e-20
                    disp_y:  5.44969e-20

 1 Nonlinear |R| = 2.278289e+03
    |residual|_2 of individual variables:
                    c_o:     6.00738e-08
                    w_o:     2.55205e-08
                    c_fe:    1.03391e-07
                    w_fe:    4.68365e-09
                    c_ni:    3.60967e-09
                    w_ni:    8.26221e-09
                    eta_Fe:  0.0826806
                    eta_O:   1.20187e-05
                    eta_air: 0.205606
                    disp_x:  8.67165e-20
                    disp_y:  5.45017e-20

 2 Nonlinear |R| = 2.216080e-01
    |residual|_2 of individual variables:
                    c_o:     5.012e-14
                    w_o:     1.35182e-08
                    c_fe:    9.69328e-14
                    w_fe:    2.0537e-08
                    c_ni:    2.21204e-15
                    w_ni:    2.10447e-08
                    eta_Fe:  3.13133e-08
                    eta_O:   1.55868e-08
                    eta_air: 3.13287e-08
                    disp_x:  3.08884e-23
                    disp_y:  2.82975e-23

 3 Nonlinear |R| = 5.702939e-08
 Solve Converged!

Outlier Variable Residual Norms:
  eta_Fe: 3.131333e-08
  eta_air: 3.132868e-08

Postprocessor Values:
+----------------+----------------+----------------------+-------------------+----------------+----------------+
| time           | total_Ni       | total_elastic_energy | total_free_energy | total_o        | total_solute   |
+----------------+----------------+----------------------+-------------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |         0.000000e+00 |      0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
|   2.603821e-07 |   1.545000e+02 |         5.599556e-48 |      7.522745e+02 |   6.230420e+01 |   7.139007e+01 |
+----------------+----------------+----------------------+-------------------+----------------+----------------+


Is that OK? 



--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd0k9KaibwyG9z_yW6-6m6FmwwYmud4m-mTo7jteouE7tA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Kong, Fande

unread,
Jan 17, 2017, 4:10:37 PM1/17/17
to moose...@googlegroups.com
I can not see the linear residual history because PhaseField guys overwrite the residual print(??).  I am sorry, and you have to rerun the code again with:

 -pc_type lu -pc_factor_mat_solver_package superlu_dist -snes_view -snes_monitor -ksp_monitor

 Fande,




On Tue, Jan 17, 2017 at 2:59 PM, Kong, Fande <fande...@inl.gov> wrote:


On Tue, Jan 17, 2017 at 1:54 PM, Alireza Toghraee <at...@mst.edu> wrote:

Thank you again guys. I added -pc_type lu -pc_factor_mat_solver_package superlu_dist  and it is converging right now.

Please report the residual back because we want to know if the Jacobian is formed correctly.

Thanks.
 
Is it possible to explain here what was the problem and how can I learn how to run in different situations? I am asking this because I have to run this code on the HPC cluster after debugging and I wanted to know if I should make other changes for running the code in HPC.

Thanks again,

Alireza,

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
 To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd0k9KaibwyG9z_yW6-6m6FmwwYmud4m-mTo7jteouE7tA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Derek Gaston

unread,
Jan 17, 2017, 4:14:26 PM1/17/17
to moose...@googlegroups.com
Anytime you're trying to understand convergence you really need to turn on the output of linear residuals!  They are printed by default so you must have "print_linear_residuals = false" in your Output block.  Remove that or set it to true.  BTW: this is preferable to using the "-ksp_monitor" PETSc option, you shouldn't use that.

There are two types of convergence in an inexact-Newton solver: linear and nonlinear.  If something isn't converging it can be because of either (or both!).

Derek

To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd0k9KaibwyG9z_yW6-6m6FmwwYmud4m-mTo7jteouE7tA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAHDS0wq5Avq_Lsy_9sGp%2BKyeu33o%3Df3aDNLRLiB1YN7ABdTMcg%40mail.gmail.com.

Slaughter, Andrew E

unread,
Jan 17, 2017, 4:17:15 PM1/17/17
to moose...@googlegroups.com
Do you have 'interval=25' in your input file somewhere? That is what looks like this is doing to me.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAHDS0wq5Avq_Lsy_9sGp%2BKyeu33o%3Df3aDNLRLiB1YN7ABdTMcg%40mail.gmail.com.

Cody Permann

unread,
Jan 17, 2017, 4:42:37 PM1/17/17
to moose...@googlegroups.com
Alireza,

I want to clear up a few things about this whole thread before you end up with a bunch of PETSc options in your input file that don't mean anything to you and worse, severely restrict the scalability or ability to run your inputs using large processor counts.

Fande was trying to ascertain whether or not your problem with adding "just one" more variable was due to a problem with your Jacobian. There are several ways to do this, he started you down the path of using a direct solution method "lu" which will get the job done but isn't the best solution if you want to run larger problems. As we pointed out "lu" is a serial preconditioner so you can't run it in parallel. It has nothing to do with adding the extra term in your equation.

Fande then suggested that you run a special version of "lu" that's made to run in parallel (superLU). It however has many of it's own issues and has limited scalability. 

At the end of the day, it's important to understand this is all debugging to figure out what's wrong with your problem. Just because it's converging now doesn't mean that you have a solution to your problem. We just know what the problem is now. 

Your new nonlinear term is enough to require a different solution strategy or better preconditioning. The latter is really what you are after if you want your code to be robust. Have you implemented Jacobians for your new kernels? Are you using any preconditioning at all yet? These are some of things you should explore moving forward. Also, let's figure out why your screen output isn't printing residuals using normal MOOSE options. This will help keep everyone trying to troubleshoot your problem sane.

Thanks,
Cody

To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd0k9KaibwyG9z_yW6-6m6FmwwYmud4m-mTo7jteouE7tA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAHDS0wq5Avq_Lsy_9sGp%2BKyeu33o%3Df3aDNLRLiB1YN7ABdTMcg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAGWYgTkO%3DMkMp9maMRkdez%2B2%2BR6Gum%3DLbMi%2B0c1R%2BgS%3DtE%2B%2BAA%40mail.gmail.com.

Alireza Toghraee

unread,
Jan 17, 2017, 4:43:42 PM1/17/17
to moose...@googlegroups.com
Dear Fande,

Here is the command and the result. It seems this time it does 


[moose][~]> mpiexec -np 16 ./projects/moose/modules/combined/modules-opt -i projects/moose/Alireza/FeO_Nit/FeO_Nitrogen_VM.i -pc_type lu -pc_factor_mat_solver_package superlu_dist -snes_view -snes_monitor -ksp_monitor

Framework Information:
MOOSE version:           git commit 11d8d1c on 2016-07-20
PETSc Version:           3.6.3
Current Time:            Tue Jan 17 15:28:44 2017
Executable Timestamp:    Sat Jul 23 11:56:29 2016

Parallelism:
  Num Processors:          16
  Num Threads:             1

Mesh: 
  Parallel Type:           replicated
  Mesh Dimension:          2
  Spatial Dimension:       2
  Nodes:                   
    Total:                 2701
    Local:                 191
  Elems:                   
    Total:                 2592
    Local:                 161
  Num Subdomains:          3
  Num Partitions:          16
  Partitioner:             metis

Nonlinear System:
  Num DOFs:                29711
  Num Local DOFs:          2101
  Variables:               { "c_o" "w_o" "c_fe" "w_fe" "c_ni" "w_ni" "eta_Fe" "eta_O" "eta_air" "disp_x" 
                             "disp_y" } 
  Finite Element Types:    "LAGRANGE" 
  Approximation Orders:    "FIRST" 

Auxiliary System:
  Num DOFs:                31104
  Num Local DOFs:          1932
  Variables:               { "local_energy" "cross_energy" } { "sigma11" "sigma22" "sigma12" } { "Fe" 
                             "d^2gc/deta_O^2" "d^2gc/deta_Odeta_air" "d^2gc/deta_air^2" "dgc/deta_O" 
                             "dgc/deta_air" "gc" } 
  Finite Element Types:    "MONOMIAL" "MONOMIAL" "MONOMIAL" 
  Approximation Orders:    "CONSTANT" "CONSTANT" "CONSTANT" 

Execution Information:
  Executioner:             Transient
  TimeStepper:             IterationAdaptiveDT
  Solver Mode:             Preconditioned JFNK





*** Warning ***
This simulation does not actually need the third derivatives of DerivativeFunctionMaterialBase elastic_free_energy
at /home/moose/projects/moose/modules/phase_field/src/materials/DerivativeFunctionMaterialBase.C, line 112


.
.
.

and then is goes to :


Time Step  0, time = 0
                dt = 0

Postprocessor Values:
+----------------+----------------+----------------------+-------------------+----------------+----------------+
| time           | total_Ni       | total_elastic_energy | total_free_energy | total_o        | total_solute   |
+----------------+----------------+----------------------+-------------------+----------------+----------------+
|   0.000000e+00 |   0.000000e+00 |         0.000000e+00 |      0.000000e+00 |   0.000000e+00 |   0.000000e+00 |
+----------------+----------------+----------------------+-------------------+----------------+----------------+

  0 SNES Function norm 1.328320437091e+08 
    0 KSP Residual norm 1.328320437091e+08 
    1 KSP Residual norm 1.056222458763e+03 
  1 SNES Function norm 3.983499112866e+04 
    0 KSP Residual norm 3.983499112866e+04 
    1 KSP Residual norm 2.213835532513e+00 
  2 SNES Function norm 3.019050002242e-02 
    0 KSP Residual norm 3.019050002242e-02 
    1 KSP Residual norm 2.454313606395e-06 
  3 SNES Function norm 3.964948275030e-07 
SNES Object: 16 MPI processes
  type: newtonls
  maximum iterations=10, maximum function evaluations=10000
  tolerances: relative=1e-10, absolute=0.01, solution=1e-50
  total number of linear solver iterations=3
  total number of function evaluations=10
  norm schedule ALWAYS
  SNESLineSearch Object:   16 MPI processes
    type: bt
      interpolation: cubic
      alpha=1.000000e-04
    maxstep=1.000000e+08, minlambda=1.000000e-12
    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
    maximum iterations=40
  KSP Object:   16 MPI processes
    type: gmres
      GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      GMRES: happy breakdown tolerance 1e-30
    maximum iterations=15, initial guess is zero
    tolerances:  relative=0.0001, absolute=1e-50, divergence=10000
    right preconditioning
    using UNPRECONDITIONED norm type for convergence test
  PC Object:   16 MPI processes
    type: lu
      LU: out-of-place factorization
      tolerance for zero pivot 2.22045e-14
      matrix ordering: natural
      factor fill ratio given 0, needed 0
        Factored matrix follows:
          Mat Object:           16 MPI processes
            type: mpiaij
            rows=29711, cols=29711
            package used to perform factorization: superlu_dist
            total: nonzeros=0, allocated nonzeros=0
            total number of mallocs used during MatSetValues calls =0
              SuperLU_DIST run parameters:
                Process grid nprow 4 x npcol 4 
                Equilibrate matrix TRUE 
                Matrix input mode 1 
                Replace tiny pivots TRUE 
                Use iterative refinement FALSE 
                Processors in row 4 col partition 4 
                Row permutation LargeDiag 
                Column permutation METIS_AT_PLUS_A
                Parallel symbolic factorization FALSE 
                Repeated factorization SamePattern_SameRowPerm
    linear system matrix followed by preconditioner matrix:
    Mat Object:     16 MPI processes
      type: mffd
      rows=29711, cols=29711
        Matrix-free approximation:
          err=1.49012e-08 (relative error in function evaluation)
          Using wp compute h routine
              Does not compute normU
    Mat Object:    ()     16 MPI processes
      type: mpiaij
      rows=29711, cols=29711
      total: nonzeros=2.96704e+06, allocated nonzeros=3.12349e+06
      total number of mallocs used during MatSetValues calls =0
        using I-node (on process 0) routines: found 573 nodes, limit used is 5
 Solve Converged!
  0 SNES Function norm 2.228903752396e+06 
    0 KSP Residual norm 2.228903752396e+06 
    1 KSP Residual norm 1.273719430289e+00 
  1 SNES Function norm 8.646359414169e+00 
    0 KSP Residual norm 8.646359414164e+00 
    1 KSP Residual norm 4.651591407172e-04 
  2 SNES Function norm 6.952018924993e-05 
SNES Object: 16 MPI processes
  type: newtonls
  maximum iterations=10, maximum function evaluations=10000
  tolerances: relative=1e-10, absolute=0.01, solution=1e-50
  total number of linear solver iterations=2
  total number of function evaluations=7
  norm schedule ALWAYS
  SNESLineSearch Object:   16 MPI processes
    type: bt
      interpolation: cubic
      alpha=1.000000e-04
    maxstep=1.000000e+08, minlambda=1.000000e-12
    tolerances: relative=1.000000e-08, absolute=1.000000e-15, lambda=1.000000e-08
    maximum iterations=40
  KSP Object:   16 MPI processes
    type: gmres
      GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
      GMRES: happy breakdown tolerance 1e-30
    maximum iterations=15, initial guess is zero
    tolerances:  relative=0.0001, absolute=1e-50, divergence=10000
    right preconditioning
    using UNPRECONDITIONED norm type for convergence test
  PC Object:   16 MPI processes
    type: lu
      LU: out-of-place factorization
      tolerance for zero pivot 2.22045e-14
      matrix ordering: natural
      factor fill ratio given 0, needed 0
        Factored matrix follows:
          Mat Object:           16 MPI processes
            type: mpiaij
            rows=29711, cols=29711
            package used to perform factorization: superlu_dist
            total: nonzeros=0, allocated nonzeros=0
            total number of mallocs used during MatSetValues calls =0
              SuperLU_DIST run parameters:
                Process grid nprow 4 x npcol 4 
                Equilibrate matrix TRUE 



It is part of the result I could capture ( since it keeps updating and adding lines).
 Here is another thing when this code was running and converging I opened another terminal and run the same command but it did not converge. While it can be because both codes are trying to access same files in same directory simultaneously, is it possible to fail in convergence due to number of cores allocated/available ?

P.S. you all again for all the support and comments. 


To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd2rLUpwVuoFvC3e%2BBRnwxZude5gyZjJrCxV-OnMytrFzQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Kong, Fande

unread,
Jan 17, 2017, 5:21:07 PM1/17/17
to moose...@googlegroups.com
The original residual is hug here. If possible, please scale the problem, and it sometimes help the converge a lot.
 
    0 KSP Residual norm 1.328320437091e+08 
    1 KSP Residual norm 1.056222458763e+03 

Most of linear solves with a direct solver converge within two steps, which implies that your Jacobian is formed correctly. It is good.  Note that: LU is used for debugging only here. It should not be used if you are going to solve large-scale problems with a large of processor cores.


In all, your Jacobian looks right. The nonlinear converges good.  There are several things you could do:

(1) Scale the problem so that the original residual becomes small, if possible.

(2) If the problem has been scaled, you could loose the converge tolerance by putting the following two parameters into the executioner block:

  nl_rel_tol = 1.0e-6
  nl_abs_tol = 1.0e-10

(3) A scalable preconditioner. If you are going to solve problems with millions of unknowns with more than hundreds of cores, you CAN NOT use the LU and we need a scalable  preconditioner. It is not easy, usually, and can not be done via several emails. Could you email me off-list and tell me a little bit about physics, and then we try some classical preconditioners.

Thanks,

Fande,

 

Derek Gaston

unread,
Jan 17, 2017, 6:09:54 PM1/17/17
to moose...@googlegroups.com
Problem may not be a scaling issue.  Or are you suggesting that it's an ill-conditioned matrix since there is only a 5 order of magnitude drop using LU?

We need to see what the residuals look like when they go back to the original preconditioning.  The original preconditioning was ASM with LU... which is not really a great choice.

Alireza: can you run the problem again with the original preconditioning and show us the linear/nonlinear residuals?

Derek

Fande,

 


To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd26sGu2OEjepA%3DFj4HHY8U9mJEGR9LgVOc2eC2gqXFeuQ%40mail.gmail.com.

Kong, Fande

unread,
Jan 17, 2017, 6:22:37 PM1/17/17
to moose...@googlegroups.com
On Tue, Jan 17, 2017 at 4:09 PM, Derek Gaston <frie...@gmail.com> wrote:
Problem may not be a scaling issue.  Or are you suggesting that it's an ill-conditioned matrix since there is only a 5 order of magnitude drop using LU?

Yes, the matrix could be ill-conditioned. But I think the "small" drop is caused by the difference between the Jacobian approximated by the finite difference (because we are using the matrix-free Newton) and the preconditioning matrix formed analytically by the user. 


Fande,

 


To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

Alireza Toghraee

unread,
Jan 17, 2017, 9:53:30 PM1/17/17
to moose...@googlegroups.com
Dear Derek,

By running with original preconditioning do you mean omitting the " -pc_type lu"  form the the execution in the command line in terminal?  In other words, should I type 

"[moose][~]> mpiexec -np 16 ./projects/moose/modules/combined/modules-opt -i projects/moose/Alireza/FeO_Nit/FeO_Nitrogen_VM.i  -pc_factor_mat_solver_package superlu_dist -snes_view -snes_monitor -ksp_monitor"

in the terminal? 

Fande,

 


To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd26sGu2OEjepA%3DFj4HHY8U9mJEGR9LgVOc2eC2gqXFeuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAFfxPjrdS433Z8r68-ibwemfqoeKVGeLYTcyJUoOSzNnn9skXw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Alireza Toghraee

unread,
Jan 17, 2017, 10:09:09 PM1/17/17
to moose...@googlegroups.com
Dear Cody,

Thanks for the explanations. I am going to look at the preconditioning examples in the MOOSE website and try to get more educated about that and probably get back to you if I have questions. Meanwhile , I looked at my input file and it seems I am using SMP type as my preconditioning.

[Preconditioning]
  # This preconditioner makes sure the Jacobian Matrix is fully populated. Our
  # kernels compute all Jacobian matrix entries.
  # This allows us to use the Newton solver below.
  [./SMP]
    type = SMP
    full = true
  [../]
[]

[Executioner]
  type = Transient
  scheme = 'bdf2'
  solve_type = 'PJFNK'

  petsc_options_iname = '-pc_type -ksp_grmres_restart -sub_ksp_type -sub_pc_type -pc_asm_overlap'
  petsc_options_value = 'asm      31                  preonly       lu           2'

  l_max_its = 15
  l_tol = 1.0e-4

  nl_max_its = 10
  nl_rel_tol = 1.0e-10
  nl_abs_tol = 1.0e-2

To answer why my MOOSE screen output does not print residuals, I looked again and there is nothing like   "print_linear_residuals = false" in the Output block. However, to prevent huge Exudos  files, I asked the code to output results every 25 interval. Is that the reason?


[Debug]
  show_var_residual_norms = true
[]

[Outputs]
  interval = 25
  gnuplot = true
  csv = true
  checkpoint = true
  #exodus = true
  [./exodus]
    type = Exodus
    show = 'c_o c_fe c_ni eta_Fe eta_O eta_air sigma11 sigma22 sigma12 disp_x disp_y local_energy cross_energy total_elastic_energy '
    execute_on = 'failed initial nonlinear timestep_end final'
  [../]
  [./console]
    type = Console
    execute_on = 'FAILED INITIAL NONLINEAR TIMESTEP_END final'
  [../]
  print_perf_log = true
[]


Bests,
Alireza

To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAGWYgTkO%3DMkMp9maMRkdez%2B2%2BR6Gum%3DLbMi%2B0c1R%2BgS%3DtE%2B%2BAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK06EuWpGXxDmR0nDj19u3iZFCkZ82jSG8NDjutZq9Ao5kbd%2BA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Cody Permann

unread,
Jan 18, 2017, 9:55:30 AM1/18/17
to moose...@googlegroups.com
On Tue, Jan 17, 2017 at 8:09 PM Alireza Toghraee <at...@mst.edu> wrote:
Dear Cody,

Thanks for the explanations. I am going to look at the preconditioning examples in the MOOSE website and try to get more educated about that and probably get back to you if I have questions. Meanwhile , I looked at my input file and it seems I am using SMP type as my preconditioning.

Yes you are, but this method uses the "optional" computeQpOffDiagJacobian() method in your kernels quite extensively to fill in the full Jacobian. If you haven't implemented that for your physics, you'll probably have just block diagonal preconditioning at best which may not be good enough depending on your physics. We have information and examples about this in the training materials.
 
[Preconditioning]
  # This preconditioner makes sure the Jacobian Matrix is fully populated. Our
  # kernels compute all Jacobian matrix entries.
  # This allows us to use the Newton solver below.
  [./SMP]
    type = SMP
    full = true
  [../]
[]

[Executioner]
  type = Transient
  scheme = 'bdf2'
  solve_type = 'PJFNK'

  petsc_options_iname = '-pc_type -ksp_grmres_restart -sub_ksp_type -sub_pc_type -pc_asm_overlap'
  petsc_options_value = 'asm      31                  preonly       lu           2'

Looks like lots of copy any paste here. These options may be good for one problem but terrible for another.
 

  l_max_its = 15
  l_tol = 1.0e-4

  nl_max_its = 10
  nl_rel_tol = 1.0e-10
  nl_abs_tol = 1.0e-2

To answer why my MOOSE screen output does not print residuals, I looked again and there is nothing like   "print_linear_residuals = false" in the Output block. However, to prevent huge Exudos  files, I asked the code to output results every 25 interval. Is that the reason?

You can control the screen output and the console output separately. If you put "interval" in the top level block like you did before, it'll apply to all "Outputs" so yes, this is why you weren't seeing residuals. Please drop the "-ksp_monitor" option and put the interval inside of your exodus block. You can also remove all of that "execute_on" stuff. 
 
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "moose-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/moose-users/8iUmbIGHBe4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to moose-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK06EuWpGXxDmR0nDj19u3iZFCkZ82jSG8NDjutZq9Ao5kbd%2BA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Alireza Toghraee
Graduate Research Assistant 
Aerospace and Mechanical Eng. Department


--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages