Unintended consequence of CHKERRQ()

3 views
Skip to first unread message

Gautam Bisht

unread,
Dec 3, 2014, 11:35:57 AM12/3/14
to pflotr...@googlegroups.com
The issue is that when SNES diverges with a "not-a-number or infinite", the code crashes without giving timestepper_BE a chance to reduce the time-step. But, if CHKERRQ() is commented out after call to SNESSolve(), the code is able to reduce dt sufficiently to converge due to "itol_res" (PFLOTRAN's custom convergence reason == inorm_residual < solver%newton_inf_res_tol). Attached below is inputdeck (with an outrages initial dt = 1yr to illustrate the issue). Apart from commenting out CHKERRQ(), is there an alternate way to let timestepper_BE reduce dt?

Below are relevant information to reproduce the error.

>hg tip
changeset:   8541:fa2d4295aa14
tag:         tip
user:        Nathan Collier <nathanie...@gmail.com>
date:        Wed Dec 03 08:52:00 2014 -0500
summary:     made output process-model identifier lines to be of uniform length (80 characters). Yeah it is a little OCD of me.

>./pflotran -pflotranin da.in

== TH FLOW =====================================================================
  1 2f: 2.55E+00 2x: 1.63E+07 2s: 1.72E+07 ir: 1.49E+00 iu: 3.10E+06 rsn:   0
  2 2f: 1.88E-01 2x: 1.57E+14 2s: 1.57E+14 ir: 5.33E-02 iu: 4.37E+13 rsn:   0
  3 2f: 1.07E-01 2x: 1.60E+14 2s: 4.28E+12 ir: 2.27E-02 iu: 9.90E+11 rsn:   0
  4 2f: 1.24E+01 2x: 1.60E+14 2s: 9.53E+11 ir: 1.16E+01 iu: 7.15E+11 rsn:   0
  5 2f: 1.29E+06 2x: 7.75E+20 2s: 7.75E+20 ir: 1.15E+06 iu: 7.21E+20 rsn: max_norm
 -> Cut time step: snes=-10 icut=  1[  0] t=  0.00000E+00 dt=  5.00000E-01
  1 2f: 2.22E+00 2x: 1.38E+07 2s: 1.47E+07 ir: 1.47E+00 iu: 3.06E+06 rsn:   0
  2 2f: 2.94E-01 2x: 1.11E+14 2s: 1.11E+14 ir: 1.44E-01 iu: 4.05E+13 rsn:   0
  3 2f: 6.05E+20 2x: 2.13E+14 2s: 1.79E+14 ir: 4.44E+20 iu: 8.93E+13 rsn: max_norm
 -> Cut time step: snes=-10 icut=  2[  0] t=  0.00000E+00 dt=  2.50000E-01
  1 2f: 2.17E+00 2x: 1.12E+07 2s: 1.21E+07 ir: 1.44E+00 iu: 3.01E+06 rsn:   0
  2 2f: 5.55E-01 2x: 8.08E+13 2s: 8.08E+13 ir: 4.07E-01 iu: 3.66E+13 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 50 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:16:17 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #1 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #2 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #3 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #4 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD 
with errorcode 72.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------


> hg diff
diff -r fa2d4295aa14 src/pflotran/timestepper_BE.F90
--- a/src/pflotran/timestepper_BE.F90 Wed Dec 03 08:52:00 2014 -0500
+++ b/src/pflotran/timestepper_BE.F90 Wed Dec 03 08:22:35 2014 -0800
@@ -310,7 +310,7 @@
     call PetscTime(log_start_time, ierr);CHKERRQ(ierr)
 
     call SNESSolve(solver%snes,PETSC_NULL_OBJECT, &
-                   process_model%solution_vec,ierr);CHKERRQ(ierr)
+                   process_model%solution_vec,ierr);!CHKERRQ(ierr)
 
     call PetscTime(log_end_time, ierr);CHKERRQ(ierr)


>./pflotran -pflotranin da.in

== TH FLOW =====================================================================
  1 2f: 2.55E+00 2x: 1.63E+07 2s: 1.72E+07 ir: 1.49E+00 iu: 3.10E+06 rsn:   0
  2 2f: 1.88E-01 2x: 1.57E+14 2s: 1.57E+14 ir: 5.33E-02 iu: 4.37E+13 rsn:   0
  3 2f: 1.07E-01 2x: 1.60E+14 2s: 4.28E+12 ir: 2.27E-02 iu: 9.90E+11 rsn:   0
  4 2f: 1.24E+01 2x: 1.60E+14 2s: 9.53E+11 ir: 1.16E+01 iu: 7.15E+11 rsn:   0
  5 2f: 1.29E+06 2x: 7.75E+20 2s: 7.75E+20 ir: 1.15E+06 iu: 7.21E+20 rsn: max_norm
 -> Cut time step: snes=-10 icut=  1[  0] t=  0.00000E+00 dt=  5.00000E-01
  1 2f: 2.22E+00 2x: 1.38E+07 2s: 1.47E+07 ir: 1.47E+00 iu: 3.06E+06 rsn:   0
  2 2f: 2.94E-01 2x: 1.11E+14 2s: 1.11E+14 ir: 1.44E-01 iu: 4.05E+13 rsn:   0
  3 2f: 6.05E+20 2x: 2.13E+14 2s: 1.79E+14 ir: 4.44E+20 iu: 8.93E+13 rsn: max_norm
 -> Cut time step: snes=-10 icut=  2[  0] t=  0.00000E+00 dt=  2.50000E-01
  1 2f: 2.17E+00 2x: 1.12E+07 2s: 1.21E+07 ir: 1.44E+00 iu: 3.01E+06 rsn:   0
  2 2f: 5.55E-01 2x: 8.08E+13 2s: 8.08E+13 ir: 4.07E-01 iu: 3.66E+13 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 50 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:17:35 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #1 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #2 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #3 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #4 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
 -> Cut time step: snes=  0 icut=  3[  0] t=  0.00000E+00 dt=  1.25000E-01
  1 2f: 2.12E+00 2x: 9.08E+06 2s: 9.85E+06 ir: 1.41E+00 iu: 2.94E+06 rsn:   0
  2 2f: 7.89E-01 2x: 5.95E+13 2s: 5.95E+13 ir: 6.28E-01 iu: 3.20E+13 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 34 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:17:35 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #7 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #8 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #9 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #10 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #12 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
 -> Cut time step: snes=  0 icut=  4[  0] t=  0.00000E+00 dt=  6.25000E-02
  1 2f: 2.04E+00 2x: 7.38E+06 2s: 8.06E+06 ir: 1.36E+00 iu: 2.86E+06 rsn:   0
  2 2f: 5.60E+04 2x: 4.28E+13 2s: 4.28E+13 ir: 4.47E+04 iu: 2.66E+13 rsn:   0
  3 2f: 1.16E+18 2x: 4.91E+13 2s: 2.30E+13 ir: 9.47E+17 iu: 1.69E+13 rsn: max_norm
 -> Cut time step: snes=-10 icut=  5[  0] t=  0.00000E+00 dt=  3.12500E-02
  1 2f: 1.94E+00 2x: 6.00E+06 2s: 6.57E+06 ir: 1.30E+00 iu: 2.74E+06 rsn:   0
  2 2f: 6.76E+04 2x: 2.91E+13 2s: 2.91E+13 ir: 5.18E+04 iu: 2.06E+13 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 14 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:17:35 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #13 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #14 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #15 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #16 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #18 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
 -> Cut time step: snes=  0 icut=  6[  0] t=  0.00000E+00 dt=  1.56250E-02
  1 2f: 1.81E+00 2x: 4.84E+06 2s: 5.31E+06 ir: 1.21E+00 iu: 2.58E+06 rsn:   0
  2 2f: 1.56E+04 2x: 1.81E+13 2s: 1.81E+13 ir: 1.22E+04 iu: 1.43E+13 rsn:   0
  3 2f: 9.46E+15 2x: 1.99E+13 2s: 7.41E+12 ir: 7.34E+15 iu: 5.84E+12 rsn: max_norm
 -> Cut time step: snes=-10 icut=  7[  0] t=  0.00000E+00 dt=  7.81250E-03
  1 2f: 1.64E+00 2x: 3.85E+06 2s: 4.21E+06 ir: 1.11E+00 iu: 2.37E+06 rsn:   0
  2 2f: 1.37E+04 2x: 9.82E+12 2s: 9.82E+12 ir: 1.02E+04 iu: 8.52E+12 rsn:   0
  3 2f: 5.35E+14 2x: 1.86E+13 2s: 1.65E+13 ir: 3.78E+14 iu: 1.62E+13 rsn: max_norm
 -> Cut time step: snes=-10 icut=  8[  0] t=  0.00000E+00 dt=  3.90625E-03
  1 2f: 1.42E+00 2x: 3.01E+06 2s: 3.25E+06 ir: 9.67E-01 iu: 2.09E+06 rsn:   0
  2 2f: 6.03E+03 2x: 4.39E+12 2s: 4.39E+12 ir: 4.38E+03 iu: 4.08E+12 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 30 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:17:35 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #19 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #20 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #21 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #22 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #23 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #24 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
 -> Cut time step: snes=  0 icut=  9[  0] t=  0.00000E+00 dt=  1.95312E-03
  1 2f: 1.16E+00 2x: 2.30E+06 2s: 2.42E+06 ir: 8.00E-01 iu: 1.76E+06 rsn:   0
  2 2f: 4.32E+03 2x: 1.47E+12 2s: 1.47E+12 ir: 3.07E+03 iu: 1.42E+12 rsn:   0
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Floating point exception
[0]PETSC ERROR: Vec entry at local location 4 is not-a-number or infinite at end of function: Parameter number 3
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.1, unknown 
[0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov by gbisht Wed Dec  3 08:17:35 2014
[0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
[0]PETSC ERROR: #25 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
[0]PETSC ERROR: #26 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
[0]PETSC ERROR: #27 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
[0]PETSC ERROR: #28 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
[0]PETSC ERROR: #29 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #30 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
 -> Cut time step: snes=  0 icut= 10[  0] t=  0.00000E+00 dt=  9.76562E-04
  1 2f: 8.79E-01 2x: 1.73E+06 2s: 1.70E+06 ir: 6.11E-01 iu: 1.37E+06 rsn:   0
  2 2f: 1.34E+03 2x: 3.17E+11 2s: 3.17E+11 ir: 9.30E+02 iu: 3.14E+11 rsn:   0
  3 2f: 3.99E+13 2x: 6.13E+11 2s: 3.04E+11 ir: 3.26E+13 iu: 2.98E+11 rsn: max_norm
 -> Cut time step: snes=-10 icut= 11[  0] t=  0.00000E+00 dt=  4.88281E-04
  1 2f: 5.96E-01 2x: 1.33E+06 2s: 1.10E+06 ir: 4.19E-01 iu: 9.69E+05 rsn:   0
  2 2f: 4.60E+04 2x: 3.54E+10 2s: 3.54E+10 ir: 3.25E+04 iu: 3.54E+10 rsn:   0
  3 2f: 9.64E+15 2x: 3.91E+11 2s: 3.55E+11 ir: 7.87E+15 iu: 3.55E+11 rsn: max_norm
 -> Cut time step: snes=-10 icut= 12[  0] t=  0.00000E+00 dt=  2.44141E-04
  1 2f: 3.54E-01 2x: 1.11E+06 2s: 6.36E+05 ir: 2.50E-01 iu: 5.95E+05 rsn:   0
  2 2f: 5.42E+04 2x: 1.33E+09 2s: 1.33E+09 ir: 3.93E+04 iu: 1.33E+09 rsn:   0
  3 2f: 3.51E+06 2x: 1.75E+10 2s: 1.75E+10 ir: 2.87E+06 iu: 1.75E+10 rsn: max_norm
 -> Cut time step: snes=-10 icut= 13[  0] t=  0.00000E+00 dt=  1.22070E-04
  1 2f: 5.74E-02 2x: 1.04E+06 2s: 3.17E+05 ir: 4.02E-02 iu: 3.07E+05 rsn:   0
  2 2f: 3.60E-02 2x: 6.66E+09 2s: 6.66E+09 ir: 2.88E-02 iu: 6.66E+09 rsn:   0
  3 2f: 5.16E+02 2x: 2.91E+08 2s: 6.95E+09 ir: 3.31E+02 iu: 6.95E+09 rsn:   0
  4 2f: 7.20E+04 2x: 7.65E+15 2s: 7.65E+15 ir: 6.73E+04 iu: 7.65E+15 rsn:   0
  5 2f: 4.01E+12 2x: 3.15E+16 2s: 3.92E+16 ir: 3.27E+12 iu: 3.92E+16 rsn: max_norm
 -> Cut time step: snes=-10 icut= 14[  0] t=  0.00000E+00 dt=  6.10352E-05
  1 2f: 8.13E-02 2x: 1.04E+06 2s: 1.33E+05 ir: 8.02E-02 iu: 1.31E+05 rsn:   0
  2 2f: 1.94E+02 2x: 1.14E+08 2s: 1.14E+08 ir: 1.33E+02 iu: 1.14E+08 rsn:   0
  3 2f: 2.16E-01 2x: 4.22E+08 2s: 5.36E+08 ir: 1.79E-01 iu: 5.36E+08 rsn:   0
  4 2f: 3.21E+28 2x: 1.07E+19 2s: 1.07E+19 ir: 2.62E+28 iu: 1.07E+19 rsn: max_norm
 -> Cut time step: snes=-10 icut= 15[  0] t=  0.00000E+00 dt=  3.05176E-05
  1 2f: 1.13E-05 2x: 1.05E+06 2s: 4.69E+04 ir: 1.13E-05 iu: 4.67E+04 rsn:   0
  2 2f: 1.21E-10 2x: 1.05E+06 2s: 6.87E+02 ir: 1.05E-10 iu: 6.87E+02 rsn: itol_res

 Step      1 Time=  3.05176E-05 Dt=  3.05176E-05 [y] snes_conv_reason:   10
  newton =  47 [      47] linear =    52 [        52] cuts = 15 [  15]
  --> SNES Linear/Non-Linear Iterations =            2  /            2
  --> SNES Residual:   1.209152E-10  1.209152E-12  1.045172E-10
  --> max chng: dpmx=   4.5981E+04 dtmpmx=   1.6500E+00
 

== TH FLOW =====================================================================
  1 2f: 1.62E-01 2x: 1.04E+06 2s: 7.34E+04 ir: 1.60E-01 iu: 7.27E+04 rsn:   0
  2 2f: 7.18E+01 2x: 6.44E+07 2s: 6.44E+07 ir: 4.68E+01 iu: 6.44E+07 rsn:   0
  3 2f: 2.52E-01 2x: 5.48E+07 2s: 1.19E+08 ir: 2.02E-01 iu: 1.19E+08 rsn:   0
  4 2f: 1.29E+09 2x: 1.44E+18 2s: 1.44E+18 ir: 1.29E+09 iu: 1.44E+18 rsn: max_norm
 -> Cut time step: snes=-10 icut=  1[ 15] t=  3.05176E-05 dt=  1.52588E-05
  1 2f: 3.23E-01 2x: 1.04E+06 2s: 3.15E+04 ir: 3.19E-01 iu: 3.13E+04 rsn:   0
  2 2f: 1.17E-03 2x: 3.08E+06 2s: 2.83E+06 ir: 1.09E-03 iu: 2.83E+06 rsn:   0
  3 2f: 3.23E-01 2x: 1.04E+06 2s: 2.83E+06 ir: 3.19E-01 iu: 2.83E+06 rsn:   0
  4 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
  5 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
  6 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
  7 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
  8 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
  9 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 10 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 11 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 12 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 13 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 14 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 15 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 16 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 17 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 18 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 19 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 20 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 21 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 22 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 23 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 24 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 25 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 26 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 27 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 28 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 29 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 30 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 31 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 32 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 33 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 34 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 35 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 36 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 37 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 38 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 39 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 40 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 41 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 42 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 43 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 44 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 45 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 46 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 47 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 48 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 49 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 50 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 51 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 52 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 53 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 54 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 55 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 56 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 57 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 58 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 59 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 60 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 61 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 62 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 63 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 64 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 65 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 66 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 67 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 68 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 69 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 70 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 71 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 72 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 73 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 74 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 75 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 76 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 77 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 78 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 79 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 80 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 81 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 82 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 83 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 84 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 85 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 86 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 87 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 88 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 89 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 90 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 91 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 92 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 93 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 94 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 95 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 96 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 97 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
 98 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 99 2f: 3.23E-01 2x: 1.04E+06 2s: 2.89E+06 ir: 3.19E-01 iu: 2.89E+06 rsn:   0
100 2f: 1.18E-03 2x: 3.14E+06 2s: 2.89E+06 ir: 1.10E-03 iu: 2.89E+06 rsn:   0
 -> Cut time step: snes= -5 icut=  2[ 15] t=  3.05176E-05 dt=  7.62939E-06
  1 2f: 6.23E-01 2x: 1.04E+06 2s: 1.39E+04 ir: 6.15E-01 iu: 1.38E+04 rsn:   0
  2 2f: 6.18E-04 2x: 1.06E+06 2s: 1.15E+05 ir: 5.25E-04 iu: 1.15E+05 rsn:   0
  3 2f: 6.23E-01 2x: 1.04E+06 2s: 1.15E+05 ir: 6.15E-01 iu: 1.15E+05 rsn:   0
  4 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
  5 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
  6 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
  7 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
  8 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
  9 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 10 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 11 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 12 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 13 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 14 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 15 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 16 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 17 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 18 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 19 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 20 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 21 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 22 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 23 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 24 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 25 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 26 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 27 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 28 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 29 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 30 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 31 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 32 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 33 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 34 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 35 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 36 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 37 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 38 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 39 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 40 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 41 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 42 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 43 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 44 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 45 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 46 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 47 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 48 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 49 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 50 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 51 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 52 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 53 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 54 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 55 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 56 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 57 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 58 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 59 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 60 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 61 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 62 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 63 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 64 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 65 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 66 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 67 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 68 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 69 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 70 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 71 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 72 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 73 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 74 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 75 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 76 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 77 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 78 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 79 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 80 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 81 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 82 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 83 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 84 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 85 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 86 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 87 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 88 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 89 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 90 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 91 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 92 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 93 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 94 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 95 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 96 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 97 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
 98 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 99 2f: 6.23E-01 2x: 1.04E+06 2s: 1.16E+05 ir: 6.15E-01 iu: 1.16E+05 rsn:   0
100 2f: 6.18E-04 2x: 1.06E+06 2s: 1.16E+05 ir: 5.25E-04 iu: 1.16E+05 rsn:   0
 -> Cut time step: snes= -5 icut=  3[ 15] t=  3.05176E-05 dt=  3.81470E-06
  1 2f: 8.15E-01 2x: 1.05E+06 2s: 6.36E+03 ir: 8.05E-01 iu: 6.33E+03 rsn:   0
  2 2f: 1.34E-03 2x: 1.05E+06 2s: 4.20E+03 ir: 1.31E-03 iu: 4.19E+03 rsn:   0
  3 2f: 4.65E-04 2x: 1.05E+06 2s: 2.99E+02 ir: 4.59E-04 iu: 2.50E+02 rsn:   0
  4 2f: 1.43E-04 2x: 1.05E+06 2s: 1.18E+02 ir: 1.41E-04 iu: 1.18E+02 rsn:   0
  5 2f: 4.35E-05 2x: 1.05E+06 2s: 8.37E+01 ir: 4.30E-05 iu: 8.37E+01 rsn:   0
  6 2f: 1.27E-05 2x: 1.05E+06 2s: 5.80E+01 ir: 1.25E-05 iu: 5.80E+01 rsn:   0
  7 2f: 3.26E-06 2x: 1.05E+06 2s: 3.69E+01 ir: 3.22E-06 iu: 3.69E+01 rsn:   0
  8 2f: 5.57E-07 2x: 1.05E+06 2s: 1.83E+01 ir: 5.50E-07 iu: 1.83E+01 rsn:   0
  9 2f: 2.98E-08 2x: 1.05E+06 2s: 4.65E+00 ir: 2.95E-08 iu: 4.65E+00 rsn:   0
 10 2f: 9.92E-11 2x: 1.05E+06 2s: 2.79E-01 ir: 9.78E-11 iu: 2.79E-01 rsn: itol_res

 Step      2 Time=  3.43323E-05 Dt=  3.81470E-06 [y] snes_conv_reason:   10
  newton = 214 [     261] linear =   214 [       266] cuts =  3 [  18]
  --> SNES Linear/Non-Linear Iterations =           10  /           10
  --> SNES Residual:   9.921423E-11  9.921423E-13  9.778422E-11
  --> max chng: dpmx=   1.6597E+03 dtmpmx=   1.9780E-01
 

da.in

Hammond, Glenn E

unread,
Dec 3, 2014, 11:39:23 AM12/3/14
to pflotr...@googlegroups.com

Can you explain why the NaNs or infs are being produced in the first place, or at least pinpoint the location and catch it there?

 

Glenn

 

--
You received this message because you are subscribed to the Google Groups "pflotran-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/pflotran-dev/CAPz1TnfKQJov5Uxvghp_31uKpxz0SdJ1XLV%3Dy2pt5gOrkY_CHg%40mail.gmail.com.

Jed Brown

unread,
Dec 3, 2014, 1:48:38 PM12/3/14
to Hammond, Glenn E, pflotr...@googlegroups.com, pets...@mcs.anl.gov
"Hammond, Glenn E" <geh...@sandia.gov> writes:

> Can you explain why the NaNs or infs are being produced in the first place, or at least pinpoint the location and catch it there?

If the SNESFunction computes NaN because the state is invalid, please
call SNESSetFunctionDomainError and don't update the residual vector (or
don't put NaN in there). SNES will clean up and return with
SNESConvergedReason SNES_DIVERGED_FUNCTION_DOMAIN.

We don't do this by default because detecting NaN collectively requires
an extra reduction. Perhaps SNES could have a mode that automatically
checks.

>
> Glenn
>
> From: pflotr...@googlegroups.com [mailto:pflotr...@googlegroups.com] On Behalf Of Gautam Bisht
> Sent: Wednesday, December 03, 2014 8:36 AM
> To: pflotr...@googlegroups.com
> Subject: [EXTERNAL] [pflotran-dev: 2695] Unintended consequence of CHKERRQ()
>
> The issue is that when SNES diverges with a "not-a-number or infinite", the code crashes without giving timestepper_BE a chance to reduce the time-step. But, if CHKERRQ() is commented out after call to SNESSolve(), the code is able to reduce dt sufficiently to converge due to "itol_res" (PFLOTRAN's custom convergence reason == inorm_residual < solver%newton_inf_res_tol). Attached below is inputdeck (with an outrages initial dt = 1yr to illustrate the issue). Apart from commenting out CHKERRQ(), is there an alternate way to let timestepper_BE reduce dt?
>
> Below are relevant information to reproduce the error.
>
>>hg tip
> changeset: 8541:fa2d4295aa14
> tag: tip
> user: Nathan Collier <nathanie...@gmail.com<mailto:nathanie...@gmail.com>>
> date: Wed Dec 03 08:52:00 2014 -0500
> summary: made output process-model identifier lines to be of uniform length (80 characters). Yeah it is a little OCD of me.
>
>>./pflotran -pflotranin da.in<http://da.in>
>
> == TH FLOW =====================================================================
> 1 2f: 2.55E+00 2x: 1.63E+07 2s: 1.72E+07 ir: 1.49E+00 iu: 3.10E+06 rsn: 0
> 2 2f: 1.88E-01 2x: 1.57E+14 2s: 1.57E+14 ir: 5.33E-02 iu: 4.37E+13 rsn: 0
> 3 2f: 1.07E-01 2x: 1.60E+14 2s: 4.28E+12 ir: 2.27E-02 iu: 9.90E+11 rsn: 0
> 4 2f: 1.24E+01 2x: 1.60E+14 2s: 9.53E+11 ir: 1.16E+01 iu: 7.15E+11 rsn: 0
> 5 2f: 1.29E+06 2x: 7.75E+20 2s: 7.75E+20 ir: 1.15E+06 iu: 7.21E+20 rsn: max_norm
> -> Cut time step: snes=-10 icut= 1[ 0] t= 0.00000E+00 dt= 5.00000E-01
> 1 2f: 2.22E+00 2x: 1.38E+07 2s: 1.47E+07 ir: 1.47E+00 iu: 3.06E+06 rsn: 0
> 2 2f: 2.94E-01 2x: 1.11E+14 2s: 1.11E+14 ir: 1.44E-01 iu: 4.05E+13 rsn: 0
> 3 2f: 6.05E+20 2x: 2.13E+14 2s: 1.79E+14 ir: 4.44E+20 iu: 8.93E+13 rsn: max_norm
> -> Cut time step: snes=-10 icut= 2[ 0] t= 0.00000E+00 dt= 2.50000E-01
> 1 2f: 2.17E+00 2x: 1.12E+07 2s: 1.21E+07 ir: 1.44E+00 iu: 3.01E+06 rsn: 0
> 2 2f: 5.55E-01 2x: 8.08E+13 2s: 8.08E+13 ir: 4.07E-01 iu: 3.66E+13 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 50 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:16:17 2014
> [0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
> [0]PETSC ERROR: #1 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> [0]PETSC ERROR: #3 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
> [0]PETSC ERROR: #4 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 72.
>
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
>
>
>> hg diff
> diff -r fa2d4295aa14 src/pflotran/timestepper_BE.F90
> --- a/src/pflotran/timestepper_BE.F90 Wed Dec 03 08:52:00 2014 -0500
> +++ b/src/pflotran/timestepper_BE.F90 Wed Dec 03 08:22:35 2014 -0800
> @@ -310,7 +310,7 @@
> call PetscTime(log_start_time, ierr);CHKERRQ(ierr)
>
> call SNESSolve(solver%snes,PETSC_NULL_OBJECT, &
> - process_model%solution_vec,ierr);CHKERRQ(ierr)
> + process_model%solution_vec,ierr);!CHKERRQ(ierr)
>
> call PetscTime(log_end_time, ierr);CHKERRQ(ierr)
>
>
>>./pflotran -pflotranin da.in<http://da.in>
>
> == TH FLOW =====================================================================
> 1 2f: 2.55E+00 2x: 1.63E+07 2s: 1.72E+07 ir: 1.49E+00 iu: 3.10E+06 rsn: 0
> 2 2f: 1.88E-01 2x: 1.57E+14 2s: 1.57E+14 ir: 5.33E-02 iu: 4.37E+13 rsn: 0
> 3 2f: 1.07E-01 2x: 1.60E+14 2s: 4.28E+12 ir: 2.27E-02 iu: 9.90E+11 rsn: 0
> 4 2f: 1.24E+01 2x: 1.60E+14 2s: 9.53E+11 ir: 1.16E+01 iu: 7.15E+11 rsn: 0
> 5 2f: 1.29E+06 2x: 7.75E+20 2s: 7.75E+20 ir: 1.15E+06 iu: 7.21E+20 rsn: max_norm
> -> Cut time step: snes=-10 icut= 1[ 0] t= 0.00000E+00 dt= 5.00000E-01
> 1 2f: 2.22E+00 2x: 1.38E+07 2s: 1.47E+07 ir: 1.47E+00 iu: 3.06E+06 rsn: 0
> 2 2f: 2.94E-01 2x: 1.11E+14 2s: 1.11E+14 ir: 1.44E-01 iu: 4.05E+13 rsn: 0
> 3 2f: 6.05E+20 2x: 2.13E+14 2s: 1.79E+14 ir: 4.44E+20 iu: 8.93E+13 rsn: max_norm
> -> Cut time step: snes=-10 icut= 2[ 0] t= 0.00000E+00 dt= 2.50000E-01
> 1 2f: 2.17E+00 2x: 1.12E+07 2s: 1.21E+07 ir: 1.44E+00 iu: 3.01E+06 rsn: 0
> 2 2f: 5.55E-01 2x: 8.08E+13 2s: 8.08E+13 ir: 4.07E-01 iu: 3.66E+13 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 50 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:17:35 2014
> [0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
> [0]PETSC ERROR: #1 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> [0]PETSC ERROR: #3 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
> [0]PETSC ERROR: #4 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #6 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> -> Cut time step: snes= 0 icut= 3[ 0] t= 0.00000E+00 dt= 1.25000E-01
> 1 2f: 2.12E+00 2x: 9.08E+06 2s: 9.85E+06 ir: 1.41E+00 iu: 2.94E+06 rsn: 0
> 2 2f: 7.89E-01 2x: 5.95E+13 2s: 5.95E+13 ir: 6.28E-01 iu: 3.20E+13 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 34 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:17:35 2014
> [0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
> [0]PETSC ERROR: #7 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #8 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> [0]PETSC ERROR: #9 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
> [0]PETSC ERROR: #10 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #11 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #12 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> -> Cut time step: snes= 0 icut= 4[ 0] t= 0.00000E+00 dt= 6.25000E-02
> 1 2f: 2.04E+00 2x: 7.38E+06 2s: 8.06E+06 ir: 1.36E+00 iu: 2.86E+06 rsn: 0
> 2 2f: 5.60E+04 2x: 4.28E+13 2s: 4.28E+13 ir: 4.47E+04 iu: 2.66E+13 rsn: 0
> 3 2f: 1.16E+18 2x: 4.91E+13 2s: 2.30E+13 ir: 9.47E+17 iu: 1.69E+13 rsn: max_norm
> -> Cut time step: snes=-10 icut= 5[ 0] t= 0.00000E+00 dt= 3.12500E-02
> 1 2f: 1.94E+00 2x: 6.00E+06 2s: 6.57E+06 ir: 1.30E+00 iu: 2.74E+06 rsn: 0
> 2 2f: 6.76E+04 2x: 2.91E+13 2s: 2.91E+13 ir: 5.18E+04 iu: 2.06E+13 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 14 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:17:35 2014
> [0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
> [0]PETSC ERROR: #13 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #14 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> [0]PETSC ERROR: #15 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
> [0]PETSC ERROR: #16 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #17 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #18 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> -> Cut time step: snes= 0 icut= 6[ 0] t= 0.00000E+00 dt= 1.56250E-02
> 1 2f: 1.81E+00 2x: 4.84E+06 2s: 5.31E+06 ir: 1.21E+00 iu: 2.58E+06 rsn: 0
> 2 2f: 1.56E+04 2x: 1.81E+13 2s: 1.81E+13 ir: 1.22E+04 iu: 1.43E+13 rsn: 0
> 3 2f: 9.46E+15 2x: 1.99E+13 2s: 7.41E+12 ir: 7.34E+15 iu: 5.84E+12 rsn: max_norm
> -> Cut time step: snes=-10 icut= 7[ 0] t= 0.00000E+00 dt= 7.81250E-03
> 1 2f: 1.64E+00 2x: 3.85E+06 2s: 4.21E+06 ir: 1.11E+00 iu: 2.37E+06 rsn: 0
> 2 2f: 1.37E+04 2x: 9.82E+12 2s: 9.82E+12 ir: 1.02E+04 iu: 8.52E+12 rsn: 0
> 3 2f: 5.35E+14 2x: 1.86E+13 2s: 1.65E+13 ir: 3.78E+14 iu: 1.62E+13 rsn: max_norm
> -> Cut time step: snes=-10 icut= 8[ 0] t= 0.00000E+00 dt= 3.90625E-03
> 1 2f: 1.42E+00 2x: 3.01E+06 2s: 3.25E+06 ir: 9.67E-01 iu: 2.09E+06 rsn: 0
> 2 2f: 6.03E+03 2x: 4.39E+12 2s: 4.39E+12 ir: 4.38E+03 iu: 4.08E+12 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 30 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:17:35 2014
> [0]PETSC ERROR: Configure options --download-hdf5=1 --with-blas-lapack-lib=/System/Library/Frameworks/Accelerate.framework/Versions/Current/Accelerate --download-parmetis=yes --download-metis=yes --with-cc=/opt/local/bin/mpicc-openmpi-mp --with-cxx=/opt/local/bin/mpicxx-openmpi-mp --with-fc=/opt/local/bin/mpif90-openmpi-mp --with-mpiexec=/opt/local/bin/mpiexec-openmpi-mp PETSC_ARCH=darwin-gnu-fort-debug --download-sowing=1
> [0]PETSC ERROR: #19 VecValidValues() line 34 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #20 SNESComputeFunction() line 2043 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> [0]PETSC ERROR: #21 SNESLineSearchApply_Basic() line 40 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/impls/basic/linesearchbasic.c
> [0]PETSC ERROR: #22 SNESLineSearchApply() line 563 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/linesearch/interface/linesearch.c
> [0]PETSC ERROR: #23 SNESSolve_NEWTONLS() line 254 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #24 SNESSolve() line 3743 in /Users/gbisht/Research/Models/PETSc/petsc_821a792/src/snes/interface/snes.c
> -> Cut time step: snes= 0 icut= 9[ 0] t= 0.00000E+00 dt= 1.95312E-03
> 1 2f: 1.16E+00 2x: 2.30E+06 2s: 2.42E+06 ir: 8.00E-01 iu: 1.76E+06 rsn: 0
> 2 2f: 4.32E+03 2x: 1.47E+12 2s: 1.47E+12 ir: 3.07E+03 iu: 1.42E+12 rsn: 0
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Floating point exception
> [0]PETSC ERROR: Vec entry at local location 4 is not-a-number or infinite at end of function: Parameter number 3
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.1, unknown
> [0]PETSC ERROR: ../../../src/pflotran/pflotran on a darwin-gnu-fort-debug named gautams-mac-pro.dhcp.lbl.gov<http://gautams-mac-pro.dhcp.lbl.gov> by gbisht Wed Dec 3 08:17:35 2014
> To view this discussion on the web visit https://groups.google.com/d/msgid/pflotran-dev/CAPz1TnfKQJov5Uxvghp_31uKpxz0SdJ1XLV%3Dy2pt5gOrkY_CHg%40mail.gmail.com<https://groups.google.com/d/msgid/pflotran-dev/CAPz1TnfKQJov5Uxvghp_31uKpxz0SdJ1XLV%3Dy2pt5gOrkY_CHg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google Groups "pflotran-dev" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pflotran-dev/92b8b510fceb47309cc0b8f7ada47ef4%40ES06AMSNLNT.srn.sandia.gov.
signature.asc

Gautam Bisht

unread,
Dec 3, 2014, 2:00:53 PM12/3/14
to pflotr...@googlegroups.com
Glenn,

During newton iteration, few gas related variables in TH mode were becoming NaNs. For DALL_AMICO ice-model, I set gas related quantities to zero as gas is not accounted for within DALL_AMICO formulation. With this fix, the SNES diverges gracefully (without NaNs) and timestepper_BE is able to cut the dt to be sufficiently small enough for the model to converge.

Thanks for suggesting to find the reason why NaNs were arising in the first place.

-Gautam. 

Nathan Collier

unread,
Dec 3, 2014, 2:37:48 PM12/3/14
to pflotr...@googlegroups.com
The SNESSolve error check is the most critical one though. Otherwise PFLOTRAN will move over errors and keep computing and you may never know anything went wrong. 

I don't follow why the gas saturation leads to NaN's in the residual. Are we dividing something by that value?

Nate

Gautam Bisht

unread,
Dec 3, 2014, 3:02:24 PM12/3/14
to pflotr...@googlegroups.com
Hi Nate,

During SNES iterations, the values of (P,T) for k+1 iteration are becoming unrealistic, which is fine because dt was outrageous too large (=1yr). Given, the outrageous (P,T), certain quantities for gas (not the gas_saturation, but mol_gas and others) are becoming NaN in THAuxVarComputeFreezing. These NaN gas-related quantities then lead to NaNs in the residual. Since, DALL_AMICO does not account for gas, I have set them to zero. A more thorough fix would be to figure out why the gas related quantities are becoming NaN and avoid them. One can use the inputdeck I sent to come up with a better fix.

For the time being, I'm calling it a success and moving forward to get some simulations done for AGU, hoping that this fix is sufficient.

-Gautam.

Gautam Bisht

unread,
Dec 3, 2014, 3:14:02 PM12/3/14
to pflotr...@googlegroups.com, Hammond, Glenn E, For users of the development version of PETSc
The user function was putting NaNs in the residual vector. I ended up fixing the user function to avoid putting NaNs in the residual vector.

-Gautam.

Gautam Bisht

unread,
Dec 4, 2014, 11:27:42 AM12/4/14
to pflotr...@googlegroups.com, Hammond, Glenn E, For users of the development version of PETSc
Hi Jed,

Earlier I completely missed your point. I believe the utility of SNESSetFunctionDomainError is that an application can determine if the state (xx) being passed to compute residual function is meaningful (eg. a pair of (P,T) could result in negative density). If the state is not meaningful, the application could simply call SNESSetFunctionDomainError() and not update the residual vector. This would cause SNES to exit with SNES_DIVERGED_FUNCTION_DOMAIN.

Looking at PFLOTRAN code more carefully, I now realize that Peter is using SNESSetFunctionDomainError() in FLASH2/MPHASE/MISCIBLE/IMMISCIBLE modes. I will now implement similar check for TH mode.

Thanks,
-Gautam.

Jed Brown

unread,
Dec 4, 2014, 11:38:42 AM12/4/14
to Gautam Bisht, pflotr...@googlegroups.com, For users of the development version of PETSc, Hammond, Glenn E
Gautam Bisht <gbi...@lbl.gov> writes:

> Hi Jed,
>
> Earlier I completely missed your point. I believe the utility of
> SNESSetFunctionDomainError
> is that an application can determine if the state (xx) being passed to
> compute residual function is meaningful (eg. a pair of (P,T) could result
> in negative density). If the state is not meaningful, the application could
> simply call SNESSetFunctionDomainError() and not update the residual
> vector. This would cause SNES to exit with SNES_DIVERGED_FUNCTION_DOMAIN.

Yes.

> Looking at PFLOTRAN code more carefully, I now realize that Peter is
> using SNESSetFunctionDomainError()
> in FLASH2/MPHASE/MISCIBLE/IMMISCIBLE modes. I will now implement similar
> check for TH mode.

Great.
signature.asc
Reply all
Reply to author
Forward
0 new messages