Run time error from PETSc (Object is in wrong state / Matrix is missing diagonal entry 0)

Shohei Ogawa

unread,

Aug 25, 2017, 12:30:01 PM8/25/17

to moose-users

Hi all,

When I run my Moose app, I get a PETSc error saying "Object is in wrong state" and "Matrix is missing diagonal entry 0." I run this program in serial with default preconditioner.

Do you think this is related to my app implementation, such as my computeQpResidual is wrong? If so, is there any way to know which variable in the simulation is causing this issue?

Thank you,
Shohei

% ../../nannopterum-opt -i only_domains.i

Framework Information:

MOOSE version: git commit 942a566 on 2017-07-26

PETSc Version: 3.7.5

Current Time: Fri Aug 25 12:20:24 2017

Executable Timestamp: Fri Aug 25 12:16:36 2017

Parallelism:

Num Processors: 1

Num Threads: 1

Mesh:

Parallel Type: replicated

Mesh Dimension: 3

Spatial Dimension: 3

Nodes:

Total: 66472

Local: 66472

Elems:

Total: 33476

Local: 33476

Num Subdomains: 5

Num Partitions: 1

Nonlinear System:

Num DOFs: 31753

Num Local DOFs: 31753

Variables: { "oxide_coverage" "theta_H_ad" "X" "sigma_Pt" "c_H_plus" "c_Pt_2plus" } "phi_solution"

"particle_diameter"

Finite Element Types: "LAGRANGE" "LAGRANGE" "LAGRANGE"

Approximation Orders: "FIRST" "FIRST" "FIRST"

Auxiliary System:

Num DOFs: 69695

Num Local DOFs: 69695

Variables: "phi_metal" "T" "sigma_C" { "rxn_net_oxidation" "rxn_net_dissolution" } { "D_Pt_2plus"

"D_H_plus" } "permittivity" "c_SO3_minus" "x_H2O"

Finite Element Types: "LAGRANGE" "LAGRANGE" "LAGRANGE" "LAGRANGE" "LAGRANGE" "LAGRANGE" "LAGRANGE"

"LAGRANGE"

Approximation Orders: "FIRST" "FIRST" "FIRST" "FIRST" "FIRST" "FIRST" "FIRST" "FIRST"

Execution Information:

Executioner: Transient

TimeStepper: IterationAdaptiveDT

Solver Mode: 0

Time Step 0, time = 0

dt = 0

Time Step 1, time = 1e-10

dt = 1e-10

0 Nonlinear |R| = 5.401621e+08

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------

[0]PETSC ERROR: Object is in wrong state

[0]PETSC ERROR: Matrix is missing diagonal entry 0

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017

[0]PETSC ERROR: ../../nannopterum-opt on a arch-darwin-c-opt named OXYGEN.ANDREW.CMU.EDU by shoheio Fri Aug 25 12:20:23 2017

[0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/mpich_petsc-3.7.5/clang-opt-superlu --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 -CC=mpicc -CXX=mpicxx -FC=mpif90 -F77=mpif77 -F90=mpif90 -CFLAGS="-fPIC -fopenmp" -CXXFLAGS="-fPIC -fopenmp" -FFLAGS="-fPIC -fopenmp" -FCFLAGS="-fPIC -fopenmp" -F90FLAGS="-fPIC -fopenmp" -F77FLAGS="-fPIC -fopenmp"

[0]PETSC ERROR: #1 MatILUFactorSymbolic_SeqAIJ() line 1733 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/mat/impls/aij/seq/aijfact.c

[0]PETSC ERROR: #2 MatILUFactorSymbolic() line 6579 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/mat/interface/matrix.c

[0]PETSC ERROR: #3 PCSetUp_ILU() line 213 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/impls/factor/ilu/ilu.c

[0]PETSC ERROR: #4 PCSetUp() line 968 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: #5 KSPSetUp() line 390 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #6 KSPSolve() line 599 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #7 SNESSolve_NEWTONLS() line 230 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/impls/ls/ls.c

[0]PETSC ERROR: #8 SNESSolve() line 4005 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/interface/snes.c

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1

:

system msg for write_line failure : Bad file descriptor

Shohei Ogawa

unread,

Aug 25, 2017, 12:35:51 PM8/25/17

to moose-users

Here is my Executioner of the input file.

[Executioner]

type = Transient

solve_type = PJFNK

petsc_options = '-ksp_converged_reason -snes_converged_reason'

#petsc_options_iname = '-pc_type -pc_hypre_type'

#petsc_options_value = 'hypre boomeramg'

end_time = 20.0

scheme = bdf2

[./TimeStepper]

type = IterationAdaptiveDT

linear_iteration_ratio = 5

cutback_factor = 0.4

dt = 1e-10

growth_factor = 1.2

optimal_iterations = 4

[../]

[]

Also, when I include

petsc_options_iname = '-pc_type -pc_hypre_type'

petsc_options_value = 'hypre boomeramg',

I get an error below. The program is trying to allocate a very small negative amount of memory. It is strange.

Out of memory trying to allocate -8 bytes

[0]PETSC ERROR: #1 PCSetUp_HYPRE() line 229 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/impls/hypre/hypre.c

[0]PETSC ERROR: #2 PCSetUp() line 968 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/interface/precon.c

[0]PETSC ERROR: #3 KSPSetUp() line 390 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #4 KSPSolve() line 599 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c

[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/impls/ls/ls.c

[0]PETSC ERROR: #6 SNESSolve() line 4005 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/interface/snes.c

zsh: segmentation fault ../../nannopterum-opt -i only_domains.i

Kong, Fande

unread,

Aug 25, 2017, 12:37:36 PM8/25/17

to moose...@googlegroups.com

On Fri, Aug 25, 2017 at 10:30 AM, Shohei Ogawa <ogawa...@gmail.com> wrote:

Hi all,

When I run my Moose app, I get a PETSc error saying "Object is in wrong state" and "Matrix is missing diagonal entry 0." I run this program in serial with default preconditioner.
Do you think this is related to my app implementation, such as my computeQpResidual is wrong? If so, is there any way to know which variable in the simulation is causing this issue?

If you do not use FDP, then the error may have nothing to do with computeQpResidual. You may forget to implemenation computeQpJacobian for one or a few variables.

Fande,

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/9348a40a-2e7e-4277-8019-5fe8a315fdd6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kong, Fande

unread,

Aug 25, 2017, 12:44:35 PM8/25/17

to moose...@googlegroups.com

On Fri, Aug 25, 2017 at 10:35 AM, Shohei Ogawa <ogawa...@gmail.com> wrote:

Here is my Executioner of the input file.

[Executioner]
type = Transient
solve_type = PJFNK
petsc_options = '-ksp_converged_reason -snes_converged_reason'
#petsc_options_iname = '-pc_type -pc_hypre_type'
#petsc_options_value = 'hypre boomeramg'
end_time = 20.0
scheme = bdf2
[./TimeStepper]
type = IterationAdaptiveDT
linear_iteration_ratio = 5
cutback_factor = 0.4
dt = 1e-10
growth_factor = 1.2
optimal_iterations = 4
[../]
[]

Also, when I include
petsc_options_iname = '-pc_type -pc_hypre_type'
petsc_options_value = 'hypre boomeramg',

I get an error below. The program is trying to allocate a very small negative amount of memory. It is strange.

It may be a good idea to check your implementation with "-snes_type test -snes_test_display" for a SMALL problem before you try to use different PETSc options.

Thanks,

Fande,

Out of memory trying to allocate -8 bytes
[0]PETSC ERROR: #1 PCSetUp_HYPRE() line 229 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/impls/hypre/hypre.c
[0]PETSC ERROR: #2 PCSetUp() line 968 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #3 KSPSetUp() line 390 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #4 KSPSolve() line 599 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 230 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 4005 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/interface/snes.c
zsh: segmentation fault ../../nannopterum-opt -i only_domains.i

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/f301982e-df9d-4925-97c0-ab78e5fa8b9d%40googlegroups.com.

Shohei Ogawa

unread,

Aug 25, 2017, 1:20:14 PM8/25/17

to moose...@googlegroups.com

Thank you very much for your comments

I have found I don't implement computeQpJacobian yet. I will work on those kernels' Jacobian expressions.

I remember according to a recent Moose tutorial document , computeQpJacobian is not always necessary to be implemented, but recommended to do so. Is it only the case when we do JFNK where we could omit Jacobian?

Shohei

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAK4PXd0oHq3LB9GHDP2jZHvFkH8aVq%3Dgbido%2BfJcAeD3T2_2KA%40mail.gmail.com.

Kong, Fande

unread,

Aug 25, 2017, 1:39:00 PM8/25/17

to moose...@googlegroups.com

On Fri, Aug 25, 2017 at 11:20 AM, Shohei Ogawa <ogawa...@gmail.com> wrote:

Thank you very much for your comments

I have found I don't implement computeQpJacobian yet. I will work on those kernels' Jacobian expressions.

I remember according to a recent Moose tutorial document , computeQpJacobian is not always necessary to be implemented, but recommended to do so. Is it only the case when we do JFNK where we could omit Jacobian?

In my opinion, we need to implement computeQpJacobian to avoid having an ill-conditioned or singular matrix. If you simply ignore computeQpJacobian, we may end up a singular matrix that can not be solved. We could ignore computeQpOffDiagJacobian. But we recommend to implement Jacobian as accurate as possible.

JFNK does not need Jacobian.

Fande,

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/CAHfiw0c143YvWPGjzaQkWy4tbgty3T4jfSsQ0-_PnpUXYwXrLQ%40mail.gmail.com.

Shohei Ogawa

unread,

Aug 25, 2017, 2:17:48 PM8/25/17

to moose-users

Thank you for the clarification.

I made my problem simpler and smaller. Now I apply some PETSc options for testing as you mentioned. ("-snes_type test -snes_test_display")

Executioner block in my input file looks like this.

[Executioner]

# petsc_options_iname = '-pc_type -pc_hypre_type'

# petsc_options_value = 'hypre boomeramg'

type = Transient

solve_type = PJFNK

petsc_options = '-ksp_converged_reason -snes_converged_reason -snes_test_display'

petsc_options_iname = -snes_type

petsc_options_value = test

end_time = 20.0

[./TimeStepper]

type = IterationAdaptiveDT

linear_iteration_ratio = 5

cutback_factor = 0.4

dt = 1e-10

growth_factor = 1.2

optimal_iterations = 4

[../]

[]

However, this causes an Invalid argument error. Am I missing something important? I copy and paste the error message from PETSc below.

Thank you,

Shohei

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------

[0]PETSC ERROR: Invalid argument

[0]PETSC ERROR: Cannot test with alternative preconditioner

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017

[0]PETSC ERROR: /Users/shohei/moose_projects/nannopterum/nannopterum-opt on a arch-darwin-c-opt named OXYGEN.ANDREW.CMU.EDU by shoheio Fri Aug 25 14:13:09 2017

[0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/mpich_petsc-3.7.5/clang-opt-superlu --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 -CC=mpicc -CXX=mpicxx -FC=mpif90 -F77=mpif77 -F90=mpif90 -CFLAGS="-fPIC -fopenmp" -CXXFLAGS="-fPIC -fopenmp" -FFLAGS="-fPIC -fopenmp" -FCFLAGS="-fPIC -fopenmp" -F90FLAGS="-fPIC -fopenmp" -F77FLAGS="-fPIC -fopenmp"

[0]PETSC ERROR: #1 SNESSolve_Test() line 24 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/impls/test/snestest.c

[0]PETSC ERROR: #2 SNESSolve() line 4005 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/interface/snes.c

Kong, Fande

unread,

Aug 25, 2017, 2:54:02 PM8/25/17

to moose...@googlegroups.com

On Fri, Aug 25, 2017 at 12:17 PM, Shohei Ogawa <ogawa...@gmail.com> wrote:

Thank you for the clarification.

I made my problem simpler and smaller. Now I apply some PETSc options for testing as you mentioned. ("-snes_type test -snes_test_display")

Executioner block in my input file looks like this.

[Executioner]

# petsc_options_iname = '-pc_type -pc_hypre_type'

# petsc_options_value = 'hypre boomeramg'

type = Transient

solve_type = PJFNK

petsc_options = '-ksp_converged_reason -snes_converged_reason -snes_test_display'

petsc_options_iname = -snes_type

petsc_options_value = test

end_time = 20.0

[./TimeStepper]

type = IterationAdaptiveDT

linear_iteration_ratio = 5

cutback_factor = 0.4

dt = 1e-10

growth_factor = 1.2

optimal_iterations = 4

[../]

[]

However, this causes an Invalid argument error. Am I missing something important? I copy and paste the error message from PETSc below.

You can not test Jacobian when you are using PJFNK.

Please use "solve_type = NEWTON"

Fande,

Thank you,
Shohei

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------

[0]PETSC ERROR: Invalid argument

[0]PETSC ERROR: Cannot test with alternative preconditioner

[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.

[0]PETSC ERROR: Petsc Release Version 3.7.5, Jan, 01, 2017

[0]PETSC ERROR: /Users/shohei/moose_projects/nannopterum/nannopterum-opt on a arch-darwin-c-opt named OXYGEN.ANDREW.CMU.EDU by shoheio Fri Aug 25 14:13:09 2017

[0]PETSC ERROR: Configure options --prefix=/opt/moose/petsc/mpich_petsc-3.7.5/clang-opt-superlu --download-hypre=1 --with-ssl=0 --with-debugging=no --with-pic=1 --with-shared-libraries=1 --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --download-fblaslapack=1 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 -CC=mpicc -CXX=mpicxx -FC=mpif90 -F77=mpif77 -F90=mpif90 -CFLAGS="-fPIC -fopenmp" -CXXFLAGS="-fPIC -fopenmp" -FFLAGS="-fPIC -fopenmp" -FCFLAGS="-fPIC -fopenmp" -F90FLAGS="-fPIC -fopenmp" -F77FLAGS="-fPIC -fopenmp"

[0]PETSC ERROR: #1 SNESSolve_Test() line 24 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/impls/test/snestest.c

[0]PETSC ERROR: #2 SNESSolve() line 4005 in /private/var/folders/7j/x417msrs6r14y_29v2078_qh0000gq/T/moose_package_build_temp/petsc-3.7.5-mpich-clang/petsc-3.7.5/src/snes/interface/snes.c

application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

--
You received this message because you are subscribed to the Google Groups "moose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to moose-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/moose-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/moose-users/fa283e1f-b2c6-4d6b-9766-848f6cbabde3%40googlegroups.com.

Reply all

Reply to author

Forward