ERROR WHEN RUNNING THE CODE ON MULTIPLE NODES OF HPC

[孙翔]

Hi, I run my parallelized code which is similar to step-18 on HPC. When I run it in multiple mpi processes on a single node. It can give me a good solution. However, when I run it on multiple nodes of HPC and each node has one mpi process, it reported an error as follows. I checked the l2 norm of the assembled system matrix which is not equal to zero. So, I really want to know what leads to the error? Thank you very much. 

Exception on processing:

--------------------------------------------------------

An error occurred in line <148> of file </scratch/07506/xs260/dealii-src/source/lac/petsc_solver.cc> in function

    void dealii::PETScWrappers::SolverBase::solve(const dealii::PETScWrappers::MatrixBase &, dealii::PETScWrappers::VectorBas$

The violated condition was:

    false

Additional information:

Iterative method reported convergence failure in step 4294967295. The residual in the last step was nan.

This error message can indicate that you have simply not allowed a sufficiently large number of iterations for your iterative$

The other situation where this error may occur is when your matrix is not invertible (e.g., your matrix has a null-space), or$

--------------------------------------------------------

Aborting!

[Wolfgang Bangerth]

On 8/2/20 1:50 AM, 孙翔 wrote:
> Hi, I run my parallelized code which is similar to step-18 on HPC. When I run
> it in multiple mpi processes on a single node. It can give me a good solution.
> However, when I run it on multiple nodes of HPC and each node has one mpi
> process, it reported an error as follows.

Are you saying that the same number of MPI processes works when you run
locally, but doesn't when run on a cluster? That doesn't make any sense --
nothing in the code depends on *where* these MPI processes run.

Are you running in debug mode in both cases?

Best
W.


--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@colostate.edu
www: http://www.math.colostate.edu/~bangerth/

[孙翔]

Hi, Wolfgang,

Yes, it cannot run on a cluster. Both of them run in release mode. I'm also curious about the error. I debug the code by outputting some specific values. 

Best,

Xiang

[Wolfgang Bangerth]

On 8/2/20 11:26 PM, 孙翔 wrote:
>
> Yes, it cannot run on a cluster. Both of them run in release mode. I'm also
> curious about the error. I debug the code by outputting some specific values.

If you see any error, the first step always should be to run in debug mode
instead. Make sure that that works, and only then is it worth your time to
think about release mode!

[孙翔]

Got it. Thanks a lot.