CP2K freeze

[Salvatore Labonia]

Hello,

we are facing freeze using CP2K on our HPC cluster.

We have totally 94 Dell server but running cp2k v9.1 compiled with intel compiler and linked with intel mpi library, customer is experiencing running freeze.

No matter the number or the type of involved nodes.

The freeze happens randomly, not at the same interaction number, even using the same running command and the same dataset for input.

Looking at processes status on nodes when freeze occurs, they seem to be running, using CPU but, if we try to attach to any process (and forked children of course), we can see that they all are sitting on a wait system call for data coming (orout going) from (to) a pipe.

No other systems call are run by processes…

Slurm thinks that job is still running.

Killing one of the stuck processes causes the death of orher processes and finally slurm realizes that job has crashed.

Is this behaviour usual in same circumstances (and therefore customer has something to do to avoid it) or could it be caused by some other reason (cp2k compilation, mpi version, intel compilers version)?

Is there any way to have a debugging execution of cp2k/mpi with a more or less verbose output in order to understand at which point/call does the freeze happen?

 Regards

Salvatore

[Krack Matthias (PSI)]

Hi Salvatore

 

You can add the keyword TRACE (or TRACE_MASTER to trace only the MPI root process) in the &GLOBAL section of the CP2K input to get a more detailed output.

Does the run freeze for any kind of CP2K input? How did you compile CP2K? Could you run the regression test successfully? It is difficult to make any suggestion without further information.

 

Best

 

Matthias

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/2bffd2de-1afd-4980-b3aa-6438990d81a9n%40googlegroups.com.

[SALVATORE LABONIA]

Hello,

cp2k 8.2 was compiled using EasyBuild

I attach some output from regression test

Regards

Salvatore

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/4B492500-D071-47FB-B7F6-9D95EA33A429%40psi.ch.

[SALVATORE LABONIA]

Hello Krack,

we have tried to run cp2k using TRACE keyword and cp2k compute job died ...

Is it possible to identify the issue using attached cp2k log file ?

Can you help us to identify the error in the regression tests ?

Regards

Salvatore

Il 03/08/2022 16:11, Krack Matthias (PSI) ha scritto:

Hi Salvatore

 

Your CP2K 8.2 build has obviously issues, because many tests completed with the status FAILED or WRONG. Thus that cp2k binary cannot be considered ready for production.

From the error messages, it is not clear to me what the problem could be. Maybe, someone else has an idea.

 

BR

 

Matthias