DFTB calculations KILLED BY SIGNAL: 9 (Killed)

[bijaya pathak]

Dear all, 

I'm running DFTB calculations on a box of 216 water molecules. I have run the same input file for smaller clusters of water (eg 8/ 25) but when I'm running the same calculation for 216, after some time, the job gets killed and shows the error:

=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES

=   RANK 44 PID 38043 RUNNING AT cn073

=   KILLED BY SIGNAL: 9 (Killed)

I have tried changing a few parameters based on other discussions, but I could not solve this issue. I'm also attaching the input file for your reference. Any suggestion will be much appreciated.

&GLOBAL
  PROJECT water-216
  RUN_TYPE MD
&END GLOBAL

&FORCE_EVAL
  METHOD Quickstep
  &DFT
    ! Charge and multiplicity
    CHARGE 0
    MULTIPLICITY 1

    &MGRID
       CUTOFF [Ry] 400
       NGRIDS 5
    &END

    &QS
       METHOD DFTB
       &DFTB
         SELF_CONSISTENT T
         DO_EWALD        T
         DISPERSION      F
         &PARAMETER
         PARAM_FILE_PATH  ./DFTB/scc/
         PARAM_FILE_NAME  scc_parameter
         SK_FILE O  H     O-H.skf
         SK_FILE H  H     H-H.skf
         SK_FILE O  O     O-O.skf
         SK_FILE H  O     H-O.skf
         &END PARAMETER
        &END DFTB
        EPS_DEFAULT 1.0E-14

    &END
    &POISSON
       PERIODIC XYZ
    &END
Can anyone suggest what could be the possible issue with this input file ? 

[Konstantin Tokarev]

Hello Bijaya,

"Killed by signal 9" means that your process was killed with SIGKILL signal. For the record, this can be done manually by issuing command `kill -9 <pid>` by user that started the process, or by system administrator (e.g., via root login, sudo, etc.). However, in your case it's most likely caused by operating system itself, specifically oom-killer (out of memory condition causes Linux to kill processes which it considers the most "bad" at the moment in terms of their "oom score"). Please check `dmesg` output, it's very likely to indicate what exactly happened to your cp2k process.

[bijaya pathak]

Dear Konstantin,

Thank you so much for your reply. I am not sure if I know correctly to get the `dmesg`  output. I tried using dmesg | grep "error" and it printed a lot of things I could not understand. How to get information out of it regarding my issue ? 

[Konstantin Tokarev]

Please try `dmesg | grep oom-killer`. If you see cp2k there, this is it. Other running process might have also been affected. Alternatively, if you investigate "killed 9" crash right after it happened, you can just look at the end of `dmesg | less` (press End key, or type "G" (shift and 'g' key) to scroll "less" to the end).

Other info you may find there is details of memory consumption before process was killed, but this may be  overwhelming to read if you aren't used to such things. What is the most important, if there are no other memory hungry processes running in the system in parallel with cp2k and you still get oom, you'll probably have to reduce your system, basis set, or find another way to reduce memory consumption. Adding some swap on NVMe SSD might be helpful, but performance should be monitored carefully to ensure that it does not cause "thrashing" (constant swapping memory in and out) which would cause catastrophic slowdown of calculations.

[bijaya pathak]

Thank you so much. Let me try to do things you have suggested.

[Konstantin Tokarev]

Actually, given that it's DFTB, it seems unlikely to me that you could consume all system memory with one calculation. OOM condition can also be caused by too low memory limits. Please check your `ulimit -m`. Also, if you are using any kind of queue or resource management system, make sure you are using it correctly.