error in qs_ks_methods.F depending on number of cores (intel compiler)

[Laurent Joly]

Dear all,

I've been trying to compile CP2K 4.1 (I also tried with the 3.0 version) using the intel compiler version 2017.1.132.

I used the Linux-x86-64-intel-host.popt and Linux-x86-64-intel-host.psmp arch files, and I had to modify the Linux-x86-64-intel-mic.psmp file following the instructions of the intel mkl link line advisor:  https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor to make it work:

$ diff Linux-x86-64-intel-mic.psmp.backup Linux-x86-64-intel-mic.psmp
231d230
<             $(MKLROOT)/lib/intel64/libmkl_core.a \
233,234c232,234
<           -Wl,--end-group \
<           $(MKLROOT)/lib/intel64/libmkl_blacs_intelmpi_lp64.a
---
>             $(MKLROOT)/lib/intel64/libmkl_core.a \
>             $(MKLROOT)/lib/intel64/libmkl_blacs_intelmpi_lp64.a \
>           -Wl,--end-group

Then I had to run: ulimit -s unlimited in order to get rid of segmentation faults, and finally it worked... on 16 cores, but as soon as I tried to increase the number of cores I got this error message:

KS energy is an abnormal value (NaN/Inf).
qs_ks_methods.F:785

I enclose the complete output. Do you have some ideas on what could be the problem?

Best regards,
Laurent

[Iain Bethune]

Hi Laurent,

We don’t have experience yet with exactly that version (ifort 17.0.1), but with the 17.0.0 compiler we find that a couple of files need to be built at reduced optimisation level - see https://dashboard.cp2k.org/archive/epcc-phi-psmp/rev_17693.txt for an example arch file which is working.

Were you able to run the CP2K regression tests with success?

Cheers

- Iain

--

Iain Bethune
Project Manager, EPCC

Email: i.be...@epcc.ed.ac.uk
Twitter: @IainBethune @PrimeGrid @CP2Kproject
Web: http://www2.epcc.ed.ac.uk/~ibethune
Tel/Fax: +44 (0)131 651 7183/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Peter Guthrie Tait Road, Edinburgh, EH9 3FD

> --
> You received this message because you are subscribed to the Google Groups "cp2k" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
> To post to this group, send email to cp...@googlegroups.com.
> Visit this group at https://groups.google.com/group/cp2k.
> For more options, visit https://groups.google.com/d/optout.
> <out>


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

[Laurent Joly]

Hi Iain,

Thank you very much for your fast answer.

I tried a simplified version of the arch file you sent (enclosed
raptor2.psmp), but I keep getting the same behavior. I also tried to
compile all the files with -O1 (enclosed raptor1.psmp and
Linux-x86-64-intel-mic.psmp), or even -O0 (enclosed raptor3.psmp), but
here again it makes no difference.

When I try to run the regtests, the system hangs after printing
"Copying tests into working directory ... done!"... But possibly this
is another problem...

Best regards,
Laurent

> You received this message because you are subscribed to a topic in the Google Groups "cp2k" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/cp2k/C2FO7VAUZEg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to cp2k+uns...@googlegroups.com.

> To post to this group, send email to cp...@googlegroups.com.
> Visit this group at https://groups.google.com/group/cp2k.
> For more options, visit https://groups.google.com/d/optout.

--
Laurent JOLY

Institut Lumière Matière - Université Lyon 1
Campus de la Doua, bâtiment Brillouin
43 bd du 11 novembre 1918
69622 Villeurbanne Cedex, France

Tél: 04 724 326 11 - Fax: 04 724 326 48
Web: http://ilm-perso.univ-lyon1.fr/~ljoly/

[Iain Bethune]

Hi Laurent,

I got hold of the 2017.1.132 release and ran a regtest of the latest trunk code. Barring one regtest with numerical problems, it ran fine, so I don’t think there is something fundamentally broken with that version. Could you send your input file, and try running with the GLOBAL%PRINT_LEVEL HIGH so some more information is printed out before the error occurs? Maybe someone can spot the problem. Did you have any success running this input with previous builds of CP2K?

- Iain
--

Iain Bethune
Project Manager, EPCC

Email: i.be...@epcc.ed.ac.uk
Twitter: @IainBethune @PrimeGrid @CP2Kproject
Web: http://www2.epcc.ed.ac.uk/~ibethune
Tel/Fax: +44 (0)131 651 7183/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Peter Guthrie Tait Road, Edinburgh, EH9 3FD

> <Linux-x86-64-intel-mic.psmp><raptor1.psmp><raptor2.psmp><raptor3.psmp>

[Iain Bethune]

Further to what I reported this morning, I found that the MKL release 2017.1.132 is buggy. I was able to get a successful build using the compiler from the 2017.1.132 release and MKL from the previous (2017.0.098) release.

- Iain

--

Iain Bethune
Project Manager, EPCC

Email: i.be...@epcc.ed.ac.uk
Twitter: @IainBethune @PrimeGrid @CP2Kproject
Web: http://www2.epcc.ed.ac.uk/~ibethune
Tel/Fax: +44 (0)131 651 7183/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Peter Guthrie Tait Road, Edinburgh, EH9 3FD

[Laurent Joly]

Hi Iain,

Thank you again for your help.

> Further to what I reported this morning, I found that the MKL release 2017.1.132 is buggy. I was able to get a successful build using the compiler from the 2017.1.132 release and MKL from the previous (2017.0.098) release.

I asked our local administrator if it was possible to install the
previous version of the intel compiler, or at least of the MKL
libraries.

>> I got hold of the 2017.1.132 release and ran a regtest of the latest trunk code. Barring one regtest with numerical problems, it ran fine, so I don’t think there is something fundamentally broken with that version. Could you send your input file, and try running with the GLOBAL%PRINT_LEVEL HIGH so some more information is printed out before the error occurs? Maybe someone can spot the problem. Did you have any success running this input with previous builds of CP2K?

Please find enclosed the input and ouput files obtained with
GLOBAL%PRINT_LEVEL HIGH. I tried with CP2K version 4.1 and 3.0 and got
similar results...

Hopefully I will soon have access to the previous MKL release, and
keep you informed. In the meantime, I'm using a gfortran+acml
compilation, which runs fine (although more slowly)...

Best regards,
Laurent

[Iain Bethune]

Hi Laurent,

I can confirm that the problem with the energy going to NaN that you observed is indeed specific to the Intel compiler (at least it works for me with a local gfortran run). I think however there is also that there some issue (or at least numerical difficulty) with your input, since I observe that the total energy actually increases in the second SCF step, when it should be strictly decreasing when using OT. Maybe someone else can spot the problem?

- Iain

--

Iain Bethune
Project Manager, EPCC

Email: i.be...@epcc.ed.ac.uk
Twitter: @IainBethune @PrimeGrid @CP2Kproject
Web: http://www2.epcc.ed.ac.uk/~ibethune
Tel/Fax: +44 (0)131 651 7183/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Peter Guthrie Tait Road, Edinburgh, EH9 3FD

> <testintel30.tgz><testintel41.tgz>

[Laurent Joly]

Hi Iain,

Our administrator installed the 2017.0 version of intel compiler and
librairies. I compiled CP2K 4.1 with it and, although I still have to
make more tests, it seems to be working.

Regarding the behavior of the SCF, I'm using the DIIS minimizer. I'm
not an expert on the OT method, but when I read in the manual:

MINIMIZER MINIMIZER {Keyword} Minimizer to be used with the OT method
This keyword cannot be repeated and it expects precisely one keyword.
Default value: CG List of valid keywords:
BROYDENBroyden mixing approximating the inverse Hessian
CGConjugate Gradients: most reliable, use for difficult systems. The
total energy should decrease at every OT CG step if the line search is
appropriate.
DIISDirect inversion in the iterative subspace: less reliable than CG,
but sometimes about 50% faster
SDSteepest descent: not recommended

I understand that only with a CG minimizer and appropriate linesearch
one should expect that the energy decreases at every timestep. Is it a
generally expected property of the OT method? In any case, only the
first SCF starting from an atomic density guess is slow to converge,
and as soon as the MD is started, the SCF converges in roughly 10
iterations, which seems okay to me.

Best regards,
Laurent

[Iain Bethune]

Glad you have it working. I’m not 100% sure on the minimizer, maybe someone else with more knowledge in this area can answer. I didn’t spot that you had chaged from CG to DIIS. Certainly with CG you do expect the energy to be strictly decreasing.

- Iain

--

Iain Bethune
Project Manager, EPCC

Email: i.be...@epcc.ed.ac.uk
Twitter: @IainBethune @PrimeGrid @CP2Kproject
Web: http://www2.epcc.ed.ac.uk/~ibethune
Tel/Fax: +44 (0)131 651 7183/6555
Mob: +44 (0)7598317015
Addr: 2404 JCMB, The King's Buildings, Peter Guthrie Tait Road, Edinburgh, EH9 3FD

[ganta.pra...@gmail.com]

hi can u post your working arch file for  Linux-x86-64-intel-host.popt and Linux-x86-64-intel-host.psmp. I am trying to compile 4.1 version and I could not able to set the directory paths and library paths properly. I will be of great reference for me to write my arch file. I am trying to install on a server. 

[Laurent Joly]

Hi,

I only compiled the psmp version. Here is the arch file I used.

Best regards,