Problem in the preconditioner when using UKS

[Daniele Ongari]

Dear CP2K developers, 
I want to report a serious problem with the preconditioner when computing the energy of Fe-MOF-74 with UKS settings.
Long story short the energy goes down, it is close to convergence but then, after recomputing the preconditioner in the outer step it starts to diverge: see the image.

Now, I attached the input and the output, but I already tried very different settings:
-smearing
-diagonalization without smearing
-different BS
-UKS false (converged, but to a higher energy, forcing the closed shell)
- starting magnetization on metals
- CG minimizer
- lower multiplicity
- 2x and 3x replicated unit cell in the shortest dimension
- lowering ENERGY_GAP to 0.001
These calculation ALL failed (I made 16 attempts in total, changing settings!).

And at the end the only one working (the 17th attempt, ironically the unlucky number for Italians!) was to use CG minimizer with no outer SCF, hence not recomputing the preconditioner. It converged after ~600 inner steps, to:

 HOMO - LUMO gap [eV] :    1.917671

 HOMO - LUMO gap [eV] :    0.235602

 ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.):            -1170.045288008604075

Please let me know what was the problem, if it is a known issue of the preconditioner and how to circumvent it: for the moment I'm using no outer steps to have a robust convergence, but I know that this may cause the convergence to a local miniimum!

Thanks a lot!

Daniele 
PhD candidate, LSMO, EPFL Sion

[hut...@chem.uzh.ch]

Hi

yes, these high spin state systems can be tricky to converge.
Your system did rather smoothly converge for me with the following
protocol:

- optimize with CG and OUTER_SCF every 10 steps until eps_scf 1.e-4.
- restart optimization with DIIS until eps_scf 1.e-7 (is rather slow at the end)

final energy: -1170.0452882317

regards

Juerg

--------------------------------------------------------------
Juerg Hutter Phone : ++41 44 635 4491
Institut für Chemie C FAX : ++41 44 635 6838
Universität Zürich E-mail: hut...@chem.uzh.ch
Winterthurerstrasse 190
CH-8057 Zürich, Switzerland
---------------------------------------------------------------

-----cp...@googlegroups.com wrote: -----
To: "cp2k" <cp...@googlegroups.com>
From: "Daniele Ongari"
Sent by: cp...@googlegroups.com
Date: 09/19/2018 12:50PM
Subject: [CP2K:10750] Problem in the preconditioner when using UKS

Dear CP2K developers,
I want to report a serious problem with the preconditioner when computing the energy of Fe-MOF-74 with UKS settings.
Long story short the energy goes down, it is close to convergence but then, after recomputing the preconditioner in the outer step it starts to diverge: see the image.

Now, I attached the input and the output, but I already tried very different settings:
-smearing
-diagonalization without smearing
-different BS
-UKS false (converged, but to a higher energy, forcing the closed shell)
- starting magnetization on metals
- CG minimizer
- lower multiplicity
- 2x and 3x replicated unit cell in the shortest dimension
- lowering ENERGY_GAP to 0.001
These calculation ALL failed (I made 16 attempts in total, changing settings!).

And at the end the only one working (the 17th attempt, ironically the unlucky number for Italians!) was to use CG minimizer with no outer SCF, hence not recomputing the preconditioner. It converged after ~600 inner steps, to:

HOMO - LUMO gap [eV] : 1.917671
HOMO - LUMO gap [eV] : 0.235602

ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -1170.045288008604075

Please let me know what was the problem, if it is a known issue of the preconditioner and how to circumvent it: for the moment I'm using no outer steps to have a robust convergence, but I know that this may cause the convergence to a local miniimum!

Thanks a lot!

Daniele
PhD candidate, LSMO, EPFL Sion

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To post to this group, send email to cp...@googlegroups.com.
Visit this group at https://groups.google.com/group/cp2k.
For more options, visit https://groups.google.com/d/optout.


[attachment "energy.inp" removed by Jürg Hutter/at/UZH]
[attachment "energy.out" removed by Jürg Hutter/at/UZH]
[attachment "Fe-MOF-74_h111.subsys" removed by Jürg Hutter/at/UZH]

[Daniele Ongari]

Thank you very much!
The protocol you proposed is a very smart compromise between efficiency and robustness.

Daniele

[Daniele Ongari]

Hi,
here I come with a similar problem for Co-MOF-74 using different settings.
In the legend of the figure:
"diis" is using OT DIIS with 50 inner and 10 outer SCF cycles
"cg" is using OT CG with 2000 inner and no outer SCF cycles


I think the problem is related to the bandgap going close to zero and making the preconditioner to diverge, but this time the energy assumes even lower energies than the final ones. 
Does it makes sense to have a lower energy being DFT not strictly variational, and this should be seen as a merely mathematical mess?

The calculation is converging when I run the OT CG minimization without outer steps: can I consider the final result of this calculation as reliable?

Thanks

Daniele

[Krack Matthias (PSI)]

Dear Daniele

 

You should take note of the electron count in the OT DIIS output and ask yourself, if the corresponding energy has any meaning.

The OT CG run converged to some state at least.

 

HTH

 

Matthias

--

[hut...@chem.uzh.ch]

Hi

with systems that are forced to have such high multiplicity it
is often difficult to find the correct minimum.
In addition I think you should tighten the accuracy. I would
decrease EPS_DEFAULT to 1e-12 or 1e-14 and maybe further increase
the cutoff.
At the end it will also be a good idea to check the orbital energies.

regards

Juerg

--------------------------------------------------------------
Juerg Hutter Phone : ++41 44 635 4491
Institut für Chemie C FAX : ++41 44 635 6838
Universität Zürich E-mail: hut...@chem.uzh.ch
Winterthurerstrasse 190
CH-8057 Zürich, Switzerland
---------------------------------------------------------------

-----cp...@googlegroups.com wrote: -----
To: "cp2k" <cp...@googlegroups.com>
From: "Daniele Ongari"
Sent by: cp...@googlegroups.com

Date: 10/03/2018 07:46PM
Subject: [CP2K:10799] Re: Problem in the preconditioner when using UKS

Il giorno mercoledì 19 settembre 2018 12:50:34 UTC+2, Daniele Ongari ha scritto:

Dear CP2K developers,
I want to report a serious problem with the preconditioner when computing the energy of Fe-MOF-74 with UKS settings.
Long story short the energy goes down, it is close to convergence but then, after recomputing the preconditioner in the outer step it starts to diverge: see the image.

Now, I attached the input and the output, but I already tried very different settings:
-smearing
-diagonalization without smearing
-different BS
-UKS false (converged, but to a higher energy, forcing the closed shell)
- starting magnetization on metals
- CG minimizer
- lower multiplicity
- 2x and 3x replicated unit cell in the shortest dimension
- lowering ENERGY_GAP to 0.001
These calculation ALL failed (I made 16 attempts in total, changing settings!).

And at the end the only one working (the 17th attempt, ironically the unlucky number for Italians!) was to use CG minimizer with no outer SCF, hence not recomputing the preconditioner. It converged after ~600 inner steps, to:

HOMO - LUMO gap [eV] : 1.917671
HOMO - LUMO gap [eV] : 0.235602

ENERGY| Total FORCE_EVAL ( QS ) energy (a.u.): -1170.045288008604075

Please let me know what was the problem, if it is a known issue of the preconditioner and how to circumvent it: for the moment I'm using no outer steps to have a robust convergence, but I know that this may cause the convergence to a local miniimum!

Thanks a lot!

Daniele
PhD candidate, LSMO, EPFL Sion

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To post to this group, send email to cp...@googlegroups.com.
Visit this group at https://groups.google.com/group/cp2k.
For more options, visit https://groups.google.com/d/optout.

[attachment "cg.inp" removed by Jürg Hutter/at/UZH]
[attachment "cg.out" removed by Jürg Hutter/at/UZH]
[attachment "diis.inp" removed by Jürg Hutter/at/UZH]
[attachment "diis.out" removed by Jürg Hutter/at/UZH]

[Daniele Ongari]

Thank you a lot!
This was a very good hint. Indeed, it is clear from the  "Total charge density on r-space grids" that something is going weird with the count of electron as soon as the energy starts to diverge!

$egrep 'outer SCF|Total charge density on r-space grids' diis.out
diis.out:  Total charge density on r-space grids:        0.0000001146
diis.out:  outer SCF iter =    1 RMS gradient =   0.47E-04 energy =      -1301.1851926636
diis.out:  Total charge density on r-space grids:        0.0000001146
diis.out:  outer SCF iter =    2 RMS gradient =   0.42E-04 energy =      -1301.2195780393
diis.out:  Total charge density on r-space grids:        0.0000001047
diis.out:  outer SCF iter =    3 RMS gradient =   0.14E-05 energy =      -1301.2275937198
diis.out:  Total charge density on r-space grids:        0.0003332641
diis.out:  outer SCF iter =    4 RMS gradient =   0.62E-05 energy =      -1301.2754195817
diis.out:  Total charge density on r-space grids:       -0.0424028684
diis.out:  outer SCF iter =    5 RMS gradient =   0.96E-04 energy =      -1318.9753098223
diis.out:  Total charge density on r-space grids:        0.5522536311
diis.out:  outer SCF iter =    6 RMS gradient =   0.13E-03 energy =      -1317.2906799313
diis.out:  Total charge density on r-space grids:        3.4918015156
diis.out:  outer SCF iter =    7 RMS gradient =   0.52E-03 energy =      -1315.3866686940
diis.out:  Total charge density on r-space grids:        4.3723629910
diis.out:  outer SCF iter =    8 RMS gradient =   0.97E-03 energy =      -1298.6196710054
diis.out:  Total charge density on r-space grids:        9.1128313578
diis.out:  outer SCF iter =    9 RMS gradient =   0.20E-02 energy =      -1284.6336624946
diis.out:  Total charge density on r-space grids:        8.4010321552
diis.out:  outer SCF iter =   10 RMS gradient =   0.20E-02 energy =      -1278.9516795867
diis.out:  Total charge density on r-space grids:        7.2155551466
diis.out:  outer SCF iter =   11 RMS gradient =   0.20E-02 energy =      -1285.4339788964
diis.out:  outer SCF loop FAILED to converge after   11 iterations or  550 steps

I'm now wondering: is there a way to stop CP2K as soon as the  "Total charge density on r-space grids"  exceeds a threshold (let's say 1.0) to avoid wasting time and switch to the CG algorithm without outer SCF?
I'm using this 2 step protocol for high-throughput DFT calculations, therefore it would be helpful not to waste time!

Regards,
Daniele

[Krack Matthias (PSI)]

Dear Daniele

 

I am not aware of any control key to stop an SCF run as soon as the electron count becomes unreasonable. It shouldn’t be too difficult, however, to implement such a control. The use of an outer SCF should converge faster in most cases. Did you try already a different preconditioner like FULL_SINGLE_INVERSE with OT CG and an outer SCF? If such choice does not result in a robust convergence behaviour then you are dealing with problematic systems which you can hardly run in a black-box manner.

 

Best,

 

Matthias

[Daniele Ongari]

Dear Dr. Krack,
for the Fe-MOF-74 case I tried already FULL_KINETIC but it didn't work, so I neglected that any preconditioner which is assumed to be less robust than FULL_ALL would work. I will try anyway to test the different ones. 
I agree that a black-box approach is not generally a good idea for these cases but practically speaking this 2 step protocol that we are using is working well, and to give an idea, out of ca. 300 frameworks (MOFs, COFs and zeolites), less then 10% passed to to the second step (i.e., CG + no outer SCF). So, I'm thinking if I can improve the efficiency even more but I'm already happy with these results!

Daniele

[Daniele Ongari]

Dear Prof. Hutter, I tried different EPS_DEFAULT but this is not affecting the final result.

Thanks,

Daniele