On the time&memory consumption of ERI calculation in CP2K hybrid functional single point

[Roosevelt Gildner]

Dear CP2K team,

I am attempting to calculate the single point energy of a huge periodic cluster (near 5k atoms, just contains C H O N, cell size 30+ angstrom, only Gamma point). Due to the demand of following analysis, I used B3LYP hybrid functional (with D3(BJ)) and 6-31G(d) basis for GAPW calculation.

I have prepared enough memory space to do this (TB scale). Meanwhile I want as accurate result (especially for DOS analysis, need unoccupied orbitals) as possible, but not too excessive time comsuption.

I have tried several parameter sets and found when the EPS_SCHWARZ and CUTOFF_RADIUS is respectively over 8 (exponent) and 10 angstrom, the time comsuption significant increases.

Below are the two typical examples:

A

EPS_SCHWARZ=1E-8.

SCREEN_ON_INITIAL_P=T

CUTOFF_RADIUS=8.0.

The total time comsuption is about 25 hours, memory peak 4.2TB(not on-the-fly). The HPC monitor tells me that the memory linear rised to the stable level and soon the first SCF step completed.(total SCF=24 steps)

B

EPS_SCHWARZ=1E-10.

SCREEN_ON_INITIAL_P=T

CUTOFF_RADIUS=10.0

The memory comsuption stablized at about 7TB. But it has been 1 day (now already 5 days) since the memory rised to the stable level, the first SCF step is not completed yet.

Here are my confusions:

1 Whether the memory rising to the stable level means the ERI calculation is completed? if yes, why the example B does not complete the first SCF step even after 1-5 days? (the memory used is not 100%, so i believe it is not on-the-fly mode)And why do example A and B take such different time comsuption?

2 I am afraid the CP2K mission B got stuck. but I am not sure. Is there any method to judge whether the mission is normal? And need I continue waiting?

3 Which parameter(s) is the key factor(s) affecting the time comsuption of hybrid functional calculation? EPS_DEFAULT? EPS_PGF_ORB? EPS_SCHWARZ? CUTOFF_RADIUS?

[Augustin Bussy]

Hi,

what you observe is somewhat expected. The limiting factor in your B3LYP calculation is the Hartree-Fock exchange, which computational performance depends strongly on the choice of EPS_PGF_ORB, EPS_SCHWARZ,  CUTOFF_RADIUS and MAX_MEMORY.

In an ideal CP2K run, when performing HFX calculations, all of the ERIs are initially calculated and stored in memory (during the first SCF step). This allows for much faster timings for the next SCF steps, because ERIs are simply fetched from memory. The keyword MAX_MEMORY from the &HF section controls how much memory is dedicated to storing the ERIs (per MPI rank). If there are more ERIs than available memory, the rest is computed on the fly, at each SCF step. This is linked to your first question: when the memory stabilizes, it means that you have reached that MAX_MEMORY upper limit, and the new ERIs are not stored. You should adapt the value of MAX_MEMORY to your hardware.

For your second and third question: HFX performance in CP2K depends a lot on the choice of EPS_SCHWARZ, EPS_PGF_ORB and CUTOFF_RADIUS. These parameters control how many ERIs need to be calculated, and going from 10E-8 to 10E-10 EPS_SCHWARZ while increasing the CUTOFF_RADIUS will lead to a large number of additional ERIs to be calculated (and CP2K might seems stuck because there is so much more to do). In most cases, I would say that EPS_SCHWARZ = 1.0E-8,  EPS_PGF_ORB =1.0E-5 - 1.0E-6 and CUTOFF_RADIUS = 6 Angstrom is accurate enough and efficient.

You can also consider the ADMM method to further accelerate your calculation (https://pubs.acs.org/doi/full/10.1021/ct1002225).

I hope that helps.

Best,

Augustin

---

From: cp...@googlegroups.com <cp...@googlegroups.com> on behalf of Roosevelt Gildner <roosevelt...@gmail.com>
Sent: Saturday, September 23, 2023 4:29 PM
To: cp2k <cp...@googlegroups.com>
Subject: [CP2K:19271] On the time&memory consumption of ERI calculation in CP2K hybrid functional single point

 

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/57a56625-e467-4a50-ab54-665e7df6d445n%40googlegroups.com.

[Roosevelt Gildner]

Dear Dr. Bussy

Really appreciate your detailed answer.

First apologizes for the lack of my detailed parameters description. In fact, I have considered about the MAX_MEMORY.

For the aforementioned example A and B, I have spared enough MAX_MEMORY. My setting strategy is: MAX_MEMORY* MPI ranks=90% of the physical total memory of the HPC array.

Therefore, with respect to the example B, I believe it is not of on-the-fly mode. For the stablized memory consumption is approximately 70%, no more than my setting, 90% of the total memory.

(The attached two figures are the HPC performance curves v.s. time. In which the deep yellow line refers to the real-time memory consumption percent, corresponding to the left Y axis)

The EPS_SCHWARZ and EPS_PGF_ORB are both set as 1E-8. I believe they are accurate enough, and actually it works well in example A. So I think these two parameters might not be the key factor.

Perhaps I have to loose the EPS_SCHWARZ and CUTOFF_RADIUS. However, it is really a pity that although I have enough physical memory space, when the memory consumption rises from 4TB (example A) to 7TB (example B), it seems to get stuck. It is really strange. Maybe the inherent limitation of CP2K?(I use psmp version, but the OpenMPI is default)

P.S. Thank you for your advice on ADMM. I have learned that ADMM is usually adopted with OT to realize the purification of the aux. basis. However, what I am interested in is the unoccupied orbitals of my structure. Therefore the strategy of ADMM+OT will not be an option.

Thank you again for your help. And for the creation of such a great ab initio software!

Regards,

RG