Analysing Memory Consumption

[Peter Gillespie]

Hello Everyone,

I am trying to determine the best memory consumption settings for a large HF/DFT hybrid functional calculation. After discovering that setting the global print level to Medium will print out the (estimated) memory consumption before and after including the HFx module, I've been able to collect some numbers to compare to the max resources available.

Firstly, just as a sanity check: Am I correct to assume that the MEMORY setting in the CP2K input file and the printouts of memory consumption are written in Mebibytes rather than Megabytes? (as the listed unit is MiB rather than MB)

Secondly, are the memory consumption data always written as per MPI process, or in total for the entire calculation?

Kind Regards

-Peter

[Alfio Lazzaro]

Hi Peter,

in the past I did some tests with the memory analysis, I hope that my answer can be useful.

First of all, for the memory consumption CP2K reads at /proc/self/statm, which returns the size in memory pages. Then it multiplies this value by the return value of the function getpagesize, which gives the memory in bytes. Finally it prints the value as:

m_memory+(1024*1024)-1)/(1024*1024)

Therefore, yes, it is Mebibytes. For a better estimation, I also suggest to use the compiler flag  -D__STATM_TOTAL, which give the total memory, i.e. including virtual memory.

Concerning the second question, I think the output comes from the rank 0 only...

Alfio

[Hans Pabst]

I wonder how this is correct in the era of THP. At least it will overestimate the actual memory consumption, but generally it depends on system settings such as available number of huge pages.