Excessive Real-Time vs. CPU-Time in Multi-Stage Molpro Calculations

[Pavel Rublev]

Greetings! 

I’m encountering a puzzling performance issue in a multi-stage calculation sequence (UHF → AVAS → MCSCF) using Molpro. Each stage reports a reasonable CPU time (on the order of ~500-1000 seconds), yet the corresponding wall-clock (real) time is extraordinarily high (~20,000-50,000 seconds per stage). This discrepancy implies that the job experiences significant delays or idle periods between these steps.  

I tried to execute my input file on different HPC with no significant difference (see provided outputs). Has anyone else encountered a similar problem, or does anyone have insights on how to mitigate such large wall-time overheads? 

Thank you in advance for your help and support! 

[Peter Knowles]

This runs fast on a Mac laptop, M1 Max, in about 5 minutes for the UHF. Maybe the memory is overcommitted in your environment?

Peter

On 5 Feb 2025, at 03:27, Pavel Rublev <litium...@gmail.com> wrote:

External email to Cardiff University - Take care when replying/opening attachments or links.

Nid ebost mewnol o Brifysgol Caerdydd yw hwn - Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.

--
You received this message because you are subscribed to the Google Groups "molpro-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to molpro-user...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/molpro-user/dc1a9cc5-2175-4d3b-afad-9b81355994a9n%40googlegroups.com.

<MOLPRO_Expanse.out><MOLPRO_Perlmutter.out>

[tibo...@gmail.com]

Something to watch out for is the number of threads being spawned, sometimes the "molpro" start script and the clusters job scheduler get confused about how many MPI/OpenMP threads should be launched, should the threads be pinned to specific CPU cores/NUMA nodes, etc.

Once I have seen a situation where all MPI threads got pinned to 1 core by SLURM, or each MPI thread spawned 128 OpenMP threads...neither was good for performance.

I can also recommend checking how much time the molpro process spends in state "D". In HPC environments it is common that the disk system (especially the default /home or general-purpose data storage partition) is relatively low-performance, and not well suited for quantum chemistry programs that write and read from the disk intensely. Usually there is either some node-local storage, or a high-speed scratch file system on the network that should be used for Molpro.

In short: Check that the expected number of MPI and OpenMP threads are running, check that no more than 1 Molpro thread is pinned to the same physical core, make sure that you dont have a disk bottleneck.

Best,

Tibor Győri