Optimizing Molpro Efficiency For Single Node Slurm Jobs

[Frank Pari]

Good afternoon,

Working with Molpro 2022 through our Slurm job scheduler has proven somewhat cumbersome for our researchers due to the resource confinements we have in place.  We've worked out a set of SBATCH directives that handles the helper process needing its own task and how to ensure the computation processes get distributed evenly across sockets and Numa channels.  But, I'm almost completely new to using Molpro and still struggling with memory usage and restarts.

First, I'd like to say thank you in advance to whomever chooses to respond to the following questions.  I wholeheartedly appreciate your efforts compensate for my ignorance and inability to parse the Molpro documentation!

Our typical compute node has:

RHEL 8

Slurm 19 (upgrading to 23 in a month or so)

64 cores

250G of RAM

400G of SWAP

Now for the questions:

1. For single server/host/node operations
   • Is it necessary to still use MPI with multiple process or is there a more efficient path
   • Is it faster to use global arrays with fewer processes vs more processes using scratch space on our shared GPFS storage (10Gbps max)
2. What is the detriment to allocating the minimum amount of megawords vs the mid-sze suggested during the run or the max needed for triples
   
   
3. For restarts
   • Is there a best practice for translating potential restart points for Molpro into Slurm checkpoints
   • Since some phases of processing use very little memory and are single threaded, is it possible to stop the job before such a phase to restart the job with more computation processes and then stop/restart it again before the next phase that needs more memory
     • Assuming the answer is no or highly specific to what's being done in the job
     • May not be worth the effort since for the majority of the processing time after the first few hours the allocated memory is fully used
4. Are there best practices for tuning a host  OS (RHEL 8) settings for optimal Molpro performance

-F