Molpro I/O

[Patrick BOUSQUET-MELOU]

A user of Normandy HPC center, CRIANN, tends to saturate the parallel file system (GPFS) of the cluster. His jobs make periodically a 20 GB/s of I/O throughput (the maximum bandwidth of the cluster storage is ~ 25 GB/s).

The researcher makes CCSD(T)-F12/VTZ computations for polycyclic aromatic hydrocarbons (up to 150 electrons systems), with Molpro 2021.1 MPI multinode jobs.

Limiting the number of simultaneous jobs of this type should help.

But are there some hints, in Molpro input parameters, for reducing I/O ?

[pala...@gmail.com]

Hello,

molpro creates during the calculation realtively large temporary files and needs to often read from them and save on them. You can choose a location of a scratch directory were these files are stored using the -d option, see here for more information. You should use ramdisk as a scratch directory located on yeach node.

I hope this helps you,

Stanislav

Dne úterý 29. června 2021 v 11:25:28 UTC+2 uživatel bousque...@gmail.com napsal:

[qia...@theochem.uni-stuttgart.de]

Molpro generally requires local disk storage and is currently not designed to make use of network file systems. If the nodes have local storage and Molpro is not using it (check "Working directory" at the beginning of the output file), the "-d" command line option can be passed to specify the directory.

If the nodes do not have local storage, one can try using tmpfs as the local scratch by passing "-d /dev/shm". Naturally, it is necessary to reduce the amount of memory allocated to Molpro with the memory card and leave sufficient physical memory for tmpfs and distribute memory (GlobalArrays). 500 MW per process should be a reasonable value for these calculations. For PNO-LCCSD calculations, disk (or tmpfs) usage should reduce with the number of nodes. Invoking the PNO calculation with "pno-lccsd(t)-f12, PNO_3EXTK_IMPL=1, PNO_4EXTK_IMPL=1" may further reduce the disk usage in multi-node calculations at the cost of extra cross-node communication.

[tibo...@gmail.com]

For a long term solution, I recommend upgrading at least some of the nodes with high endurance SSDs, to act as node-local scratch storage.

If you are using SLURM as the queuing system, it should not be too difficult to put the upgraded nodes into a separate "highIO" partition, with a different TmpDisk value.

If node upgrades are not feasible, then the advice given above is probably the best you are going to get. You either convince the user to use more approximate methods that use less I/O or figure out how to hook up your queuing system to give Molpro jobs a ramdisk.

I have one more thing to add. It may be possible to mitigate the I/O deluge by (ab)using the Linux page cache's ability to cache writes in RAM for a long time. It works well if you have lots of free RAM and slow node-local storage, but I have no idea if GPFS uses the page cache for writes, like normal block devices do. We use it to spare our SSDs from as much write as possible, but it requires some risky settings in /etc/sysctl.conf.