Does CP2K allow a multi-GPU run?

[ASSIDUO Network]

I currently have access to 4 GPUs to run an AIMD simulation, but only one of the GPUs are being used. Is there a way to use the other 3, and if so, can you tell me how to set it up with a PBS job?

[ASSIDUO Network]

I'm asking, since the administrator running my country's HPC is saying that although I'm requesting access to 4 GPUs, CP2K is only using 1. I checked the following output:

 DBCSR| ACC: Number of devices/node                                            4

And it shows that CP2K is picking up 4 GPUs.

[Alfio Lazzaro]

The way to use 4 GPUs per node is to use 4 MPI ranks. How many ranks are you using?

[Lenard Carroll]

This is what my PBS file looks like:

#!/bin/bash

#PBS -P <PROJECT>

#PBS -N <JOBNAME>

#PBS -l select=1:ncpus=40:ngpus=4

#PBS -l walltime=08:00:00

#PBS -q gpu_4

#PBS -m be

#PBS -M none

module purge

module load chpc/cp2k/8.1.0/cuda10.1/openmpi-4.0.0/gcc-7.3.0

source $SETUP

cd $PBS_O_WORKDIR

cp2k.psmp -i gold.inp -o gold_pbc.out

~                                                                                                                       ~                                         

--
You received this message because you are subscribed to the Google Groups "cp2k" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cp2k+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/70ba0fce-8636-4b75-940d-133ce4dbf0can%40googlegroups.com.

[Alfio Lazzaro]

Hi,

Your command to run CP2K doesn't mention MPI (mpirun, mpiexc, ...). Are you running with multiple ranks?

You can check those lines in the output:

 GLOBAL| Total number of message passing processes                            32

 GLOBAL| Number of threads for this process                                    4

And check your numbers.

I can guess you have 1 rank and 40 threads.

To use 4 GPUs you need 4 ranks (and less threads per rank), i.e. something like

export OMP_NUM_THREADS=10

mpiexec -n 4 ./cp2k.psmp -i gold.inp -o gold_pbc.out

Please check with your sysadmin on how to run with multiple MPI ranks.

Hope it helps.

Alfio

[Lenard Carroll]

He suggested I try out:

mpirun -n 10 cp2k.psmp -i gold.inp -o gold_pbc.out

as he is hoping that will cause the 1 GPU to use 10 CPUs over the selected 4 GPUs.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/92e4f88d-fde8-4127-ab5f-0b98bbbba8ebn%40googlegroups.com.

[Lenard Carroll]

It says the following for me:

 GLOBAL| Total number of message passing processes                             1

 GLOBAL| Number of threads for this process                                   32

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/92e4f88d-fde8-4127-ab5f-0b98bbbba8ebn%40googlegroups.com.

[Lenard Carroll]

Correction, he told me to use:

mpirun -np 10 cp2k.psmp -i gold.inp -o gold_pbc.out

but it didn't run correctly.

[Alfio Lazzaro]

Could you try what I suggested:

export OMP_NUM_THREADS=10

mpirun -np 4 ./cp2k.psmp -i gold.inp -o gold_pbc.out

Please check the corresponding log.

As I said above, you need an MPI rank per GPU and you told us that you have 4 GPUs, so you need 4 ranks (or multiple). With 10 you get unbalance.

[Lenard Carroll]

Shall do. I already set it up, but it's in a long queue.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/59a635d8-0f0c-4dc5-abaf-b8bbe3c18da5n%40googlegroups.com.

[Alfio Lazzaro]

Wait, I see you have 32 threads in total, so need to have 32/4 = 8 threads.

Please change

export OMP_NUM_THREADS=8

[Lenard Carroll]

I am using 30 threads now over 3 GPUs, so I used:

export OMP_NUM_THREADS=10

mpiexec -n 3 cp2k.psmp -i gold50.inp -o gold50.out

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/ec4efd81-6314-4ce7-b22c-148b362d2ba6n%40googlegroups.com.

[Alfio Lazzaro]

And it works? Check the output and the performance... It can be that your particular test case doesn't use the GPU at all, so could you attach the log (at least the final part of it)

[Lenard Carroll]

Not sure yet. The job is still in the queue. As soon as it is finished I'll post the log file info here.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/d29306aa-e0b8-4797-9298-13dab23e9083n%40googlegroups.com.

[Lenard Carroll]

Here's the log file. The job ended prematurely.

[Alfio Lazzaro]

I'm sorry, I cannot assist you, I'm not an expert on how to use CP2K ('m not a domain scientist). Without the total log, I can help you...

I assume you should have a log file from PBS where you can see the error message. I can assume it is a memory limit.

Have you executed on a CPU only?

[Lenard Carroll]

Oh you meant the error file. Please find it attached.

I have run on CPU only and one GPU. It works.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/6852eb71-6886-4fe7-8f4a-3ad8318a289dn%40googlegroups.com.

[fabia...@gmail.com]

Hi,

cp2k is crashing when COSMA tries to access a gpu ("error: GPU API call : invalid resource handle"). On cray systems there is the environment variable "export CRAY_CUDA_MPS=1" that has to be set. Otherwise only one mpi rank can access a specific GPU device. Maybe there is a similar setting for your cluster?

Also cp2k can be memory hungry. Setting "ulimit -s unlimited" is often needed.

I hope this helps,

Fabian

[Lenard Carroll]

I'll play a bit around with it, but ultimately I'll have to forward this to an HPC administrator.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/c2033277-5fdf-4e98-9329-e9a289a5b277n%40googlegroups.com.

[rongpei cui]

Hello, I am also participating in HPC. I would like to ask for some questions about running test files.

[Lenard Carroll]

Dear Fabian. COSMA wasn't installed with CP2K, so that can't be the issue. The HPC system is not CRAY, but I did ask the HPC admin to look into it.

On Thu, Apr 22, 2021 at 8:02 PM fabia...@gmail.com <fabia...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/c2033277-5fdf-4e98-9329-e9a289a5b277n%40googlegroups.com.

[Alfio Lazzaro]

The error in the log says that COSMA is used:

#7  0x2dfb43b in check_runtime_status

at /apps/chpc/chem/gpu/cp2k/8.1.0/tools/toolchain/build/cosma-2.2.0/libs/Tiled-MM/src/Tiled-MM/util.hpp:17

#8  0x2dfb43b in _ZNK3gpu13device_stream13enqueue_eventEv

at /apps/chpc/chem/gpu/cp2k/8.1.0/tools/toolchain/build/cosma-2.2.0/libs/Tiled-MM/src/Tiled-MM/device_stream.hpp:62

#9  0x2dfb43b in _ZN3gpu11round_robinIdEEvRNS_12tiled_matrixIT_EES4_S4_RNS_13device_bufferIS2_EES7_S7_iiiS2_S2_RNS_9mm_handleIS2_EE

at /apps/chpc/chem/gpu/cp2k/8.1.0/tools/toolchain/build/cosma-2.2.0/libs/Tiled-MM/src/Tiled-MM/tiled_mm.cpp:248

....

[Lenard Carroll]

I see. The HPC admin said he compiled it without COSMA. I have informed him of this.

To view this discussion on the web visit https://groups.google.com/d/msgid/cp2k/67838a92-f8a7-4c35-a1bf-3d0edfccc0dcn%40googlegroups.com.

[ASSIDUO Network]

Thanks for letting me know. I forwarded this to the admin and he confirmed that COSMA was compiled with CP2K, even though he specified it not to be. He has fixed this now and I've been able to use all my GPU resources.

Thanks for the help.