NVLINK - Cuda aware MPI - single node performance

29 views
Skip to first unread message

Michael Laufer

unread,
Oct 17, 2020, 5:51:04 PM10/17/20
to PyFR Mailing List
Hi,
We have 4 Nvidia SXM2 V100's on a single node. I noticed that our system-available  MPI library (Openmpi 3.1.0) was not compiled with any Cuda support, and I am worried that we may be losing out on some performance. This version of the V100 should support extremely fast GPU-GPU communication (Through NVLINK), but I am unsure if we are effectively using this capability without compiling MPI with cuda support.

Do you know if there can expect any performance gains by compiling Openmpi with cuda support? 
 
Thank you,

Freddie Witherden

unread,
Oct 18, 2020, 11:08:41 AM10/18/20
to pyfrmai...@googlegroups.com
Hi Michael,

On 17/10/2020 16:51, Michael Laufer wrote:
> Do you know if there can expect any performance gains by compiling
> Openmpi with cuda support? 

In general the CUDA awareness only helps cases which are heavily strong
scaled, where performance is limited by the interconnect. If, whilst
running a simulation, nvidia-smi reports that all four GPUs are close to
100% utilisation then this is unlikely to be the case.

Regards, Freddie.
Reply all
Reply to author
Forward
0 new messages