Horovod Performance with OpenMPI

[Shruti Sharma]

Hi 

I am currently running Horovod benchmarks in an intra-node setup. However, I have observed that increasing the number of GPUs does not result in a proportional increase in total throughput. Specifically, the throughput per GPU with a single GPU is approximately 842.6 ± 2.4, whereas with two GPUs, the total throughput is around 485.7 ± 44.8, which translates to approximately 242.8 ± 22.4 per GPU.

The configuration for the test is:

MPI : OpenMPI - 5.0.6

HOROVOD : 0.28.1

pytorch : 1.12.1

GPU : NVIDIA A100 

CUDA : 11.8

Python : 3.10

GCC : 8.5.0

command : mpirun -n 1 --report-bindings python pytorch_synthetic_benchmark.py -batch-size=64 --model=resnet50

[gpu39:59123] Rank 0 bound package[0][core:0]

Model: resnet50

Batch size: 64

Number of GPUs: 1

Running warmup...

Running benchmark...

Iter #0: 844.3 img/sec per GPU

Iter #1: 844.0 img/sec per GPU

Iter #2: 843.6 img/sec per GPU

Iter #3: 843.5 img/sec per GPU

Iter #4: 843.5 img/sec per GPU

Iter #5: 842.0 img/sec per GPU

Iter #6: 841.3 img/sec per GPU

Iter #7: 841.8 img/sec per GPU

Iter #8: 841.1 img/sec per GPU

Iter #9: 841.1 img/sec per GPU

Img/sec per GPU: 842.6 +-2.4

Total img/sec on 1 GPU(s): 842.6 +-2.4

Run with two GPU(s) on the same node

command : mpirun -n 2 --report-bindings python pytorch_synthetic_benchmark.py -batch-size=64 --model=resnet50

[gpu39:59166] Rank 0 bound package[0][core:0]

[gpu39:59166] Rank 1 bound package[0][core:1]  

Model: resnet50

Batch size: 64

Number of GPUs: 2

Running warmup...

Running benchmark...

Iter #0: 235.7 img/sec per GPU

Iter #1: 251.5 img/sec per GPU

Iter #2: 217.0 img/sec per GPU

Iter #3: 239.4 img/sec per GPU

Iter #4: 257.2 img/sec per GPU

Iter #5: 258.3 img/sec per GPU

Iter #6: 248.4 img/sec per GPU

Iter #7: 242.6 img/sec per GPU

Iter #8: 238.0 img/sec per GPU

Iter #9: 240.3 img/sec per GPU

Img/sec per GPU: 242.8 +-22.4

Total img/sec on 2 GPU(s): 485.7 +-44.8

[George Bosilca]

What's the network on your cluster ? Without a very good network you cannot obtain anything closer to the single GPU, because the data exchanged between the two GPUs will become the bottleneck.

  George.

To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@lists.open-mpi.org.

[Pritchard Jr., Howard]

Hi Shruti,

 

What version of NCCL is installed on the system?  Horovod has environment variables you can set to force use of NCCL.

 

You may want to use nvidia-smi to double check whether the benchmark is actually using both GPUs when using two mpi processes.

 

Also, you may want to consult with a technical assistant like chat-gpt o4-mini about this problem.  The assistant may prove to be quite helpful.

 

Howard

[George Bosilca]

Please ignore my prior answer, I just noticed you are running single-node.

In addition to Howard's suggestions, check if you have nvlink between GPUs. 

  George.