cifar10 fails when training on multiple Nvidia Tesla P100 GPUs

[Michael Chen]

Hi,

I've been looking into an issue where cifar10 fails to start training when running on multiple P100 GPU cards.  1 P100 is fine, 1 or more M40 is fine, this only failed on multiple P100.  

Here is the corresponding github issue I've submitted: https://github.com/NVIDIA/caffe/issues/422

cifar10 gets stuck at this line: CUDA_CHECK(cudaStreamSynchronize(comm_stream_->get())); inside parallel.cpp

Has anyone else encountered this problem and has a work around or solved it?

Thanks,

Michael

[Michael Chen]

Hi,

Just wanted to update everyone on the solution.

We engaged with Nvidia and found out it was related to some ACS bios settings that was not set correctly because our BIOS was not up to date.  Once the ACS setting was correct, GPU cards were able to talk to each other through the P2P bus.

Thanks,

Michael