Run caffe in multithread and multiGPU windows c++

[pedromol...@gmail.com]

I`m working in a c++ project (VS2105) in windows environement that lauches multiple threads. PC has two NVIDIA GeForce GTX 1080. Each thread uses a different caffe classifier in GPU mode.

If all threads use one GPU, the program is working well. It works well both using GPU 0 or using GPU1. The problem comes when I use a different GPU for each thread. For example, if the program launches two treads and classifier in thread 1 uses GPU 0 (Caffe::SetDevice(0);) and classifier in thread 2 uses GPU 1(Caffe::SetDevice(1);), prediction starts working but suddenly I get the error cudnn_conv_layer.cu:28] Check failed: status == CUDNN_STATUS_SUCCESS (8 vs. 0) CUDNN_STATUS_EXECUTION_FAILED




I have seen similar errors from people working in multithreaded projects with phyton interface related to CuDNN thread safe issues working with one GPU, but in sigle GPU the program works fine.

My models are trained on single GPU. Should I train separate models in GPU 0 and in GPU 1?. Should I configure classifiers in any special way to use it in multiGPU multithread environement?.

Im using windows branch of caffe with cuda 8.0 and using c++ interface of caffe linked as static library to the project.