specifying more than one gpu from python?

[Jeremy Rutman]

i'd like to start nets on more than one gpu . the code below works if I stay on one gpu (n_nets =1,2) but once I ask for another gpu I hit the dreaded 

Check failed: status == CUDNN_STATUS_SUCCESS (8 vs. 0)  CUDNN_STATUS_EXECUTION_FAILED 
*** Check failure stack trace: ***

Aborted

I have tried this on a machine with 4xM60 gpus and another with 2XK80 gpus, same problem each time.

Anyone with a clue?

the code that works if n_nets<3 : 

for i in range(n_nets):
 gpu = i/2
 caffe.set_device(gpu)
 print('device '+str(gpu)+' net # '+str(i))
 deployproto = os.path.join(modelpath,'ResNet-101-deploy.prototxt')
 caffemodel = os.path.join(modelpath,caffemodels[i])
 print('deployproto {} caffemodel {}'.format(deployproto,caffemodel))
 binary_net = caffe.Net(deployproto,caffe.TEST,weights=caffemodel)
 binary_nets.append(binary_net)

[Jeremy Rutman]

splitting the gpu usage into different python files (one for each gpu) doesnt help; each can be run individually but not together.