Terrible results when using GPU but not with CPU

[Sreejith Balakrishnan]

Hi everyone,

Previously I posted  about how my convolution results are looking strange when training  my network and how my network gets stuck at 0.07 accuracy regardless of the input. Please take a look at the attachment.

After around 1 month of troubleshooting, I accidentally found the issue. If I comment out 

caffe.set_device(0)
caffe.set_gpu()
 

then the results are much better. Please find the new convolution result attached. Also, the accuracy increases as I train. 

I also tested a pretrained network with and without "caffe.set_gpu()". There is a big difference in the result I get.

Right now, I am training without caffe.set_gpu. But since I bought a brand new GTX1080 for this purpose, I want to make sure I am fully utilizing it. May I know what is going on and why I am getting this error? (strange thing I noticed is that NVIDIA GPU Activity program still says python.exe is using my GPU)

I am using caffe on windows with python interface. I compiled with CUDNN.

Best regards,

Sreejith