Hi everyone,
Previously I posted about how my convolution results are looking strange when training my network and how my network gets stuck at 0.07 accuracy regardless of the input. Please take a look at the attachment.
After around 1 month of troubleshooting, I accidentally found the issue. If I comment out
caffe.set_device(0)
caffe.set_gpu()
then the results are much better. Please find the new convolution result attached. Also, the accuracy increases as I train.
I also tested a pretrained network with and without "caffe.set_gpu()". There is a big difference in the result I get.
Right now, I am training without caffe.set_gpu. But since I bought a brand new GTX1080 for this purpose, I want to make sure I am fully utilizing it. May I know what is going on and why I am getting this error? (strange thing I noticed is that NVIDIA GPU Activity program still says python.exe is using my GPU)
I am using caffe on windows with python interface. I compiled with CUDNN.
Best regards,
Sreejith