CUDNN_STATUS_NOT_INITIALIZED when using Caffe with cuDNN

[Tiferet Gazit]

I am trying to use cuDNN to speed up my Caffe networks.

When I "make runtest" with cuDNN (and also when I run my own Caffe models) I get the following error for many of the cuDNN layers (for example,cudnn_softmax_layer.cpp):

 

Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0)  CUDNN_STATUS_NOT_INITIALIZED

*** Check failure stack trace: ***

    @     0x2b0f3ac86daa  (unknown)

    @     0x2b0f3ac86ce4  (unknown)

    @     0x2b0f3ac866e6  (unknown)

    @     0x2b0f3ac89687  (unknown)

    @     0x2b0f3c62a01e  caffe::CuDNNSoftmaxLayer<>::LayerSetUp()

    @     0x2b0f3c5f9b36  caffe::SoftmaxWithLossLayer<>::LayerSetUp()

    @           0x498be0  caffe::Layer<>::SetUp()

    @           0x498f6f  caffe::GradientChecker<>::CheckGradientExhaustive()

    @           0x597f7f  caffe::SoftmaxWithLossLayerTest_TestGradientUnnormalized_Test<>::TestBody()

    @           0x7cff8d  testing::internal::HandleExceptionsInMethodIfSupported<>()

    @           0x7c7f21  testing::Test::Run()

    @           0x7c8006  testing::TestInfo::Run()

    @           0x7c8147  testing::TestCase::Run()

    @           0x7c849e  testing::internal::UnitTestImpl::RunAllTests()

    @           0x7cfb0d  testing::internal::HandleExceptionsInMethodIfSupported<>()

    @           0x7c757e  testing::UnitTest::Run()

    @           0x44f832  main

    @     0x2b0f3d168ec5  (unknown)

    @           0x455664  (unknown)

    @              (nil)  (unknown)

make: *** [runtest] Aborted (core dumped)

 

I am using a Quadro K2200 card with NVIDIA driver version 340.93 on an Ubuntu 14.04.2 machine.

 

My Caffe Makefile defines the following CUDA architecture, although I have tried changing it in various ways to no avail (with 50 I am unable to build Caffe at all):

CUDA_ARCH := -gencode arch=compute_35,code=sm_35 \

                                -gencode arch=compute_35,code=compute_35

 

I have tried everything I could find online, but haven't been able to solve the problem. I would greatly appreciate some help. Thank you!

[Tiferet Gazit]

Updating the solution I finally figured out: 

Apparently, in addition to the correct CUDA driver, I also had an older driver installed on my computer, and the older one was the one being used in practice. Once I removed the older driver I was able to build Caffe with the CUDA architecture defined in the original example Makefile, and was then able to run all the tests, as well as my own networks.