error during make runtest - CUDA driver version is insufficient for CUDA runtime version

[Anton Ovinnikov]

Good day.

I'm building caffe on ubuntu 14.04(x86-64). Walked through installation notes on the caffe site, but stuck into this. The issue is that I'm using latest driver and cuda toolkit, both are not from vanilla repositories(driver from xorg-edgers ppa and toolkit from official nvidia site).

This is the error I receive:

CXX/LD -o .build_release/test/test_all.testbin src/caffe/test/test_caffe_main.cpp
.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>

commands:
  train           train or finetune a model
  test            score a model
  device_query    show GPU diagnostic information
  time            benchmark model execution time

  Flags from tools/caffe.cpp:
    -gpu (Run in GPU mode on given device ID.) type: int32 default: -1
    -iterations (The number of iterations to run.) type: int32 default: 50
    -model (The model definition protocol buffer text file..) type: string
      default: ""
    -snapshot (Optional; the snapshot solver state to resume training.)
      type: string default: ""
    -solver (The solver definition protocol buffer text file.) type: string
      default: ""
    -weights (Optional; the pretrained weights to initialize finetuning. Cannot
      be set simultaneously with snapshot.) type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle 
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Note: Randomizing tests' orders with a seed of 29074 .
[==========] Running 1356 tests from 214 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from FilterLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] FilterLayerTest/1.TestForward
E0712 02:41:24.558691 20345 common.cpp:93] Cannot create Cublas handle. Cublas won't be available.
E0712 02:41:24.568986 20345 common.cpp:100] Cannot create Curand generator. Curand won't be available.
E0712 02:41:24.569087 20345 common.cpp:120] Curand not available. Skipping setting the curand seed.
[       OK ] FilterLayerTest/1.TestForward (21 ms)
[ RUN      ] FilterLayerTest/1.TestReshape
[       OK ] FilterLayerTest/1.TestReshape (0 ms)
[ RUN      ] FilterLayerTest/1.TestGradient
[       OK ] FilterLayerTest/1.TestGradient (291 ms)
[----------] 3 tests from FilterLayerTest/1 (312 ms total)

[----------] 1 test from SolverTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] SolverTest/0.TestInitTrainTestNets
[       OK ] SolverTest/0.TestInitTrainTestNets (4 ms)
[----------] 1 test from SolverTest/0 (4 ms total)

[----------] 11 tests from PoolingLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN      ] PoolingLayerTest/3.TestForwardMaxPadded
F0712 02:41:24.864735 20345 syncedmem.cpp:57] Check failed: error == cudaSuccess (35 vs. 0)  CUDA driver version is insufficient for CUDA runtime version
*** Check failure stack trace: ***
    @     0x2ae03ce67daa  (unknown)
    @     0x2ae03ce67ce4  (unknown)
    @     0x2ae03ce676e6  (unknown)
    @     0x2ae03ce6a687  (unknown)
    @     0x2ae03e76f36c  caffe::SyncedMemory::gpu_data()
    @     0x2ae03e7c8de2  caffe::Blob<>::gpu_data()
    @     0x2ae03e8808a5  caffe::PoolingLayer<>::Forward_gpu()
    @           0x4530a8  caffe::Layer<>::Forward()
    @           0x5fedad  caffe::PoolingLayerTest_TestForwardMaxPadded_Test<>::TestBody()
    @           0x758ac3  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x74f707  testing::Test::Run()
    @           0x74f7ae  testing::TestInfo::Run()
    @           0x74f8b5  testing::TestCase::Run()
    @           0x752bf8  testing::internal::UnitTestImpl::RunAllTests()
    @           0x752e87  testing::UnitTest::Run()
    @           0x44a16a  main
    @     0x2ae03f47bec5  (unknown)
    @           0x44fc59  (unknown)
    @              (nil)  (unknown)
make: *** [runtest] Aborted (core dumped)

Here some diagnostic printouts which I can provide:

$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Feb_16_22:59:02_CST_2015
Cuda compilation tools, release 7.0, V7.0.27

$ nvidia-settings -q NvidiaDriverVersion

Attribute 'NvidiaDriverVersion' (toshakins-Aspire-5750G:0.0): 352.21
Attribute 'NvidiaDriverVersion' (toshakins-Aspire-5750G:0[gpu:0]): 352.21

$  lspci | grep -i nvidia

01:00.0 VGA compatible controller: NVIDIA Corporation GF108M [GeForce GT 630M] (rev a1)

The output of lspci is pretty interesting. Because actually I'm using GT540M.

[Artem Molchanov]

Hi, have you happened to solve the problem ? I have a similar error for cuda-7.5 on Ubuntu 14.04. The error appears when I do make runtest

[Artem Molchanov]

Ok, here is what helped me:

sudo apt-get install cuda-drivers

[Ahmed E.A Abdalla]

Did you manage to fix this issue?