set_mode_gpu() ok, set_device() crashes....

[Jeremy Rutman]

I am not sure what's going on with my install - I was wanting to use the cmake so I tried

cmake -DUSE_CUDNN=ON -DBUILD_python=ON -DBUILD_python_layer=ON ..

which gives a summary including the following that seems to indicate gpu support is on (since CPU_ONLY is OFF)

  

-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0-rc3
--   Git               :   42cd785
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   1
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   ALLOW_LMDB_NOLOCK :   OFF

and everything seems to go to completion , although I do not see any gpu-specific tests - anyone know what the 'google test filter = -*GPU* BELOW means ? yes gpu tests? no gpu tests?

Linking CXX executable ../../../test/test.testbin
[100%] Built target test.testbin
Scanning dependencies of target runtest
libdc1394 error: Failed to initialize libdc1394
Note: Google Test filter = -*GPUFrom the cpu_only:off ABOVE AND  following it seems like I can set_mode_gpu which means i have correctly compiled for gpu , but then the set_device failure suggests the opposite - any advice?

From the cpu_only:off above and the  following it seems like I can set_mode_gpu which means i have correctly compiled for gpu , but then the set_device failure suggests the opposite - any advice?

This crashes out of python rather ungracefully btw.

In [1]: import caffe
libdc1394 error: Failed to initialize libdc1394

In [2]: caffe.set_mode_gpu()

In [3]: caffe.set_device(0)
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0726 13:44:19.262065  1092 common.cpp:66] Cannot use GPU in CPU-only Caffe: check mode.

[Jeremy Rutman]

in case it helps the commands after cmake were 

make all -j"$(nproc)"
make install
make runtest

[Jeremy Rutman]

The build in question is in a docker container.

Further invesitgation found the following output during build - 

CMake Warning at cmake/Dependencies.cmake:63 (message):
  -- CUDA is not detected by cmake.  Building without it...

so this is a similar issue to the one here.  Looking at the path in the container

root@3f2a6ff54c23:/workspace# ls /usr/local/cuda
LICENSE  README  doc  lib64  targets  version.txt

I see its not the same as on the original machine

root@brainik80a:/home/jeremy# ls /usr/local/cuda
bin  doc  extras  include  jre  lib  lib64  libnsight  libnvvp  nvvm  pkgconfig  samples  share  src  tools  version.txt

so I guess the original docker base 

FROM nvidia/cuda:7.5-cudnn5-runtime

is not installing the full cuda libs for some reason.  Anyone have a clue?

[Felix Abecassis]

For further reference, this question has been answered here:
https://github.com/NVIDIA/nvidia-docker/issues/153