Hello,
I'm new to caffe and I'm having some troubles installing caffe-segnet it on a remote machine. I ran make all, make test, make pycaffe and make distribute without any problems. The remote machine uses PBS (Portable Batch System) in order to schedule jobs and we are supposed to allocate resources through a bash script before sending them to a queue. When I run make runtest normally on the bash shell it throws the following error:
.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>
commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time
Flags from tools/caffe.cpp:
-gpu (Run in GPU mode on given device ID.) type: int32 default: -1
-iterations (The number of iterations to run.) type: int32 default: 50
-model (The model definition protocol buffer text file..) type: string
default: ""
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: ""
-solver (The solver definition protocol buffer text file.) type: string
default: ""
-weights (Optional; the pretrained weights to initialize finetuning. Cannot
be set simultaneously with snapshot.) type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Note: Randomizing tests' orders with a seed of 16424 .
[==========] Running 1412 tests from 222 test cases.
[----------] Global test environment set-up.
[----------] 5 tests from MemoryDataLayerTest/3, where TypeParam = caffe::GPUDevice<double>
[ RUN ] MemoryDataLayerTest/3.TestSetup
E0507 16:43:49.812096 21232 common.cpp:93] Cannot create Cublas handle. Cublas won't be available.
E0507 16:43:49.813638 21232 common.cpp:100] Cannot create Curand generator. Curand won't be available.
[ OK ] MemoryDataLayerTest/3.TestSetup (6 ms)
[ RUN ] MemoryDataLayerTest/3.TestForward
[ OK ] MemoryDataLayerTest/3.TestForward (17 ms)
[ RUN ] MemoryDataLayerTest/3.TestSetBatchSize
[ OK ] MemoryDataLayerTest/3.TestSetBatchSize (2 ms)
[ RUN ] MemoryDataLayerTest/3.AddMatVectorDefaultTransform
[ OK ] MemoryDataLayerTest/3.AddMatVectorDefaultTransform (2 ms)
[ RUN ] MemoryDataLayerTest/3.AddDatumVectorDefaultTransform
[ OK ] MemoryDataLayerTest/3.AddDatumVectorDefaultTransform (2 ms)
[----------] 5 tests from MemoryDataLayerTest/3 (30 ms total)
[----------] 5 tests from AdaGradSolverTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN ] AdaGradSolverTest/2.TestAdaGradLeastSquaresUpdate
E0507 16:43:49.840687 21232 common.cpp:120] Curand not available. Skipping setting the curand seed.
F0507 16:43:49.844883 21232 syncedmem.cpp:57] Check failed: error == cudaSuccess (38 vs. 0) no CUDA-capable device is detected
*** Check failure stack trace: ***
@ 0x7f3181773f8d google::LogMessage::Fail()
@ 0x7f31817794c2 google::LogMessage::SendToLog()
@ 0x7f31817760e0 google::LogMessage::Flush()
@ 0x7f31817763cd google::LogMessageFatal::~LogMessageFatal()
@ 0x7f317e05eba8 caffe::SyncedMemory::to_gpu()
@ 0x7f317e05e689 caffe::SyncedMemory::gpu_data()
@ 0x7f317df6811d caffe::Blob<>::gpu_data()
@ 0x7f317e071514 caffe::InnerProductLayer<>::Forward_gpu()
@ 0x7f317e05dfba caffe::Net<>::ForwardFromTo()
@ 0x7f317e05e306 caffe::Net<>::ForwardPrefilled()
@ 0x571745 caffe::GradientBasedSolverTest<>::ComputeLeastSquaresUpdate()
@ 0x572937 caffe::GradientBasedSolverTest<>::TestLeastSquaresUpdate()
@ 0x84718d testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x839081 testing::Test::Run()
@ 0x839167 testing::TestInfo::Run()
@ 0x8392a7 testing::TestCase::Run()
@ 0x83e1af testing::internal::UnitTestImpl::RunAllTests()
@ 0x846d3d testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8386aa testing::UnitTest::Run()
@ 0x4d3bdf main
@ 0x7f317d407d1d __libc_start_main
@ 0x4d3939 (unknown)
make: *** [runtest] Aborted
I figured I'd have to send the "make runtest" through PBS by requesting usage of the GPU. However, when I send the job to the queue it fails with the following error:
make: *** No rule to make target `/usr/include/leveldb/db.h', needed by `.build_release/cuda/src/caffe/test/test_im2col_kernel.o'. Stop.
What could be causing this problem on my installation? Am I missing something on my Makefile? I have managed to install caffe-segnet using the CPU-only flag using the attached Makefile.config by uncommenting the "CPU_ONLY := 1" line, however training takes way too long without the support of a GPU.
Thanks in advance.