CaffeIssue: Makefile:532: recipe for target 'runtest' failed make: *** [runtest] Abort(core dumped)

[Prodromos Alampritis]

I have been following this tutorial here: https://chunml.github.io/ChunML.github.io/project/Installing-Caffe-Ubuntu/

I have installed CUDA 8.0.61 and i have the following issue when running the following command:

make all & make test && make runtest && make pycaffe

it prints this message:(see last lines)

[1] 2492
make: Nothing to be done for 'all'.
make: Nothing to be done for 'test'.
[1]+  Done                    make all
.build_release/tools/caffe
caffe: command line brew
usage: caffe <command> <args>

commands:
  train           train or finetune a model
  test            score a model
  device_query    show GPU diagnostic information
  time            benchmark model execution time

  Flags from tools/caffe.cpp:
    -gpu (Optional; run in GPU mode on given device IDs separated by ','.Use
      '-gpu all' to run on all available GPUs. The effective training batch
      size is multiplied by the number of devices.) type: string default: ""
    -iterations (The number of iterations to run.) type: int32 default: 50
    -level (Optional; network level.) type: int32 default: 0
    -model (The model definition protocol buffer text file.) type: string
      default: ""
    -phase (Optional; network phase (TRAIN or TEST). Only used for 'time'.)
      type: string default: ""
    -sighup_effect (Optional; action to take when a SIGHUP signal is received:
      snapshot, stop or none.) type: string default: "snapshot"
    -sigint_effect (Optional; action to take when a SIGINT signal is received:
      snapshot, stop or none.) type: string default: "stop"
    -snapshot (Optional; the snapshot solver state to resume training.)
      type: string default: ""
    -solver (The solver definition protocol buffer text file.) type: string
      default: ""
    -stage (Optional; network stages (not to be confused with phase), separated
      by ','.) type: string default: ""
    -weights (Optional; the pretrained weights to initialize finetuning,
      separated by ','. Cannot be set simultaneously with snapshot.)
      type: string default: ""
.build_release/test/test_all.testbin 0 --gtest_shuffle 
Cuda number of devices: 0
Setting to use device 0
Current device id: 0
Current device name: 
Note: Randomizing tests' orders with a seed of 54883 .
[==========] Running 2139 tests from 285 test cases.
[----------] Global test environment set-up.
[----------] 7 tests from TileLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] TileLayerTest/0.TestTrivialGradient
E0208 12:28:10.838369  3278 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.
E0208 12:28:10.998697  3278 common.cpp:121] Cannot create Curand generator. Curand won't be available.
E0208 12:28:10.998919  3278 common.cpp:141] Curand not available. Skipping setting the curand seed.
[       OK ] TileLayerTest/0.TestTrivialGradient (380 ms)
[ RUN      ] TileLayerTest/0.TestSetup
[       OK ] TileLayerTest/0.TestSetup (0 ms)
[ RUN      ] TileLayerTest/0.TestTrivialSetup
[       OK ] TileLayerTest/0.TestTrivialSetup (0 ms)
[ RUN      ] TileLayerTest/0.TestForwardChannels
[       OK ] TileLayerTest/0.TestForwardChannels (0 ms)
[ RUN      ] TileLayerTest/0.TestGradientChannels
[       OK ] TileLayerTest/0.TestGradientChannels (176 ms)
[ RUN      ] TileLayerTest/0.TestGradientNum
[       OK ] TileLayerTest/0.TestGradientNum (172 ms)
[ RUN      ] TileLayerTest/0.TestForwardNum
[       OK ] TileLayerTest/0.TestForwardNum (0 ms)
[----------] 7 tests from TileLayerTest/0 (728 ms total)

[----------] 5 tests from DeconvolutionLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] DeconvolutionLayerTest/0.TestGradient3D
[       OK ] DeconvolutionLayerTest/0.TestGradient3D (136 ms)
[ RUN      ] DeconvolutionLayerTest/0.TestSimpleDeconvolution
[       OK ] DeconvolutionLayerTest/0.TestSimpleDeconvolution (0 ms)
[ RUN      ] DeconvolutionLayerTest/0.TestGradient
[       OK ] DeconvolutionLayerTest/0.TestGradient (483 ms)
[ RUN      ] DeconvolutionLayerTest/0.TestNDAgainst2D
[       OK ] DeconvolutionLayerTest/0.TestNDAgainst2D (571 ms)
[ RUN      ] DeconvolutionLayerTest/0.TestSetup
[       OK ] DeconvolutionLayerTest/0.TestSetup (0 ms)
[----------] 5 tests from DeconvolutionLayerTest/0 (1190 ms total)

[----------] 15 tests from ConvolutionLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] ConvolutionLayerTest/0.TestSetup
[       OK ] ConvolutionLayerTest/0.TestSetup (0 ms)
[ RUN      ] ConvolutionLayerTest/0.TestGradientGroup
[       OK ] ConvolutionLayerTest/0.TestGradientGroup (17 ms)
[ RUN      ] ConvolutionLayerTest/0.TestSimpleConvolution
[       OK ] ConvolutionLayerTest/0.TestSimpleConvolution (0 ms)
[ RUN      ] ConvolutionLayerTest/0.TestSimple3DConvolution
[       OK ] ConvolutionLayerTest/0.TestSimple3DConvolution (1 ms)
[ RUN      ] ConvolutionLayerTest/0.Test1x1Convolution
[       OK ] ConvolutionLayerTest/0.Test1x1Convolution (1 ms)
[ RUN      ] ConvolutionLayerTest/0.Test1x1Gradient
[       OK ] ConvolutionLayerTest/0.Test1x1Gradient (461 ms)
[ RUN      ] ConvolutionLayerTest/0.TestGradient3D
[       OK ] ConvolutionLayerTest/0.TestGradient3D (400 ms)
[ RUN      ] ConvolutionLayerTest/0.TestSimpleConvolutionGroup
[       OK ] ConvolutionLayerTest/0.TestSimpleConvolutionGroup (0 ms)
[ RUN      ] ConvolutionLayerTest/0.TestDilatedConvolution
[       OK ] ConvolutionLayerTest/0.TestDilatedConvolution (1 ms)
[ RUN      ] ConvolutionLayerTest/0.TestGradient
[       OK ] ConvolutionLayerTest/0.TestGradient (51 ms)
[ RUN      ] ConvolutionLayerTest/0.TestDilated3DConvolution
[       OK ] ConvolutionLayerTest/0.TestDilated3DConvolution (7 ms)
[ RUN      ] ConvolutionLayerTest/0.Test0DConvolution
[       OK ] ConvolutionLayerTest/0.Test0DConvolution (0 ms)
[ RUN      ] ConvolutionLayerTest/0.TestDilatedGradient
[       OK ] ConvolutionLayerTest/0.TestDilatedGradient (13 ms)
[ RUN      ] ConvolutionLayerTest/0.TestNDAgainst2D
[       OK ] ConvolutionLayerTest/0.TestNDAgainst2D (225 ms)
[ RUN      ] ConvolutionLayerTest/0.TestSobelConvolution
[       OK ] ConvolutionLayerTest/0.TestSobelConvolution (0 ms)
[----------] 15 tests from ConvolutionLayerTest/0 (1177 ms total)

[----------] 5 tests from BenchmarkTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] BenchmarkTest/0.TestTimerStop
[       OK ] BenchmarkTest/0.TestTimerStop (0 ms)
[ RUN      ] BenchmarkTest/0.TestTimerMilliSeconds
[       OK ] BenchmarkTest/0.TestTimerMilliSeconds (300 ms)
[ RUN      ] BenchmarkTest/0.TestTimerSeconds
[       OK ] BenchmarkTest/0.TestTimerSeconds (301 ms)
[ RUN      ] BenchmarkTest/0.TestTimerStart
[       OK ] BenchmarkTest/0.TestTimerStart (0 ms)
[ RUN      ] BenchmarkTest/0.TestTimerConstructor
[       OK ] BenchmarkTest/0.TestTimerConstructor (0 ms)
[----------] 5 tests from BenchmarkTest/0 (601 ms total)

[----------] 8 tests from AdamSolverTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum
[       OK ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccum (3 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithWeightDecay
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithWeightDecay (13 ms)
[ RUN      ] AdamSolverTest/0.TestSnapshot
[       OK ] AdamSolverTest/0.TestSnapshot (9 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverythingShare
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverythingShare (73 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverything
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdateWithEverything (41 ms)
[ RUN      ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare
[       OK ] AdamSolverTest/0.TestLeastSquaresUpdateWithEverythingAccumShare (1 ms)
[ RUN      ] AdamSolverTest/0.TestAdamLeastSquaresUpdate
[       OK ] AdamSolverTest/0.TestAdamLeastSquaresUpdate (8 ms)
[ RUN      ] AdamSolverTest/0.TestSnapshotShare
[       OK ] AdamSolverTest/0.TestSnapshotShare (10 ms)
[----------] 8 tests from AdamSolverTest/0 (158 ms total)

[----------] 15 tests from ConvolutionLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] ConvolutionLayerTest/2.TestGradientGroup
F0208 12:28:14.531548  3278 syncedmem.hpp:22] Check failed: error == cudaSuccess (30 vs. 0)  unknown error
*** Check failure stack trace: ***
    @     0x7f7e555cb5cd  google::LogMessage::Fail()
    @     0x7f7e555cd433  google::LogMessage::SendToLog()
    @     0x7f7e555cb15b  google::LogMessage::Flush()
    @     0x7f7e555cde1e  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f7e517414b1  caffe::SyncedMemory::mutable_cpu_data()
    @     0x7f7e517108b0  caffe::Blob<>::Reshape()
    @     0x7f7e51710d2a  caffe::Blob<>::Reshape()
    @     0x7f7e51710ddc  caffe::Blob<>::Blob()
    @           0x495395  caffe::ConvolutionLayerTest<>::ConvolutionLayerTest()
    @           0x49599b  testing::internal::TestFactoryImpl<>::CreateTest()
    @           0x93eb83  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @           0x9380e3  testing::TestInfo::Run()
    @           0x938245  testing::TestCase::Run()
    @           0x93951f  testing::internal::UnitTestImpl::RunAllTests()
    @           0x939843  testing::UnitTest::Run()
    @           0x46f3ad  main
    @     0x7f7e50a69830  __libc_start_main
    @           0x476f99  _start
    @              (nil)  (unknown)
Makefile:532: recipe for target 'runtest' failed
make: *** [runtest] Aborted (core dumped)

What am i missing here?

[Przemek D]

Have you installed the CUDA driver? What is the output of nvidia-smi on your machine?

[Prodromos Alampritis]

when you say CUDA driver what do you mean? which one?
i just run the command: nvidia-smi on my machine and i get this message here:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Can you tell me which driver to install and what is/how to get nvidia-smi? Are there any requirements i need to check beforehand?

[Prodromos Alampritis]

Also just run this command too: sudo ubuntu-drivers devices
== cpu-microcode.py ==
driver   : intel-microcode - distro free

== /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0 ==
modalias : pci:v000010DEd00000FCDsv000017AAsd00003801bc03sc00i00
model    : GK107M [GeForce GT 755M]
vendor   : NVIDIA Corporation
driver   : xserver-xorg-video-nouveau - distro free builtin
driver   : nvidia-340 - distro non-free
driver   : nvidia-390 - third-party non-free recommended
driver   : nvidia-387 - third-party non-free
driver   : nvidia-384 - third-party non-free

if that helps at all

[Przemek D]

The message you got from nvidia-smi means that the driver is either not installed or inactive - the program itself can run (it comes with the CUDA runtime as far as I know), but it doesn't hear back from the driver.

Is it possible that your driver is installed, but the x-server is still using ubuntu's built-in nouveau? You should look into this and make sure the OS loads the Nvidia driver and not the nouveau one.

[Prodromos Alampritis]

Solved. have blacklisted nouveau and through Software & updates -> Additional drivers i have installed the nvidia driver and rebooted.

[Zhiqi Yang]

hi przeemek,

I've met the same problem above, but my nvidia-smi works ok, can directly show the GPU processes.

What should I do now to solve the problem?

[ RUN      ] RNNLayerTest/3.TestGradientNonZeroCont

E0309 16:15:16.977672 182769 common.cpp:114] Cannot create Cublas handle. Cublas won't be available.

E0309 16:15:17.463361 182769 common.cpp:121] Cannot create Curand generator. Curand won't be available.

F0309 16:15:17.919613 182769 syncedmem.hpp:22] Check failed: error == cudaSuccess (2 vs. 0)  out of memory

*** Check failure stack trace: ***

    @     0x7f56daba0778  (unknown)

    @     0x7f56daba06b2  (unknown)

    @     0x7f56daba00b4  (unknown)

    @     0x7f56daba3055  (unknown)

    @     0x7f56d0051103  caffe::SyncedMemory::mutable_cpu_data()

    @     0x7f56d0021348  caffe::Blob<>::Reshape()

    @     0x7f56d00217aa  caffe::Blob<>::Reshape()

    @           0x4911ef  caffe::RNNLayerTest<>::ReshapeBlobs()

    @           0x49161b  caffe::RNNLayerTest<>::RNNLayerTest()

    @           0x4919ab  testing::internal::TestFactoryImpl<>::CreateTest()

    @           0x8fa9e3  testing::internal::HandleExceptionsInMethodIfSupported<>()

    @           0x8f3cf3  testing::TestInfo::Run()

    @           0x8f3e55  testing::TestCase::Run()

    @           0x8f4138  testing::internal::UnitTestImpl::RunAllTests()

    @           0x8f4413  testing::UnitTest::Run()

    @           0x46f86f  main

    @     0x7f56cf3b8b45  (unknown)

    @           0x477559  (unknown)

    @              (nil)  (unknown)

Makefile:532: recipe for target 'runtest' failed

make: *** [runtest] Aborted

在 2018年2月8日星期四 UTC+8下午10:25:55，Przemek D写道：

[Przemek D]

Your problem is different, much simpler. Just read the error log carefully - your GPU runs out of memory.