make runtest fail

[Jeremy Rutman]

Hi
I am trying to install caffe on ubuntu16.04 with cuda9.1 and cudnn7.  After downloading source I changed makefile.config to use opencv3 but not cudnn.
When I try

make all -j8
make test
make runtest

I get some failed tests namely:

[  FAILED  ] 2 tests, listed below:
[  FAILED  ] BatchReindexLayerTest/2.TestGradient, where TypeParam = caffe::GPUDevice<float>
[  FAILED  ] BatchReindexLayerTest/3.TestGradient, where TypeParam = caffe::GPUDevice<double>

Can anyone give some feedback?

[Jeremy Rutman]

when compiling with cudnn I get more or less the same error

[==========] 2123 tests from 281 test cases ran. (381956 ms total)
[  PASSED  ] 2122 tests.
[  FAILED  ] 1 test, listed below:

[  FAILED  ] BatchReindexLayerTest/3.TestGradient, where TypeParam = caffe::GPUDevice<double>

during the tests some more detailed info is printed:

The difference between computed_gradient and estimated_gradient is 0.6739168052989799, which exceeds threshold_ * scale, where
computed_gradient evaluates to 0.6739168052989799,
estimated_gradient evaluates to 0, and
threshold_ * scale evaluates to 0.01.
debug: (top_id, top_data_id, blob_id, feat_id)=0,143,0,119; feat = 0.6739168052989799; objective+ = -0.037382097476710979; objective- = -0.037382097476710979
[  FAILED  ] BatchReindexLayerTest/3.TestGradient, where TypeParam = caffe::GPUDevice<double> (1755 ms)

[Rolf Reintjes]

I have the same problem. See my post here:
https://groups.google.com/forum/#!topic/caffe-users/52xp37IdLkU

[Przemek D]

We're looking into it - see issue 6164. So far we know that it only happens on CUDA 9.1, regardless if you use cuDNN or not (I tested this myself), regardless of your OS (verified on Ubuntu 16.04 and CentOS 7.4) and might be caused by some optimization options in nvcc.