Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

[mahima....@gmail.com]

Hi

With the latest auto - updates, I have been having problems with using caffe (standalone and with digits)

One of the error messages is Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM

The result of Nvidia-smi is
Fri Jun  3 20:22:40 2016      
+-----------------------------

-------------------------+                      
| NVIDIA-SMI 352.93     Driver Version: 352.93         |                      
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:05:00.0      On |                  N/A |
| 22%   59C    P8    17W / 250W |    302MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 0000:06:00.0     Off |                  N/A |
| 22%   57C    P8    19W / 250W |     23MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 0000:09:00.0     Off |                  N/A |
| 22%   52C    P8    17W / 250W |     23MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 0000:0A:00.0     Off |                  N/A |
| 22%   46C    P8    16W / 250W |    135MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                              
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1600    G   /usr/bin/X                                     156MiB |
|    0      2640    G   compiz                                         117MiB |
|    0      4003    G   unity-control-center                             2MiB |
|    3      3307    C   /usr/bin/python                                110MiB |
+-----------------------------------------------------------------------------+

nvcc - V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17


My original problem started with an autoupdate, which resulted in the following installation.
2016-05-29 14:06:17 install libcudnn5:amd64 <none> 5.0.5-1+cuda8.0
2016-05-29 14:06:20 install cuda-ld-conf-7-5:amd64 <none> 1.0.0-3
2016-05-29 14:06:21 install libnccl1:amd64 <none> 1.2.1-1+cuda7.5

These are messages I got, when I tried running examples in caffe directory.

E0529 17:27:19.065212  3520 common.cpp:121] Cannot create cuDNN handle. cuDNN won't be available.
F0529 17:27:19.065306  3520 db_lmdb.hpp:23] Check failed: mdb_status == 0 (2 vs. 0) No such file or directory
*** Check failure stack trace: ***
    @     0x7f73aae87daa  (unknown)
    @     0x7f73aae87ce4  (unknown)
    @     0x7f73aae876e6  (unknown)
    @     0x7f73aae8a687  (unknown)
    @     0x7f73ab30dbf8  (unknown)
    @     0x7f73ab2c42e4  (unknown)
    @     0x7f73ab1b8cdf  (unknown)
    @     0x7f73a955aa4a  (unknown)
    @     0x7f73a9339182  start_thread
    @     0x7f73a9a6347d  (unknown)
    @              (nil)  (unknown)

In digits.

ERROR: Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

Based on some threads in the digits and caffe groups, one thing led to another, until I had to reinstall the drivers, cuda and caffe

The current driver version is NVIDIA binary driver 352.93

PLEASE HELP. I AM UNABLE TO SEE WHERE THE PROBLEM IS

Regards

Mahima

[15535...@qq.com]

Hi 

I have the same problem. Did you solve it?

Thank you!

[Xinge]

Did you solve it? My GPU is Titan X, and i get the same error.

在 2016年9月18日星期日 UTC+8下午4:19:08，15535...@qq.com写道：

[Jun Muzi]

Could you provides more error information? 

If you hit

cudnn_ndconv_layer.cu:43] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0)  CUDNN_STATUS_BAD_PARAM

This is due to the fixed setting in cudnn lib, you must set the length with 16.

在 2016年9月22日星期四 UTC+8下午9:46:13，Xinge写道：

[Thorsten Laude]

Hi,

Could you elaborate what you mean with the following? Which length needs to be set and how?

This is due to the fixed setting in cudnn lib, you must set the length with 16.

Thanks,

Thorsten 

[Thorsten Laude]

Just in case anybody experiences the same error:

I worked around the cudnn bug by configuring caffe with cmake "USE_CUDNN=OFF ..". Then everything runs smoothly.