Is it possible to run Cuda 8.0 + CuDNN 4.0 in caffe? (error == cudaSuccess (8 vs. 0))

[john1...@gmail.com]

Hello all, I am running a source code caffe-based which does not support cuDNN 5.0. It only can compile in cuDNN 4.0.  

However, I am using TiTanX pascal, hence I installed Cuda 8.0 before. Based on the requirement of the source code, I download and used cuDNN 4.0.7. I compile the caffe successfully based on the Makefile.config

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \
 -gencode arch=compute_20,code=sm_21 \
 -gencode arch=compute_30,code=sm_30 \
 -gencode arch=compute_35,code=sm_35 \
 -gencode arch=compute_50,code=sm_50 \
 -gencode arch=compute_50,code=compute_50\
 -gencode arch=compute_53,code=compute_53

However, when I run the training part, I got an error as follows

I0209 12:56:54.263430 11389 solver.cpp:281] Learning Rate Policy: step
F0209 12:56:54.281630 11389 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0)  invalid device function

How can I fix the error? It looks that the issue is from cuda and cuDNN.

 

[john1...@gmail.com]

It is very funny that different way to build caffe can solve the issue. 

First way: 

mkdir build
cd build
cmake ..
make all -j8 && make pycaffe

Second way:

make all -j8
make pycaffe -j8

In a first way, I got the error as above. While the second way does not have the error and I can run training phase. I did not figure out what is happen. Which way is recommended to build caffe?