CUDNN_STATUS_NOT_INITIALIZED

[spssachd...@gmail.com]

Hi, 

Thanks for making a flexible and define by run lib for neural net. 

I am trying to install chainer to work along with cuda and cudnn. 

My system is CentOS 6.5

I am working in a lab, where I don't have a root access.  

I already have cuda installed (there are other lab members who can use gpu)

I installed chainer using:

pip2.7 install chainer --user

But, I get an error saying CUDNN_STATUS_NOT_INITIALIZED. I would request you to please help me on this. If there is another info that you need please feel free to ask. 

Error Message:

 Traceback (most recent call last):

  File "mnist.py", line 248, in <module>

    opt.update(model, z, x)

  File "/home/sidak/.local/lib/python2.7/site-packages/chainer/optimizer.py", line 376, in update

    loss = lossfun(*args, **kwds)

  File "mnist.py", line 185, in __call__

    self.X = self.sample(z)

  File "mnist.py", line 176, in sample

    h = F.relu(self.lin0(z))

  File "/home/sidak/.local/lib/python2.7/site-packages/chainer/functions/activation/relu.py", line 91, in relu

    return ReLU(use_cudnn)(x)

  File "/home/sidak/.local/lib/python2.7/site-packages/chainer/function.py", line 123, in __call__

    outputs = self.forward(in_data)

  File "/home/sidak/.local/lib/python2.7/site-packages/chainer/function.py", line 227, in forward

    return self.forward_gpu(inputs)

  File "/home/sidak/.local/lib/python2.7/site-packages/chainer/functions/activation/relu.py", line 46, in forward_gpu

    handle = cudnn.get_handle()

  File "/home/sidak/.local/lib/python2.7/site-packages/cupy/cudnn.py", line 20, in get_handle

    handle = cudnn.create()

  File "cupy/cuda/cudnn.pyx", line 221, in cupy.cuda.cudnn.create (cupy/cuda/cudnn.cpp:1619)

  File "cupy/cuda/cudnn.pyx", line 224, in cupy.cuda.cudnn.create (cupy/cuda/cudnn.cpp:1564)

  File "cupy/cuda/cudnn.pyx", line 206, in cupy.cuda.cudnn.check_status (cupy/cuda/cudnn.cpp:1363)

cupy.cuda.cudnn.CuDNNError: CUDNN_STATUS_NOT_INITIALIZED: CUDNN_STATUS_NOT_INITIALIZED

I followed another post on the group, which said that I should use cudnnv2 instead of v3. I have tried with that and also tried different environment variable settings. https://groups.google.com/forum/#!topic/chainer/QNWff7EYElw

The environment variables that I have tried:

( I have cuda at both these locations. Also tried separately)

 

export PATH=/usr/local/cuda/bin:$PATH

export CUDA_PATH=/usr/local/cuda:$CUDA_PATH

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

export PATH=/opt/cuda-6.5/bin:$PATH

export CUDA_PATH=/opt/cuda-6.5:$CUDA_PATH

export LD_LIBRARY_PATH=/opt/cuda-6.5/lib64:$LD_LIBRARY_PATH

(Env variables with cudnn v2)

export CPATH=~/Downloads/cudnn6.5:$CPATH

export LIBRARY_PATH=~/Downloads/cudnn6.5:$LIBRARY_PATH

export LD_LIBRARY_PATH=~/Downloads/cudnn6.5:$LD_LIBRARY_PATH

(Env variables with cudnn v3)

export CPATH=~/Downloads/cuda/include:$CPATH

export LIBRARY_PATH=~/Downloads/cuda/lib64:$LIBRARY_PATH

export LD_LIBRARY_PATH=~/Downloads/cuda/lib64:$LD_LIBRARY_PATH

(Also in the middle, I got a different error, saying cuda not enabled. I think that was due to improper reinstallation without setting the no-cache-dir flag)

Thank you so much!

Regards,

Sidak

[Yuya Unno]

Hi,

I'm not familiar with the details of CUDA errors, but I found this message when I used old NVIDIA driver with newer cuDNN in Amazon EC2.

Please check these points:

- Where did you (or your admin) install CUDA? /usr/local/cuda? Show me result of `which nvcc`.

- Please check nvidia driver version. Show me result of nvidia-smi command.

- Where does cudnn.h exist? Please check /usr/local/cuda/include

- Show me `pip` result with `pip install chainer -vvvv --no-cache-dir`. It shows detailed install logs.

Use `--no-cache-dir` option when use `pip` command like `pip install chainer --user --no-cache-dir`. Without this option `pip` command uses cached egg files already built.

--
You received this message because you are subscribed to the Google Groups "Chainer User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chainer+u...@googlegroups.com.
To post to this group, send email to cha...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/chainer/ca85f2a7-d3fe-4139-8eeb-70c80aec0751%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[spssachd...@gmail.com]

Hi, 

Thank you for your reply.

Result of 'which nvcc':

/opt/cuda-6.5/bin/nvcc

Result of nvidia smi

http://pastebin.com/NdwHY3Ux

Cudnn.h

It exits in ~/Downloads/cudnn6.5/  , but not in /usr/local/cuda/include

Since, I don't have root access, I have set the environment variables to point to this. (You can find the environment variables I tried in my first mail)

pip install chainer -vvvv --no-cache-dir  Result

http://pastebin.com/5SFet6h2

Thanks,

Sidak

[Yuya Unno]

Thanks.

- Install log looks good. Chainer was successfully built and linked.

- I think driver 340.65 is old. CUDA 6.5 includes nvidia driver 343.19.

So, I recommend you to update the driver.

If you can't, i recommend you to disable cudnn.

To view this discussion on the web visit https://groups.google.com/d/msgid/chainer/8222f411-efa2-4473-a949-a551ac340a83%40googlegroups.com.