Could not initialize pygpu error with CUDA 9.0 and latest Theano

[Michael Klachko]

I have CUDA 9.0 and CuDNN 7.0.5 on my Ubuntu 16.04, and Tensorflow works fine. In order to install theano, I first installed miniconda, then ran "conda install theano pygpu" and it seemed to have installed fine.

However, here's what I get:

$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
Using cuDNN version 7005 on context None
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 227, in <module>
    use(config.device)
  File "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 214, in use
    init_dev(device, preallocate=preallocate)
  File "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 159, in init_dev
    pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
  File "pygpu/blas.pyx", line 149, in pygpu.blas.gemm
  File "pygpu/blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm
pygpu.gpuarray.GpuArrayException: (b'cuLinkAddData: CUDA_ERROR_UNKNOWN: unknown error', 3)

 

Here's the packages I have installed in this environment:

$ conda list
# packages in environment at /home/michael/miniconda2/envs/las:
#
# Name                    Version                   Build  Channel
binutils_impl_linux-64    2.28.1               had2808c_3
binutils_linux-64         7.2.0                        26
ca-certificates           2018.03.07                    0
certifi                   2018.4.16                py36_0
gcc_impl_linux-64         7.2.0                habb00fd_3
gcc_linux-64              7.2.0                        26
gxx_impl_linux-64         7.2.0                hdf63c60_3
gxx_linux-64              7.2.0                        26
intel-openmp              2018.0.0                      8
libedit                   3.1                  heed3624_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 7.2.0                hdf63c60_3
libgfortran-ng            7.2.0                hdf63c60_3
libgpuarray               0.7.5                h14c3975_0
libstdcxx-ng              7.2.0                hdf63c60_3
mako                      1.0.7            py36h0727276_0
markupsafe                1.0              py36hd9260cd_1
mkl                       2018.0.2                      1
mkl-service               1.1.2            py36h17a0993_4
mkl_fft                   1.0.1            py36h3010b51_0
mkl_random                1.0.1            py36h629b387_0
ncurses                   6.0                  h9df7e31_2
nose                      1.3.7            py36hcdf7029_2
numpy                     1.14.2           py36hdbf6ddf_1
openssl                   1.0.2o               h20670df_0
pip                       10.0.1                   py36_0
pygpu                     0.7.5            py36h14c3975_0
python                    3.6.5                hc3d631a_2
readline                  7.0                  ha6073c6_4
scipy                     1.0.1            py36hfc37229_0
setuptools                39.1.0                   py36_0
six                       1.11.0           py36h372c433_1
sqlite                    3.23.1               he433501_0
theano                    1.0.1            py36h6bb024c_0
tk                        8.6.7                hc745277_3
wheel                     0.31.0                   py36_0
xz                        5.2.3                h5e939de_4
zlib                      1.2.11               ha838bed_2

 

Here's my .theanorc file:

[global]
device = cuda0
optimizer_including = cudnn
floatX = float32

[dnn]
include_path = /usr/local/cuda/include
library_path = /usr/local/cuda/lib64

[lib]
cnmem = 0.7

[nvcc]
fastmath = True

[blas]
# Only used for device = cpu
ldflags = -lopenblas

[cuda]
root = /usr/local/cuda/bin

 

Nvidia driver: Driver Version: 384.81

[Arnaud Bergeron]

This is a new one.  It is also very weird since gemm doesn't involve cuLinkAddData.  This may be an error message from something else.

First things first, since you are on cuda 9.0, I would recommend that you update your driver to 384.111 or 390.*.  If that doesn't help, then I'll need some help reproducing the problem since I don't get that in any of my environments.

--

---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klachko]

After struggling with this error for a day, I decided to upgrade CUDA to 9.1 and CuDNN to 7.1. After that I got "your driver might be too old" error, which was resolved by updating the driver to 396.24. Also, in the process I found out I had older CuDNN files in /usr/lib/x86_64-linux-gnu/ directory. Not sure how they got there, perhaps because sometimes I installed CuDNN using .deb package, and sometimes by manually copying the files. So it's probably not a good idea to mix .deb and .run cuda installation methods.

Anyway, now theano works fine now, but unfortunately my Tensorflow is broken because it does not support cuda 9.1 yet... Will probably have to compile it from source.

[Frédéric Bastien]

You could have multiple cuda version installed to have TF working.

[Michael Klachko]

What is the best way to do that? Should I use separate conda environments for theano and tensorflow, and create LD_LIBRARY_PATH in each? Does dnn.libary_path in theanorc override LD_LIBRARY_PATH?

[Frédéric Bastien]

creating 2 env seem the right thing to do. I do not think dnn.libary_path override LD_LIBRARY_PATH.