Using CUDA from Ubuntu PPA

[benmccann]

Hi,
I have a version of CUDA installed from an Ubuntu PPA:
https://launchpad.net/~aaron-haviland/+archive/cuda-4.0

I'm able to compile and run a sample CUDA program of my own. However,
I'm not sure how to setup Theano to run with this version of CUDA.
The docs talk about a directory structure that doesn't quite exist. I
currently have THEANO_FLAGS set to "cuda.root=/usr,device=gpu". I
chose cuda.root as /usr since nvcc is in /usr/bin/nvcc. I'm not sure
how to edit LD_LIBRARY_PATH. When I run the sample program it says
it's using the CPU. Any tips or suggestions?

Btw, it would be good to get this working since I believe this PPA
package might become an official Ubuntu package in the next version of
Ubuntu (11.10).

Thanks,
Ben

[Frédéric Bastien]

Hi,

Can you try with this:

THEANO_FLAGS set to "device=gpu,floatX=float32"

I suppose the package place everything in the common place. So there
is no needd to specify where nvcc is.

Currently only float32 is supported on the GPU. We use a special type
called floatX to allow changing when we start a program between
float32 and float64. It default to float64.

Fred

[Ben McCann]

Thanks for the suggestion.  Still didn't work though.  Any other ideas on troubleshooting why it might be preferring the CPU?

I'm running the sample program from deeplearning.net which prints whether it uses the CPU or GPU.  It's printing that it's using the cpu.

Some other details: my graphics card is a CUDA-supported GeForce 210, I'm running 32-bit Ubuntu 11.04, I installed the CUDA PPA which put the CUDA binaries in /usr/bin and the CUDA libraries in /usr/lib

Thanks again for the help,

Ben

2011/5/14 Frédéric Bastien <no...@nouiz.org>

[Ben McCann]

ping.  anyone know how I can get Theano running on my GPU or steps I can take to troubleshoot?

Thanks,

Ben

[Ian Goodfellow]

You mentioned earlier you're not sure how to edit LD_LIBRARY_PATH.
Assuming you're using a bash shell, you can do this with the "export"
command, and probably want to put the export in your ~/.bashrc file.
Make sure LD_LIBRARY_PATH contains the path to your lib64 directory
containing the cuda .so files.

[Ian Goodfellow]

By the way, if you import theano in gpu mode on a gpu-enabled machine,
you should see a message telling you what GPU you're using. Do you get
that far?

THEANO_FLAGS="device=gpu" python
Python 2.5.1 (r251:54863, Jun 15 2008, 18:24:56)
[GCC 4.3.0 20080428 (Red Hat 4.3.0-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
Using gpu device 0: GeForce GTX 470

You could also try checking that the following is true:
'nvcc' in os.listdir(theano.config.cuda.root+'/bin')

[Ben McCann]

Getting further!  When I import theano I now get a message telling me what GPU I'm using.  I do know how to export variables, but was just saying I didn't know what values to set them to since my setup's a bit different than the tutorial's.

I took the code from the "Using the GPU" tutorial under the "Putting it all Together" section into a file named cuda_test.py. When I run it, the screen fills with hundreds of errors:

$ python cuda_test.py

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpzvu5gL/../cuda_ndarray/cuda_ndarray.so:17: warning: null character(s) ignored

I have in my .bashrc:

export THEANO_FLAGS='device=gpu,floatX=float32'

export LD_LIBRARY_PATH='/usr/lib'

Running the suggested commands in the Python REPL (btw, it would be helpful to have this in the docs):

>>> import theano

Using gpu device 0: GeForce 210

>>> import os

>>> 'nvcc' in os.listdir(theano.config.cuda.root+'/bin')

True

Thanks,

Ben

[Frédéric Bastien]

HI,

Those are warning, not error. What it print after?
replace export LD_LIBRARY_PATH='/usr/lib' with export
LD_LIBRARY_PATH="/usr/lib:$LD_LIBRARY_PATH"
(you must use double quote, not single quote...)

To remove the warning, can you try the note that talk about
compiler_bindir in this page:
http://deeplearning.net/software/theano/tutorial/using_gpu.html?highlight=compiler_bindir

Fred

[Ben McCann]

I made the changes you suggested to LD_LIBRARY_PATH and rebooted, but it doesn't / shouldn't make any difference to what my LD_LIBRARY_PATH is being set to.

If I let the program run to completion it prints 15,000 lines of warnings like the ones mentioned and then the error below.  I'm running Cuda 4.0 and have already set my system gcc default to 4.4.  Is Theano compatible with Cuda 4.0?

Traceback (most recent call last):

  File "cuda_test.py", line 11, in <module>

    f = function([], T.exp(x))

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 105, in function

    allow_input_downcast=allow_input_downcast)

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 270, in pfunc

    accept_inplace=accept_inplace, name=name)

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1109, in orig_function

    fn = Maker(inputs, outputs, mode, accept_inplace = accept_inplace).create(defaults)

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 984, in create

    _fn, _i, _o = self.linker.make_thunk(input_storage = input_storage_lists)

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 321, in make_thunk

    output_storage = output_storage)[:3]

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1184, in make_all

    output_storage = node_output_storage)

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 780, in make_thunk

    cthunk, in_storage, out_storage, error_storage = self.__compile__(input_storage, output_storage)

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 729, in __compile__

    output_storage)

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1043, in cthunk_factory

    module = get_module_cache().module_from_key(key=key, fn=self.compile_cmodule)

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 429, in module_from_key

    module = fn(location=location)  # WILL FAIL FOR BAD C CODE

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 972, in compile_cmodule

    preargs=preargs)

  File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/nvcc_compiler.py", line 228, in nvcc_module_compile_str

    raise Exception('nvcc return status', p.returncode, 'for cmd', ' '.join(cmd))

Exception: ('nvcc return status', 1, 'for cmd', 'nvcc -shared -g --maxrregcount=32 -O3 -m32 -Xcompiler -Wno-write-strings,-fno-math-errno,-Wno-unused-label,-Wno-unused-variable,-fPIC,-m32 -Xlinker -rpath,/usr/lib -Xlinker -rpath,/usr/lib64 -I/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/cuda_ndarray -I/usr/include -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -o /home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpD4H2I0/0348d714aff443efa17990e312b68180.so mod.cu /home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/tmpD4H2I0/../cuda_ndarray/cuda_ndarray.so -L/home/bmccann/.theano/compiledir_Linux-2.6.38-8-generic-i686-with-Ubuntu-11.04-natty-i686-2.7.1+/cuda_ndarray -L/usr/lib -L/usr/lib -L/usr/lib64 -lpython2.7 -lcudart -lcublas')

2011/5/17 Frédéric Bastien <no...@nouiz.org>

[Ian Goodfellow]

I'm not sure about Cuda 4.0, but you don't need to reboot after
changing LD_LIBRARY_PATH. If you change it by modifying your bashrc
file, all you need to do is run
source ~/.bashrc

[Frédéric Bastien]

Hi,

The real error is missing. Can you make sure you only remove the 15k
line of the warning and nothing else?

To my knowledge, nobody tested theano with cuda 4. Can you try with
another version? From the changelog of cuda 4.0, it should be
working...

Fred

[Ben McCann]

Hmm, sorry, I thought that was the error.  I've attached the whole output, but with most of the warning lines removed to keep the size down. 

I may try installing an older version of Cuda if I get the chance, but I'm afraid that will be much harder.  gcc-4.3 isn't even available on Ubuntu 11.04 anymore.  There will be many more hoops to jump through.  If anyone does try Theano on Cuda 4.0, I'd be interested to know the results.  Seems like we should make an effort to support 4.0 as well.

Thanks again for the help,

Ben

2011/5/17 Frédéric Bastien <no...@nouiz.org>

[Arnaud Bergeron]

Actually the long list of warnings in cuda_ndarray.so indicates that
g++ is trying to parse the file as a source file. I have no idea how
fix that on the top of my head. You could look at the command line
that theano tries to run and run it manually until it works and report
what made it work.

2011/5/17 Ben McCann <b...@benmccann.com>:

--
La brigade SnW veut vous recruter - http://www.brigadesnw.ca

[Ian Goodfellow]

Installing the older version of cuda is pretty easy; just download the
drivers directly from nvidia and follow the instructions for their
installer. It takes about 5 minutes, and the last time I did it was a
few weeks ago so it should definitely be working.

[Arnaud Bergeron]

2011/5/17 Ian Goodfellow <goodfel...@gmail.com>:

> Installing the older version of cuda is pretty easy; just download the
> drivers directly from nvidia and follow the instructions for their
> installer. It takes about 5 minutes, and the last time I did it was a
> few weeks ago so it should definitely be working.

But they require specific versions of gcc that might be harder to install.

--

[Ian Goodfellow]

Oh, does 11.10 default to a newer compiler than 10.10 does? I'm still
running 10.10 and the nvidia drivers work fine with whatever
build-essential installs.
But also, I thought that in the default case installing the drivers
did not require compilation. Or do you mean nvcc will only produce
outputs that are compatible with the older gcc?

[Arnaud Bergeron]

2011/5/17 Ian Goodfellow <goodfel...@gmail.com>:

> Oh, does 11.10 default to a newer compiler than 10.10 does? I'm still
> running 10.10 and the nvidia drivers work fine with whatever
> build-essential installs.
> But also, I thought that in the default case installing the drivers
> did not require compilation. Or do you mean nvcc will only produce
> outputs that are compatible with the older gcc?

I mean nvcc has a weird dependency with gcc and tends to work only
with one specific version. You may get lucky and make it work with
another, but you may also end up with weird bugs.

[Ben McCann]

Yeah, Cuda 3.0 requires gcc 4.3.  You unfortunately can't use a newer version from what I've read.  You might be able to use an older version, but I'm not sure.  Cuda 4.0 requires gcc 4.4.  gcc 4.5 is installed by default on Ubuntu 11.04, which is what I'm running.  I was able to downgrade to gcc 4.4, which should work fine with Cuda 4.0.  However, gcc 4.3 isn't available in the default Ubuntu repositories anymore, which I think will make running Cuda 3.0 much more difficult.  If Theano was confirmed to work with Cuda 4.0 that would make things much easier.

[Olivier Delalleau]

How did you install cuda 4.0 on Ubuntu 11.04?
I got an Ubuntu 11.04 with an nvidia GPU I could do tests with (hoping it is cuda-compatible...), but I can't find Ubuntu 11.04 listed on http://developer.nvidia.com/cuda-toolkit-40

-=- Olivier

2011/5/17 Ben McCann <b...@benmccann.com>

[Ben McCann]

I got it from the PPA:

https://launchpad.net/~aaron-haviland/+archive/cuda-4.0

I'm guessing you could also download the version listed for Ubuntu 10.10 and it would work:

http://developer.nvidia.com/cuda-toolkit-40#Linux

Also, it's recommended that you downgrade your version of GCC:

  sudo apt-get install g++-4.4 gcc-4.4

  sudo rm /usr/bin/gcc /usr/bin/g++

  sudo ln -s /usr/bin/gcc-4.4 /usr/bin/gcc

  sudo ln -s /usr/bin/g++-4.4 /usr/bin/g++

[Olivier Delalleau]

Ok thanks, sorry I had missed it in your OP.
I had some install trouble... won't be able to look into it more until Friday, sorry.

-=- Olivier

2011/5/18 Ben McCann <b...@benmccann.com>

[Ben McCann]

No problem, thanks for taking a look.

[Olivier Delalleau]

Unfortunately I'm having trouble with Cuda. Theano fails early with the Cuda error:
Unable to get the number of gpus available: no CUDA-capable device is detected

I believe I didn't install the cuda drivers properly. I actually wasn't able to boot Ubuntu anymore after installing them, and ended up messing with the drivers (with help from someone more familiar with Linux than myself). I suspect I erased the cuda drivers ;)
On another hand, I just realized my gpu (GeForce 7800 GT) is not listed on the list of cuda-compatible chipsets (http://www.nvidia.com/object/cuda_gpus.html), so I think I'll just give up to avoid breaking my system again.
Sorry I couldn't help more,

[Frédéric Bastien]

Hi,

I have make theano work on another system that I have access with cuda
4.0. So the problem is related to your installation and/or PA and/or
the gcc version.

HTH

Fred

On Fri, May 20, 2011 at 2:09 PM, Olivier Delalleau

[Ben McCann]

Awesome.  Thanks so much for checking that for me.  I'm on vacation right now but am planning to investigate more when I get back in two weeks

[Eldad]

Hello,

I am experiencing a similar problem:

I have installed the CUDA packs and NVIDIA driver using the PPA
mentioned above, on ubuntu 11.04 (Natty) 64-bit:
The installation seems successful - as the SDK examples from NVIDIA's
webpage can be compiled [using gcc/g++-4.4] and run on my GeForce 9500
GT .

The problem:
After installing Theano [pip install Theano] I experience the
following failures -
* The theano.test() gets stuck in some never-ending loop;
* The test script "thing.py" (from http://deeplearning.net/software/theano/tutorial/using_gpu.html
) outputs an error similar to what's described above, when
THEANO_FLAGS device=gpu.

Any suggestion for solutions are mostly welcome!
Eldad

[Frédéric Bastien]

Hi,

I don't have time to install this version of OS and test it. But it is
not the first time that there is problem with CUDA 4.0. In one case it
worked...

Can you delete the Theano cache? "Theano/bin/theano-cache clear". If
it don't work, try CUDA 3.2, but some people talked about problem with
the version of gcc available on that OS...

Fred

On Wed, Jun 22, 2011 at 2:18 AM, Eldad <eldad...@gmail.com> wrote:
> I am experiencing a similar problem to Ben McCann's:
> 1. CUDA installed from the PPA on ubuntu 11.04 (Natty) 64-bit;
> 2. SDK examples downloaded from the website, and compiled successfully
> using gcc/g++-4.4; based on http://forums.nvidia.com/index.php?
> showtopic=196892 )
> 3. Theano installed using pip.
>
> Problems:
> * theano.test() gets into some incredibly long (did not stop by itself
> not even after 10min) loop of with some compilation errors of the time
> reported above)
> * thing.py works fine for the cpu but has similar errors as the above
> for gpu.
>
> Have not tested yet with older versions of CUDA.
> Help would be highly appreciated!
> Thanks

>> >>>>>>> <aberge...@gmail.com> wrote:
>>
>> >>>>>>>> 2011/5/17 Ian Goodfellow <goodfellow....@gmail.com>:

>> >>>>>>>> > Oh, does 11.10 default to a newer compiler than 10.10 does? I'm
>> >>>>>>>> > still
>> >>>>>>>> > running 10.10 and the nvidia drivers work fine with whatever
>> >>>>>>>> > build-essential installs.
>> >>>>>>>> > But also, I thought that in the default case installing the
>> drivers
>> >>>>>>>> > did not require compilation. Or do you mean nvcc will only
>> produce
>> >>>>>>>> > outputs that are compatible with the older gcc?
>>
>> >>>>>>>> I mean nvcc has a weird dependency with gcc and tends to work only
>> >>>>>>>> with one specific version.  You may get lucky and make it work with
>> >>>>>>>> another, but you may also end up with weird bugs.
>>
>> >>>>>>>> > On Tue, May 17, 2011 at 3:26 PM, Arnaud Bergeron

>> >>>>>>>> > <aberge...@gmail.com> wrote:
>> >>>>>>>> >> 2011/5/17 Ian Goodfellow <goodfellow....@gmail.com>:

[James Bergstra]

I lost a little bit the track of this thread, but I have reproduced
the original problem, which is that nvcc prints out a zillion lines
about ignored null characters in the .so file. Arnaud rightly pointed
out that [the new version of] nvcc is treating the cuda_ndarray.so
file as a source file. The reason it is doing that, is that
cuda_ndarray.so is listed directly in the nvcc command line as if it
were a source file. Seems crazy to me - why are we doing that?

This afternoon I verified a fix that I haven't committed yet, but will
soon. The fix is to create a file "libcuda_ndarray.so" that is a
symbolic link to cuda_ndarray.so. That way files that need to link
against cuda_ndarray's C API can do that using the normal -Ldir
-lcuda_ndarray syntax. In my by-hand fix I added the cuda_ndarray
folder to LD_LIBRARY_PATH. In a proper fix,
sandbox/cuda/nvcc_compiler.py or whatever file that is would add the
architecture-dependent .theano/<arch>/cuda_ndarray to the
LD_LIBRARY_PATH sometime before invoking nvcc the first time.

Everyone fine with the fix of this issue? I'll push it tomorrow.
There may have been other issues raised later in this thread -- I'm
not addressing them that I'm aware of.

James

2011/6/22 Frédéric Bastien <no...@nouiz.org>:

--
http://www-etud.iro.umontreal.ca/~bergstrj

[Olivier Delalleau]

Note that symbolic links do not work well with Windows (although there was a similar problem with Windows related to that same command line, and someone posted another solution to this recently that I haven't been able to look into yet, so it may not be a big deal to do something Windows-unfriendly here at this point).
Also, it is cleaner to invoke nvcc with the proper -L flag rather than messing with LD_LIBRARY_PATH.

Good job figuring this out :) I must admit I've always wonder why it worked the way we've been doing it so far...

-=- Olivier

2011/6/30 James Bergstra <james.b...@gmail.com>

[Ben McCann]

James, thanks so much for figuring this out.  It will make it far easier to use Theano.  Did you commit the fix?

Will there be another release of Theano coming up anytime soon to push the bug fix?

Thanks again for the help,

Ben

[Ben McCann]

I figured out how to install theano from source, which was really easy (sudo pip install --upgrade hg+http://hg.assembla.com/theano).  However, I'm still having some difficulty.  Anyone familiar with the error below or have an suggestions?

$ python cuda_test.py 

COMPILING NEW CVM

Traceback (most recent call last):

  File "cuda_test.py", line 1, in <module>

    from theano import function, config, shared, sandbox

  File "/usr/local/lib/python2.7/dist-packages/theano/__init__.py", line 52, in <module>

    import compile

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/__init__.py", line 2, in <module>

    import function_module

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 17, in <module>

    import mode as mode_module

  File "/usr/local/lib/python2.7/dist-packages/theano/compile/mode.py", line 7, in <module>

    import theano.gof.vm

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 367, in <module>

    import lazylinker_c

  File "/usr/local/lib/python2.7/dist-packages/theano/gof/lazylinker_c.py", line 40, in <module>

    code = open(cfile).read()

IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/theano/gof/lazylinker_c.c'

[Olivier Delalleau]

It's the first .c file we added and we forgot to add the .c files to setup.py so that they are properly installed.
Should be fixed now :)

-=- Olivier

2011/7/9 Ben McCann <b...@benmccann.com>

[Ben McCann]

Success!  Thanks so much for the quick fix Olivier.

I wrote up some instructions on how to use the CUDA from the Ubuntu PPA in case anyone's interested.  I think it's easier than the normal route now that the Theano fixes to allow it are in.

Thanks again,

Ben

[Frédéric Bastien]

Hi,

I added a link to your instruction and the command you gived for pip
with the bleeding-edge version. They should apprear in the next hours.

thanks

Fred