CUDA on a CentOS system

[Paolo Zaffino]

Dear Dr. Sharp and dear plastimatch users,

I would like to install the cuda driver/toolkit on a CentOS 6.2 system in order to use the GPU acceleration.
I found this guide:
https://www.centos.org/modules/newbb/viewtopic.php?topic_id=32811
Is this the right procedure (without install the SDK)?
I have never installed CUDA and any advice is welcome.
Thank you very much.

Best regards.

Paolo

[James Shackleford]

Hi Paolo,

The tutorial looks good.  I will give my comments below in red with the tutorial you posted in order to provide a little more insight.

From: https://www.centos.org/modules/newbb/viewtopic.php?topic_id=32811

-----<SNIP>-----<SNIP>-----<SNIP>-----

Download the following from nvidia:

NVIDIA-Linux-x86_64-270.41.19.run

This is the display/cuda driver.  Plastimatch has been tested up to the most recent version (290.10) with no issues.  If you need a prior version it would be due to some funniness with CentOS.  Because this is also a display driver (in addition to being the CUDA driver), your current display driver will be replaced.

cudatoolkit_4.0.17_linux_64_rhel6.0.run

This contains the cuda compiler (nvcc), the include (.h files) and shared libraries need to compile cuda programs.  I have tested 4.0 with success, so this should be fine.

gpucomputingsdk_4.0.17_linux.run

This file only installs source examples and an example build system for developers.  This is not necessary, but it might be useful to have just to test if your CUDA driver and toolkit installations are working properly.  For example, if you have trouble with Plastimatch, download this and try to build some of the examples.

####### Install nVidia Driver ##############################

Edit /boot/grub/grub.conf and add the following kernel options:

rdblacklist=nouveau nouveau.modeset=0

Nouveau is an open-source nvidia graphics card driver.  By adding it to the black list you are ensuring that CentOS will not try to load it.  This prevents conflicts with the closed source binary nvidia driver (cuda driver) you are about to install.  The nvidia driver (from NVIDIA-Linux-x86_64-270.41.19.run) will replace nouveau.

Reboot to runlevel 3 (add "3" without quotes to kernel options)

Runlevel 3 will put you into a text-only (non-graphical) console mode.  This is necessary because you cannot install the nvidia driver while it is in use by a graphics X11 session.

# yum groupinstall 'Development Tools' 'Development Libraries'

I am not familiar with how CentOS packages things, but it looks to me like this just installs the standard GNU development tool chain.

# sh NVIDIA-Linux-x86_64-270.41.19.run

This actually installs the nvidia graphics/cuda driver.  Make sure you are root when doing this (as indicated by the # prompt in the example).

# reboot

Graphics should work now.

X11 should now be using the nvidia graphics/cuda driver instead of nouveau.  If you type lsmod in a terminal, you should see the nvidia module loaded (it is named 'nvida') and nouveau should NOT be loaded (because we blacklisted it earlier).

####### Install CUDA Toolkit ##############################

# sh cudatoolkit_4.0.17_linux_64_rhel6.0.run

This will install the cuda compiler (nvcc) and associated libraries, includes, and documentation.  I recommend the default location of /usr/local/cuda

Create /etc/ld.so.conf.d/cuda.conf with the following two lines:
/usr/local/cuda/lib64
/usr/local/cuda/lib
This will tell the GNU linker (ld) where to find the cuda shared libraries installed by the toolkit.  If you are on a 64-bit machine you will have both 32-bit and 64-bit libraries, so you will need to add both /usr/local/cuda/lib64 and /usr/local/cuda/lib to the /etc/ld/so.conf.d/cuda.conf file you are going to create.  If you are on 32-bit linux, you will only need to add /usr/local/cuda/lib

# ldconfig

This tells the GNU linker (ld) to reload its configuration files.  Since you just added a configuration file for cuda (cuda.conf), you must do this so that ld will become aware of the cuda libraries.  You will only need to do this once.

Add the following to your $PATH via ~/.bash_profile
/usr/local/cuda/bin
This is just puts the cuda build utilities (such as nvcc) in your path.

Log out and back in to update your path; or some other slick command to do it without loging out.

The other "slick command" is: $ exec bash --login -i

I suppose you could run cuda-enabled programs now... but I didn't try without installing the SDK as well.
Like I said earlier, the SDK is optional.  The rest here just describes installing the example projects and compiling them.  This tests two things: 1) if the cuda toolkit is compiling correctly, and 2) that the cuda driver is running the compiled programs correctly.

####### Install CUDA SDK ################################

As a regular user:

$ sh gpucomputingsdk_4.0.17_linux.run

The example cuda programs need the following:

# yum install freeglut freeglut.i686 freeglut-devel freeglut-devel.i686 libXi-devel libXi-devel.i686 libXmu libXmu.i686 libXmu-devel libXmu-devel.i686

NOTE: you probably don't need the i686 versions...

####### Test #########################################

If you actually ran ldconfig as root and updated your $PATH, then the cuda examples should compile

$ cd ~/NVIDIA_GPU_Computing_SDK/C
$ make clean
$ make
$ cd bin/linux/release
$ ./randomFog

-----<SNIP>-----<SNIP>-----<SNIP>-----

Hope this helps and good luck!

Let us know if you run into any problems.

--Tony

[Paolo Zaffino]

Hi Tony,

thank you very much for your advice.
I'll take you update.

Paolo

[Paolo Zaffino]

Hi Tony,

I followed the guide and I installed the cuda driver/toolkit/SDK without problem.
When I configured plastimatch cmake returned me some warnings about some flags (sorry but I have forgotten to save the log) and the CUDA_FOUND field was on OFF. I set it and PLM_BUILD_CUDA_TEST on ON and I compiled plastimatch without errors (infact plastimatch works).
The problem is that I don't understand if plastimatch uses cuda or not.
Is there a way to understand this?
Another thing that I noted is that on this pc cmake dosen't show me all the cuda fields (CUDA_NVCC_FLAGS, CUDA_TOOLKIT_INCLUDE, etc etc) that instead I am able to see on another pc without cuda software.
Unfortunately the machine where was installed cuda is in lab now and so I can't have access to it before Monday.

Thank you very much.
Regards.

Paolo




Il 07/02/2012 17:47, James Shackleford ha scritto:

[Gregory Sharp]

Hi Paolo,

I suspect that it didn't work. You can check if CUDA acceleration
is included in the executable by looking for these lines during compile:

Building NVCC (Device) object
src/plastimatch/./cuda_compile_generated_bspline_cuda.cu.o Building NVCC
(Device) object
src/plastimatch/./cuda_compile_generated_cuda_kernel_util.cu.o
[ 11%] Building NVCC (Device) object
src/plastimatch/./cuda_compile_generated_cuda_mem.cu.o

You can check if CUDA is used at runtime by looking for lines
like these:

Allocating GPU Memory..................done.
GPU Memory: 330 MB

You can also look at CPU usage, which will be low if GPU is being used.

Greg

> > The other "slick command" is: $ *exec bash --login -i*

> >
> > I suppose you could run cuda-enabled programs now... but I didn't try
> > without installing the SDK as well.
> > Like I said earlier, the SDK is optional. The rest here just
> > describes installing the example projects and compiling them. This
> > tests two things: 1) if the cuda toolkit is compiling correctly, and
> > 2) that the cuda driver is running the compiled programs correctly.
> >
> > ####### Install CUDA SDK ################################
> >
> > As a regular user:
> >
> > $ sh gpucomputingsdk_4.0.17_linux.run
> >
> > The example cuda programs need the following:
> >
> > # yum install freeglut freeglut.i686 freeglut-devel
> > freeglut-devel.i686 libXi-devel libXi-devel.i686 libXmu libXmu.i686
> > libXmu-devel libXmu-devel.i686
> >
> > NOTE: you probably don't need the i686 versions...
> >
> > ####### Test #########################################
> >
> > If you actually ran ldconfig as root and updated your $PATH, then the
> > cuda examples should compile
> >
> > $ cd ~/NVIDIA_GPU_Computing_SDK/C
> > $ make clean
> > $ make
> > $ cd bin/linux/release
> > $ ./randomFog

> > *-----<SNIP>-----<SNIP>-----<SNIP>-----*

> >
> >
> >
> > Hope this helps and good luck!
> >
> > Let us know if you run into any problems.

> > *
> > *
> > *--Tony*
>

--
Greg Sharp, PhD
Department of Radiation Oncology
Massachusetts General Hospital

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

[Paolo Zaffino]

Dear Dr. Sharp,

Yes, I think that there is a problem with CUDA (all the CPUs work if I
run a registration).
I attached the warning that cmake (version 2.6.4) returns.
I don't understand why cmake doesn't show me the CUDA flags.
As soon as possible I'll recompile plastimatch and I'll check if I get
those lines (but I'm pretty sure that they don't appear).
Thank you very much.

Regards.
Paolo

[Paolo Zaffino]

Dear Dr. Sharp and Plastimatch community,

My workstation with CentOS 6.2 doesn't compile plastimatch with the CUDA support.
The CUDA software is ok because I can compile the example included in the SDK package.
I'm using gcc 4.4.6 and cmake 2.6.4
CMake doesn't find CUDA (the CUDA binary folder is in the path).
Does anyone has an idea?

Thank you so much.
Regards.

Paolo

---

Da: Gregory Sharp <gcs...@partners.org>
A: plast...@googlegroups.com
Inviato: Martedì 14 Febbraio 2012 16:12
Oggetto: Re: [Plastimatch] Re: CUDA on a CentOS system

[Paolo Zaffino]

Dear Dr. Sharp and Plastimatch community,

today I solved the problem!
I downloaded and compiled cmake 2.8.8 and it recognized CUDA.
So the problem was the older version of cmake (that I got by the official centos repositories).
Once succesfully terminated the cmake step I was able to compile Plastimatch and run a registration using the GPU.
Anyway I noted two things:

1) when I use CUDA the video reactivity is no good...it seems to have a lot of little freezes. Anyway the registration task goes on without problem. Until now I have never used CUDA and so I don't know if this is a "normal" behavior (the CPUs aren't completly stopped when a GPU registration is running).

2) I don't get a good registration because it escapes before to achive the same image that I get without che GPU acceleration. I set two deformable stages, each of 100 iterations. Without CUDA it runs all this iterations but if I enable CUDA it stops respectively at 70 and 49 iterations.
On the screen it prints me this:

ABNORMAL_TERMINATION_IN_LNSRCH                             
 Line search cannot locate an adequate point after 20 function
  and gradient evaluations.  Previous x, f and g restored.
 Possible causes: 1 error in function or gradient evaluation;
                  2 rounding error dominate computation.

Do you have any idea about?

[Gregory Sharp]

Hi Paolo,

That is good news. Indeed the CUDA requires at least cmake 2.8.0,
but somehow we were too focused on driver or library versions!

I can't answer much about the video glitches, because now I have
dedicated Tesla card. But when I used to run on my graphics card
(this was Windows XP), I sometimes got a frozen screen. Anyone else
have any confirmation?

For (2), can you run a little test? Just run "make test"
in the build directory. Do all the tests pass?
Next, look at the score for the first couple of iterations.
Do you get the same scores for CPU and GPU?

Greg

[Paolo Zaffino]

Dear Dr. Sharp,

yes, this is a good news! :)

My video card is a Nvidia quadro 4000, I don't know if this information can be useful.

I ran the "make test" command and all the tests were passed.
Also I did the log files for the registration with and without the GPU acceleration, I attached them.
The cost function is very different and, as the "waring" says, when I use the GPU it seems not found a correct direction.

Thank you very much.
Best regards.

Paolo

---

Da: Gregory Sharp <gcs...@partners.org>
A: plast...@googlegroups.com

Cc: p.za...@yahoo.it
Inviato: Venerdì 11 Maggio 2012 2:35

Oggetto: Re: [Plastimatch] Re: CUDA on a CentOS system

[Sharp, Gregory C.]

Hi Paolo,

 

Thank you for the log file, it is useful.  We have seen this problem before, but

don't know the exact cause. 

 

One possibility that we discuss is difference in machine precision.  The GPU algorithms

are all done using 32-bit float.  CPU is done mostly 32-bit, but uses 64-bit for accumulation.

This gives CPU a superior numeric precision.  Another possibility is an undiscovered bug.

 

One workaround you could try is re-starting with another stage.  Whenever you start a 

new stage, the LBFGS-B optimizer gets reset, and you get new line searches.

 

Greg

 

---

From: plast...@googlegroups.com [mailto:plast...@googlegroups.com] On Behalf Of Paolo Zaffino
Sent: Friday, May 11, 2012 7:21 AM
To: plast...@googlegroups.com
Subject: Re: [Plastimatch] Re: CUDA on a CentOS system

[Paolo Zaffino]

Dear Dr Sharp,

If I must do further tests I'm here.
Just for notice I have another desktop with a GForce video card (with the same OS and driver) and I get an out of memory for the GPU registration.

Thanks a lot.
Paolo