Kilsort on Ubuntu 16.04

[cescho...@gmail.com]

Hi,

We just set up KiloSort on our Ubuntu Linux server. It was a bit painstaking, principally because the current version Matlab does not support CUDA 8. In the interest of saving time for anybody else who would like to go down that road, here is what worked:

• GPU: GTX 1080
• OS: Ubuntu Linux 16.04 Desktop
• Matlab version: R2016a. This release does not support CUDA > 7.5.
• Driver: NVIDIA 367

sudo add-apt-repository ppa:graphics-drivers/ppa

sudo apt-get update 

sudo apt-get install nvidia-367 

• CUDA version: 7.5. Download CUDA here. Note that there is no version for Ubuntu 16.04. However, in our hands the release for 15.04 works fine (use the runfile(local) installer type and decline driver install). Note that this release does not support gcc > 4.9, hence the next step.
  
• C version: gcc-4.9 and g++-4.9 (just in case). Ubuntu 16.04 comes with is 5.3 so it is necessary to install this older version and tell CUDA where to find it (assuming you want to keep 5.3 as default):

sudo ln -s /usr/bin/gcc-4.9 /usr/local/cuda/bin/gcc

sudo ln -s /usr/bin/g++-4.9 /usr/local/cuda/bin/g++

All of this would probably not be an issue if Matlab supported CUDA 8 (which has been released for Ubuntu 16.04 and is the recommended release for the GTX 1080). However CUDA 8 is still an RC version and from what I can gather it is unlikely that Matlab will support it anytime soon.

Carl Schoonover

Axel Lab

Columbia University

[cescho...@gmail.com]

After an unfortunate incident we had to reinstall the OS on our Linux server. The installation instructions I posted back in September I now realize were a bit sparse; below follows a full blow-by-blow. This is for installation on Ubuntu 16.04 64bit with a NVIDIA GTX 1080 GPU. The reason this is so convoluted is that the current version of MatLab (R2016b) does not support the current version of CUDA (8), and instead only supports an older version (7.5), which in turn does not support the default gcc that ships with 16.04.

By the way apologies to Marius for butchering the name of his software in the post title; and THANK YOU for sharing it!

-Carl


Install the NVIDIA drivers

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get update
sudo apt-get install nvidia-367

Install a version of gcc that is supported by CUDA 7.5
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-4.9 g++-4.9

Some of these libraries are needed to compile the CUDA samples to test successful CUDA installation (I'm not entirely sure which are necessary, can always uninstall after testing)
sudo apt-get install freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev

Download CUDA 7.5 from https://developer.nvidia.com/cuda-75-downloads-archive  For ubuntu 16.04 64bit choose:
- Operating System: Linux
- Architecture: x86_64
- Distribution: Ubuntu
- Version: 15.04 (note that there is no 16.04 version available for CUDA 7.5; however, this version works fine on our system)
- Installer type: runfile(local)

After downloading, navigate to the directory containing the downloaded file and run
sudo sh cuda_7.5.18_linux.run --override
- type accept to accept the EULA
- ignore the warning that you are attempting to install on an unsupported configuration by typing y
- type n to decline the graphics driver (the one you've installed is more recent)
- type y to install the CUDA 7.5 toolkit (notes below assumes you accept the default installation folder)
- type y to install the symbolic link
- type y to install the samples (can be used to test that installation was successful; notes assume default installation folder; this folder can be moved or deleted after testing without affecting CUDA)

Edit .bashrc
nano ~/.bashrc
and add this line:
PATH=$PATH:/usr/local/cuda-7.5/bin
Save, exit and run
source ~/.bashrc

Create and edit a new file:
sudo nano /etc/ld.so.conf.d/CUDAlibs.conf
and add this line:
/usr/local/cuda-7.5/lib64
Save, exit and run (the second command is to confirm all is in order)
sudo ldconfig
/sbin/ldconfig -p | grep cuda-7.5

Tell CUDA to use gcc-4.9 instead of the default gcc

sudo ln -s /usr/bin/gcc-4.9 /usr/local/cuda/bin/gcc
sudo ln -s /usr/bin/g++-4.9 /usr/local/cuda/bin/g++

Confirm that CUDA installation worked
cd ~/NVIDIA_CUDA-7.5_Samples/5_Simulations/nbody
make
./nbody

KiloSort installation instructions in the readme on GitHub https://github.com/cortex-lab/KiloSort, reproduced here.
Clone KiloSort and npy-matlab in your code folder
git clone https://github.com/cortex-lab/KiloSort.git
git clone https://github.com/kwikteam/npy-matlab.git

Open Matlab and navigate to KiloSort/CUDA/ folder, run mexGPUall.m

Test successful installation by navigating to KiloSort/eMouse.
First, set correct fpath (line 3) and paths for KiloSort, npy-matlab and config file (lines 7-9). Note that the config file is in KiloSort/eMouse
Then run master_eMouse.m and check output in phy.

[John Wolf]

Carl -

Thanks for posting this tutorial, and the off-line discussion.

For others benefit, I was able to get this up and running relatively easily on our Linux server running Ubuntu 16.04, Matlab 2017a and CUDA 8.0, and I am by no means a Linux guru.

The version of gcc that was installed with Ubuntu's latest update was compatible with CUDA 8, so we didn't have to go through the process of using a different gcc.

We did need to add the path to the CUDA binaries to get things running.

It works well for us to have one server running headless to do the sorting so we could put a beefy card in it, and we have a RAID that is mounted to the server for the data/config files that are called from MATLAB running on the server.

Hope this helps others interested in getting this running.

Thanks to Carl for the detailed post, and of course Marius as well!

--John

[Mike Sintsov]

@John! 

Could be a bit an offtopic. Could you please, describe in more details your workflow on the clustering and the server config, please? We're getting an HW update and I'm trying to make something like a server for my lab rather than private computation.

BR,

Mike