CUDA separable compilation? (CUDA 6.5 on Ubuntu 14.04 via Travis CI)

[Gabriel Hare]

I'm looking for help understanding how to use GPU Ocelot when building a separately compiled library as described here:

https://devblogs.nvidia.com/parallelforall/separate-compilation-linking-cuda-device-code/

In a separable compilation, nvcc is used to generate the X.o files. These files are then merged using nvcc into an intermediate Y.o file, which is finally made into a library.

What I'm unsure of is how to use GPU Ocelot in this process. Any advice or links to examples would be very welcome!

Now, for some background...

I'm working on a project that requires running on an NVidia GPU and would like to use GPU Ocelot to (among other things) support continuous integration testing. The code needs to be separately compiled, since it is necessary to link to both CUDA 6.5 code (which does not support C++11) and code that relies on C++11. (No, I can't upgrade to CUDA 7+.)

For the continuous integration testing I'm using Travis CI, which provides machines with Ubuntu 14.04 installed, but no GPU.

Thus far I've managed to install the ocelot_2.1.2181_amd64.deb from https://code.google.com/archive/p/gpuocelot/downloads. (This required getting Boost 1.49 and its dependencies, most of which is deprecated.)

With Boost 1.49 and friends installed, running "OcelotConfig -l" yields "-locelot" so I assume everything is working correctly. 

To set up my build environment I use CMake and the FindCUDA module. I've introduced an configure.ocelot file, and now I need to figure out how to modify my CMakeLists.txt files to link against -locelot correctly.

Finally, copied below is the verbose output of part of a build... (The FindCUDA module makes this process surprisingly convoluted, but the relevant build commands are all in there.)

[ 60%] Building NVCC (Device) object ImageAnalysis/CMakeFiles/ImageAnalysis.dir/ImageAnalysis_generated_particle.cpp.o

cd /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir && /usr/local/bin/cmake -E make_directory /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//.

cd /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir && /usr/local/bin/cmake -D verbose:BOOL=1 -D build_configuration:STRING=Release -D generated_file:STRING=/home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o -D generated_cubin_file:STRING=/home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o.cubin.txt -P /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.cmake

-- Removing /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o

/usr/local/bin/cmake -E remove /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o

-- Generating dependency file: /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.NVCC-depend

/usr/local/cuda-6.5/bin/nvcc -M -D__CUDACC__ /home/ghare/Documents/Loci/ImageAnalysis/particle.cpp -o /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.NVCC-depend -ccbin /usr/bin/c++ -m64 -Xcompiler ,\"-fvisibility=hidden\",\"-fPIC\",\"-O3\",\"-DNDEBUG\" -arch=compute_20 -code=sm_20 -code=compute_20 -DNVCC -I/usr/local/cuda-6.5/include -I/home/ghare/Documents/googletest-build/include -I/usr/local/cuda-6.5/include

-- Generating temporary cmake readable file: /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp

/usr/local/bin/cmake -D input_file:FILEPATH=/home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.NVCC-depend -D output_file:FILEPATH=/home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp -P /usr/local/share/cmake-3.4/Modules/FindCUDA/make2cmake.cmake

-- Copy if different /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp to /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend

/usr/local/bin/cmake -E copy_if_different /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend

-- Removing /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp and /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.NVCC-depend

/usr/local/bin/cmake -E remove /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.depend.tmp /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//ImageAnalysis_generated_particle.cpp.o.NVCC-depend

-- Generating /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o

/usr/local/cuda-6.5/bin/nvcc /home/ghare/Documents/Loci/ImageAnalysis/particle.cpp -x=cu -dc -o /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o -ccbin /usr/bin/c++ -m64 -Xcompiler ,\"-fvisibility=hidden\",\"-fPIC\",\"-O3\",\"-DNDEBUG\" -arch=compute_20 -code=sm_20 -code=compute_20 -DNVCC -I/usr/local/cuda-6.5/include -I/home/ghare/Documents/googletest-build/include -I/usr/local/cuda-6.5/include

Generated /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o successfully.

[ 80%] Building NVCC intermediate link file ImageAnalysis/CMakeFiles/ImageAnalysis.dir/ImageAnalysis_intermediate_link.o

cd /home/ghare/Documents/Loci-build/ImageAnalysis && /usr/local/cuda-6.5/bin/nvcc -arch=compute_20 -code=sm_20 -code=compute_20 -m64 -ccbin "/usr/bin/c++" -dlink /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particleFilter.cpp.o /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_particle.cpp.o /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir//./ImageAnalysis_generated_v3.cpp.o -o /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir/./ImageAnalysis_intermediate_link.o -Xcompiler -fPIC

cd /home/ghare/Documents/Loci-build && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /home/ghare/Documents/Loci /home/ghare/Documents/Loci/ImageAnalysis /home/ghare/Documents/Loci-build /home/ghare/Documents/Loci-build/ImageAnalysis /home/ghare/Documents/Loci-build/ImageAnalysis/CMakeFiles/ImageAnalysis.dir/DependInfo.cmake --color=

make[2]: Leaving directory `/home/ghare/Documents/Loci-build'

make -f ImageAnalysis/CMakeFiles/ImageAnalysis.dir/build.make ImageAnalysis/CMakeFiles/ImageAnalysis.dir/build

make[2]: Entering directory `/home/ghare/Documents/Loci-build'

[100%] Linking CXX static library libImageAnalysis.a

cd /home/ghare/Documents/Loci-build/ImageAnalysis && /usr/local/bin/cmake -P CMakeFiles/ImageAnalysis.dir/cmake_clean_target.cmake

cd /home/ghare/Documents/Loci-build/ImageAnalysis && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/ImageAnalysis.dir/link.txt --verbose=1

/usr/bin/ar qc libImageAnalysis.a  CMakeFiles/ImageAnalysis.dir/ImageAnalysis_generated_particleFilter.cpp.o CMakeFiles/ImageAnalysis.dir/ImageAnalysis_generated_particle.cpp.o CMakeFiles/ImageAnalysis.dir/ImageAnalysis_generated_v3.cpp.o CMakeFiles/ImageAnalysis.dir/ImageAnalysis_intermediate_link.o

/usr/bin/ranlib libImageAnalysis.a

[Gabriel Hare]

Solved-ish - see my subsequent post (https://groups.google.com/forum/#!topic/gpuocelot/fFsnE8zelTQ)

Briefly, AFAIK lib ocelot.so only needs to be included in the final linking. HOWEVER, since the resulting code was unable to run successfully that might not be correct.