CUDA_ERROR_NO_DEVICE

[Lila Rieber]

I'm trying to run juicebox on the Moab (the paid version of Torque ) resource manager with the Portable Batch System (PBS) for scheduling.  

I tried submitting the following script to the queue:

#PBS -W umask=0007 
#PBS -l walltime=20:00:00 
#PBS -l nodes=1:ppn=4 
#PBS -j oe 
#PBS -l mem=40gb 
module load cuda/7.0 
/storage/home/lur159/group/lab/lila/java/jre1.8.0_91/bin/java -Djava.awt.headless=true -Djava.library.path=../../../juicebox/native_launcher/natives -Xmx16000m -Xms16000m -jar juicebox_tools.jar $* hiccups https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined.hic all_hiccups_loops 

This produced the following output:

No configurations specified, using default settings
Reading file: https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined.hic
INFO [2016-07-05 13:55:52,798]  [DirectoryManager.java:149] [main]  IGV Directory: /storage/home/lur159/igv
INFO [2016-07-05 13:55:53,109]  [HttpUtils.java:809] [main]  Range-byte request succeeded
HiC file version: 7
Default settings for 5kb and 10kb being used
Running HiCCUPS for resolution 10000
GPU/CUDA Installation Not Detected
Exiting HiCCUPS
jcuda.CudaException: Failed to initialize the driver: CUDA_ERROR_NO_DEVICE
at jcuda.utils.KernelLauncher.initialize(KernelLauncher.java:606)
at jcuda.utils.KernelLauncher.<init>(KernelLauncher.java:586)
at jcuda.utils.KernelLauncher.create(KernelLauncher.java:393)
at jcuda.utils.KernelLauncher.create(KernelLauncher.java:321)
at jcuda.utils.KernelLauncher.compile(KernelLauncher.java:270)
at juicebox.tools.utils.juicer.hiccups.GPUController.<init>(GPUController.java:55)
at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:339)
at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:283)
at juicebox.tools.HiCTools.main(HiCTools.java:79)

However, when I run the script on its own, without submitting to the queue, it doesn't produce this error. CUDA is clearly being loaded, because "module load cuda/7.0" doesn't produce an error. There seems to be an issue in the interaction between juicebox_tools.jar and PBS.

[Machol, Ido]

HI Lila,

Looking at the output:

GPU/CUDA Installation Not Detected
Exiting HiCCUPS
jcuda.CudaException: Failed to initialize the driver: CUDA_ERROR_NO_DEVICE

It looks like your environment is not configured the same way through login and through queue submission. One thing you might want to try is adding 

#!/bin/bash -l

to beginning of your script. it might solve your problem.

For HICCUPS to work, it requires nvcc command accessible through your PATH variable. 

I suspect it is available via login, but not via queue submission.

Best,

Ido

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/b5926fa0-68b6-4d86-b923-9bad0a497f8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lila Rieber]

Here's the script I'm running now:

#!/bin/bash -l

 

#PBS -W umask=0007 
#PBS -l walltime=20:00:00 
#PBS -l nodes=1:ppn=4 
#PBS -j oe 
#PBS -l mem=40gb 

 

module load cuda/7.0 

nvcc 

/storage/home/lur159/group/lab/lila/java/jre1.8.0_91/bin/java -Djava.awt.headless=true -Djava.library.path=../../../juicebox/native_launcher/natives -Xmx16000m -Xms16000m -jar juicebox_tools.jar $* hiccups https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined.hic all_hiccups_loops 

That produces the following output:

nvcc fatal   : No input files specified; use option --help for more information

No configurations specified, using default settings
Reading file: https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined.hic

INFO [2016-07-05 14:59:38,164]  [DirectoryManager.java:149] [main]  IGV Directory: /storage/home/lur159/igv
INFO [2016-07-05 14:59:38,460]  [HttpUtils.java:809] [main]  Range-byte request succeeded

HiC file version: 7
Default settings for 5kb and 10kb being used
Running HiCCUPS for resolution 10000
GPU/CUDA Installation Not Detected
Exiting HiCCUPS
jcuda.CudaException: Failed to initialize the driver: CUDA_ERROR_NO_DEVICE
at jcuda.utils.KernelLauncher.initialize(KernelLauncher.java:606)
at jcuda.utils.KernelLauncher.<init>(KernelLauncher.java:586)
at jcuda.utils.KernelLauncher.create(KernelLauncher.java:393)
at jcuda.utils.KernelLauncher.create(KernelLauncher.java:321)
at jcuda.utils.KernelLauncher.compile(KernelLauncher.java:270)
at juicebox.tools.utils.juicer.hiccups.GPUController.<init>(GPUController.java:55)
at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:339)
at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:283)
at juicebox.tools.HiCTools.main(HiCTools.java:79)

 

So it is seeing CUDA/nvcc. 

[Muhammad Saad Shamim]

For future readers of this thread, the test script can be found at

https://github.com/sa501428/GPUTest

- Muhammad Saad Shamim

---------- Forwarded message ----------
From: Muhammad Saad Shamim 
Date: Tue, Jul 5, 2016 at 4:20 PM
Subject: Re: CUDA_ERROR_NO_DEVICE
To: Lila Rieber

​Sounds great! We basically just need to figure out the command to specify a node with a GPU. For Rice/SLURM, it's something like this

  #!/bin/bash -l

  #SBATCH --mem-per-cpu=2G

  #SBATCH --gres=gpu:kepler:1

  ...

  module load gcccuda/2.6.10;

  export PATH=\$PATH:/usr/local/cuda/bin/;

  ...

  (call HiCCUPS)​

Best,

- Muhammad Saad Shamim

On Tue, Jul 5, 2016 at 4:17 PM, Lila Rieber

​ 

wrote:

I've asked the cluster help desk, because it seems to be an issue with the cluster rather than juicebox_tools. Thank you for your help.

On Tue, Jul 5, 2016 at 5:08 PM, Lila Rieber wrote:

You're correct, it said that 0 devices were found.

On Tue, Jul 5, 2016 at 4:54 PM, Muhammad Saad Shamim wrote:

Tweaked test file attached. Can you recompile and run?

(Added a line to see how many devices found. My guess is 0, which would mean suggestion #2 needs to solved.)

- Muhammad Saad Shamim

On Tue, Jul 5, 2016 at 3:49 PM, Muhammad Saad Shamim wrote:

Can you try redirecting the output, maybe:

./test_cuda > result_test_cuda

- Muhammad Saad Shamim

On Tue, Jul 5, 2016 at 3:47 PM, Lila Rieber wrote:

It produced no output. No error but no device info. 

On Tue, Jul 5, 2016 at 4:38 PM, Muhammad Saad Shamim wrote:

*quick GPU detection test

- Muhammad Saad Shamim

On Tue, Jul 5, 2016 at 3:38 PM, Muhammad Saad Shamim wrote:

I've attached a quick CUDA test.

Can you run the following on PBS: 

nvcc -o test_cuda test_cuda.cu 

./test_cuda

It should print out something similar to

Device Number: 0

  Device name: GeForce GT 750M

  Memory Clock Rate (KHz): 2508000

  Memory Bus Width (bits): 128

  Peak Memory Bandwidth (GB/s): 80.256000

- Muhammad Saad Shamim

On Tue, Jul 5, 2016 at 3:22 PM, Muhammad Saad Shamim wrote:

Hey Lila,

I hope you are doing well!

A few things:

Are you using the latest 7.0 jar?

(http://aidenlab.org/commandlinetools/)

Most clusters have a special command/flag to actually request the GPU. Not sure what the #PBS flag to request this is, but it might be something to email the admin about. nvcc may be accessible, but the GPU may not have properly connected.

Can you also try using the absolute path for the native libs folder in the java line? (and these are the jcuda 7.0 libs, right?)

I'll put together a minimal JCUDA test script, but in the meantime, maybe one of the above (perhaps #2) may resolve the issue.

​Best,​

- Muhammad Saad Shamim

/storage/home/lur159/group/lab/lila/java/jre1.8.0_91/bin/java -Djava.awt.headless=true -Djava.library.path=../../../juicebox/native_launcher/natives -Xmx16000m -Xms16000m -jar juicebox_tools.jar $* hiccupshttps://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined.hic all_hiccups_loops 

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/f171f86c-210b-4fb2-95f4-5d69b340ab56%40googlegroups.com.