CUDA_ERROR_ILLEGAL_ADDRESS

[Muhammad Shamim]

We've seen this a few times now, so starting a new thread for it.

Instance 1-----------------------------------------------------------------------------------------

---------- Forwarded message ----------

From: Muhammad Saad Shamim 

Date: Wed, Jul 29, 2015 at 2:14 PM

Subject: Re: Running HICCUPs

To: Neva Cherniavsky Durand 

​This appears to be an issue with large submatrices.

Jcuda is unable to handle that size for memory allocation so I need to override some GPU setting.

But it should work fine for now with -m 1000

- Muhammad Saad Shamim

On Wed, Jul 29, 2015 at 1:35 PM, Neva Cherniavsky Durand wrote:

Now I get a runtime error

bash juicebox.sh hiccups -c chrX -r 5000 -m 5000 ../[input].hic out1 out2

Using Matrix Size 5000

Accessing ../[input].hic

HiC file version: 8

Selected KR Normalization

Time to load chr chrX matrix: 14ms

CUDA_ERROR_ILLEGAL_ADDRESS

jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:120)

at juicebox.tools.clt.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:223)

at juicebox.tools.clt.HiCCUPS.run(HiCCUPS.java:146)

at juicebox.tools.HiCTools.main(HiCTools.java:68)

Instance 2----------------------------------------------------------------------------

https://groups.google.com/forum/#!searchin/3d-genomics/CUDA/3d-genomics/QpBJanRDrfw/N3ANPReGAQAJ

Instance 3----------------------------------------------------------------------------

On Wed, Jun 8, 2016 at 5:13 PM, XYZ wrote:

I think the error is system specific but can post about selecting the gpu to the forum if that’s what you mean?

On Wed, Jun 8, 2016 at 5:09 PM, Muhammad Saad Shamim wrote:

Cursory google search was not helpful, but I've emailed the jcuda folk regarding the specifying GPU question. 

On Wed, Jun 8, 2016 at 5:02 PM, Muhammad Saad Shamim wrote:

Ah, I see.

Hiccups is single precision (floats).

This would be something that would need to be specified in the code when the jcuda library is called. Or perhaps there's an easier way to specify the default via nvcc/cuda (which will then be called by jcuda).

On Wed, Jun 8, 2016 at 4:46 PM, XYZ wrote:

Hi Muhammad,

Ok, I switched to another machine and have hiccups running with -m 8192 so my problem must have been system specific.

Is hiccups single or double precision?  Also, how can I tell hiccups which GPU to use?  Is this in the source code?

We have a couple of GPUs on the machine it’s running on and I want to switch to a GPU with more memory so I can use bigger submatrices.

Thanks.

Best,

XYZ

On Jun 8, 2016, at 11:17 AM, XYZ wrote:

Hi Muhammad,

Ok thanks.

Here’s a link to the hic file: [removed]

Let me know if it doesn’t work.

I’ll try running on a single chromosome now.

Best,

XYZ

On Jun 8, 2016, at 11:11 AM, Muhammad Saad Shamim wrote:

Hey XYZ,

Hope you are doing well.

That seems strange - would you be able to send me a link to the hic file so I can try debugging on one of our machines?

In the meantime, you can get the optimization to run faster by specifying a single chromosome. (-c chr_name)

This will change the thresholds a little bit vs running for all chromosomes, but will save a lot of time while trying different parameters.

- Muhammad Saad Shamim

On Wed, Jun 8, 2016 at 1:03 PM, XYZ wrote:

Hi Muhammad,

I got another large data dump and my maps are now at high resolution. The issue I’m running into now with hiccups is for these larger maps it is giving an error that is dependent on the submatrix size.

If I try to run:

${juiceDir}/scripts/juicebox hiccups [input].hic -m 512 -k KR [output]

I get:

Reading file: .........hic

HiC file version: 8

Running HiCCUPS for resolution

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:398)

at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:277)

at juicebox.tools.HiCTools.main(HiCTools.java:78)

Changing -m to 256 works, but the computation is taking about 10x longer so it is becoming hard to efficiently optimize parameters.  I can run -m 512 on my lower res maps no problem, and the documentation says with a dedicated GPU (I’ve tried a Tesla K80 and Titan X) I should be able to go up to 2048, but can’t get above 512 for these hi-res maps.  I’ve tried both CUDA 7.0 and CUDA 7.5 on our cluster and same problems.

I’m not familiar with CUDA so am having a hard time debugging this.  Can you please help?

Thanks.

Best,

XYZ

Instance 4-----------------------------------------------------------------------------

https://groups.google.com/forum/#!topic/3d-genomics/-fotKItfmfQ

--------------------------------------------------------------------------------------------------------------

Some thoughts:

I think this bug is occurring when a matrix is passed to the GPU, but exceeds the GPU's available memory/thread/RAM. Hence the illegal memory address error. JCuda/CUDA by default will use device 0 (i.e. the first GPU detected) on a device with >= 1 GPU. Perhaps the matrix may be too large if the GPU is also being used by some other process (maybe JCuda is using the CUDA multi-process server)? But then when no one else is using that GPU or HiCCUPS is called with a smaller sub-matrix, everything runs fine.

[Yunjiang Qiu]

Hi Muhammad,

I tried with -m 1024, but it is still not working. What's weird is that I only get the error when starting to run on 10Kb matrices, while it is totally fine to run with 5Kb.

Any thoughts?

Thank you very much!

Best,

Yunjiang

[Yunjiang Qiu]

So what is the solution? Simply increase -m?

On Monday, July 11, 2016 at 12:13:23 PM UTC-7, Muhammad Shamim wrote:

[Muhammad Saad Shamim]

​Usually decreasing -m if the GPU is also being used by other processes or doesn't have enough memory for the larger submatrix size. Or trying with a different GPU instance if your cluster uses multiple GPU nodes.​

- Muhammad Saad Shamim

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/8a8f2906-97ea-4263-9f74-4b94a897d211%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Yunjiang Qiu]

Hi Muhanmad,

I tried to lower -m to 256 but it is still not working. Do you have some ideas?

Thanks,

Yunjiang

[Muhammad Saad Shamim]

Hey Yunjiang,

Can you remind me again of what you've tried?

IIRC, HiCCUPS works fine with -m 1024 for some resolutions, but not others.

Is that right?

If so, I think there may be an issue with the hic map rather than HiCCUPS.

Which resolutions fail to run?

​Best,​

- Muhammad Saad Shamim

[Yunjiang Qiu]

I tried to use -m 512 or -m 256, and neither works.

Here is the error message. What’s weird is that 5kb is fine, but 10kb is not.

Reading file: test.hic

HiC file version: 8

Using the following configurations for HiCCUPS:

Config res: 5000 peak: 4 window: 7 fdr: 10% radius: 20000

Config res: 10000 peak: 2 window: 5 fdr: 10% radius: 20000

Unable to assess map sparsity; continuing with HiCCUPS

Running HiCCUPS for resolution 5000

50%

100%

Running HiCCUPS for resolution 10000

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

        at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

        at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

        at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

        at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:476)

        at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:353)

        at juicebox.tools.HiCTools.main(HiCTools.java:77)

[Michael Johnston]

I am having a peculiar error with HiCCUPS that results in this error message as well.

I have three different hic files that I am annotating with the command:

juicer_tools hiccups -r 5000,10000,25000 /some/path/inter_30.hic /some/path/inter_30_loops.txt

HiCCUPS works at all three resolutions for two of the three files.

For one file, it consistently fails at 25 kb resolution, and always ~3% into processing the file.

Running HiCCUPS for resolution 25000

0% 

1% 

2% 

3% 

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:484)

at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:361)

at juicebox.tools.HiCTools.main(HiCTools.java:86)

I have tried adjusting memory with -m of 96, 256, 512, and 1024, but the error is always the same.

Any guess as to why HiCCUPS might fail at a particular resolution and file location each time?

Thanks,

- Michael

[Muhammad Saad Shamim]

Hey Michael,

Can you confirm that the hic map at 25kb has working normalizations in juicebox and that it's not an issue with the hi-c file?

Is HiCCUPS able to run if you switch normalizations from KR to VC or VC_SQRT?

- Muhammad Saad Shamim

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/b05653af-d41b-456a-8d37-1b979b43ed58%40googlegroups.com.

[Michael Johnston]

I can see the data in juicebox at 25 kb resolution.

This works when displaying normalizations:
none, coverage, coverage (sqrt), or balanced

I ran HiCCUPS with each normalization.

VC and VC_SQRT worked.

NONE did not.

Data not available for chr1 at 25000 resolution

0% 

Data not available for chr2 at 25000 resolution

0% 

Data not available for chr3 at 25000 resolution

0% 

Data not available for chr4 at 25000 resolution

0% 

Data not available for chr5 at 25000 resolution

1% 

Data not available for chr6 at 25000 resolution

1% 

...

I am not sure why I am able to visualize the data in juicebox if this information cannot be found in the hic file.

- Michael

[Neva Durand]

Could you trying running in verbose mode?  This is a strange error.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAMfeVFskhFR8s28G9bZnt-RArbn-HHZZMBnYp%2BvG4XMTzSV94w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

[Michael Johnston]

Hi Neva and Muhammad,

My apologies for the delay. I have rerun hiccups with -v.

I additionally went back and reran juicer pre, just in case there was something wrong with the original .hic file. (The only other change in between was to modify the chrom.sizes file to change which chromosomes would be displayed.)

Hiccups fails to run on either the "new" or "old" hic file. 

The error is specific to this data set; two other data sets run the same commands with no problem. 

The error is also specific to the 25-kb resolution; 5-kb and 10-kb are fine. 

If hiccups is rerun multiple times, each files consistently fails at a file-specific percentage completion; new @ 32% and old @ 3%. Please find the logs files attached.

I am including one additional log file (postproc) from a larger script where I was running arrowhead and hiccups at multiple resolutions on the new hic file. In this log, the sample fails at the same 32% completion, but threw a CUDA_ERROR_LAUNCH_FAILED instead of CUDA_ERROR_ILLEGAL_ADDRESS, although I'm not sure why there would have been a difference.

I hope the logs can help track this down!

Thanks for your help,

- Michael

[Muhammad Saad Shamim]

Hey Michael,

This is definitely a strange bug.

Are you running on a cluster where some nodes don't have GPUs or is this on a server/computer with dedicated GPUs?

If the latter, are you running HiCCUPS in parallel with screen?

Also, can you dump the kr norm vector for the chromosomes at 25 kb and see if any values are abnormally high or low? (ignoring NaNs)

https://github.com/theaidenlab/juicer/wiki/Data-Extraction

- Muhammad Saad Shamim

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAMfeVFv9u%2B185N7bt2cSWsM%3D9mOtGqK8bd-o9nTKejmLwF%3DZ%2Bg%40mail.gmail.com.

[Michael Johnston]

Hi Muhammad and Neva,

This is being run on a cluster where certain nodes do not have GPUs, but these HiCCUPS jobs are being specifically dispatched to the node with a dedicated GPU available. All of my samples were processed on this same node.

I am not running HiCCUPS in parallel with screen. I submit the job to the queue (via Torque / PBS / qsub), and then each of my samples are run serially on the same node. 

I have dumped the KR norm values for the failing sample in question. 

cat dump_norm_KR_all_failingSample_25kb.txt | sort -nr | head -n 10

25000 123541 9958

753.8636289752404

235.8111925451782

117.30099929738184

84.20076964655459

67.84595726506149

57.945373585599086

51.08230155185938

45.85290630630369

41.9214394649931

And here is the dump for a successful sample. 

cat dump_norm_KR_all_succeedingSample_25kb.txt | sort -nr | head -n 10

25000 123541 9958

1354.9015987389598

443.88122704481265

298.35557074398537

281.7763080620471

280.10165150834933

273.1941234500391

272.95292651890986

253.55091973072618

252.85958188433014

Overall, I think that the failing samples values seem lower, but I can't tell much more than that. I have attached the dumps in case that helps troubleshoot.

My apologies for the delay.

I went back and reanalyzed the data from scratch in case there was any sort of file corruption along the way, but I am still getting the same error from the same dataset, specifically at 25kb resolution.

Please let me know if you might have any ideas.

Thank you again.

- Michael

--
You received this message because you are subscribed to a topic in the Google Groups "3D Genomics" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/3d-genomics/PYWXfjbPr24/unsubscribe.
To unsubscribe from this group and all its topics, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAFrDNA%3DCctvWQ9awwGpcRMVe7fFykjwt%2B0xVMcn0UP7k4C9FSw%40mail.gmail.com.

[Michael Johnston]

I am running into the same error with the GM12878 hg38 data link, except at 10kb res.

I am using:

juicer_tools.1.6.2_jcuda.0.7.5.jar

I redownloaded this version of juicer_tools today and confirmed that the md5sum is the same as the one I have been using.

Command + output:

path/to/juicer_tools hiccups -m 2048 -r 5000,10000,25000,50000,100000 -f 0.1,0.1,0.1,0.1,0.1 -p 4,2,1,2,1 -i 7,5,3,4,2 -d 20000,20000,50000,100000,200000 https://hicfiles.s3.amazonaws.com/hiseq/gm12878/combined_30.hic /path/to/output/inter_30_loops

Reading file: https://hicfiles.s3.amazonaws.com/hiseq/gm12878/combined_30.hic

INFO [2017-12-29 12:47:26,585]  [HttpUtils.java:833] [main]  Range-byte request succeeded

HiC file version: 8

Using the following configurations for HiCCUPS:

Config res: 10000 peak: 2 window: 5 fdr: 10% radius: 20000

Config res: 50000 peak: 2 window: 4 fdr: 10% radius: 100000

Config res: 100000 peak: 1 window: 2 fdr: 10% radius: 200000

Config res: 5000 peak: 4 window: 7 fdr: 10% radius: 20000

Config res: 25000 peak: 1 window: 3 fdr: 10% radius: 50000

Running HiCCUPS for resolution 10000

2% 

4% 

6% 

8% 

10% 

Data not available for 5 at 10000 resolution

12% 

14% 

16% 

18% 

20% 

22% 

24% 

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:481)

at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:359)

at juicebox.tools.HiCTools.main(HiCTools.java:97)

Thoughts?

Thanks!

- Michael

[Michael Johnston]

Hi,

Are you able to reproduce this error using the GM12878 hg38 data link? 

This resolution-specific error is causing issues for me, as I can't compare loops between different samples that fail at different resolutions.

Thanks again,

- Michael

[Ittai Eres]

Hello,

I am running into this same error with some of my data at 10kb resolution. Strangely, I managed to analyze 7 other individuals with hiccups perfectly fine, but this 8th individual always throws me this error. I've tried using the jars for both cuda 7.5 and cuda 8.0, to no avail--the same error results:

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_ILLEGAL_ADDRESS

at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:481)

at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:359)

at juicebox.tools.HiCTools.main(HiCTools.java:98)

The command I'm running is:

~/path/to/juicer_tools hiccups -r 10000 -k KR /path/to/inter_30.hic /path/to/output

And I am using a dedicated GPU on a cluster. Specific node info:

Memory: 56G

CPUs: 28 x Intel E5-2680v4 @ 2.4 GHz

Other info: 4X Nvidia K80 GPU

I've tried running it on other GPUs on our cluster and instead get a GLIBC_2.14 error, which I thought I might be able to fix by natively compiling jcuda within the GPU but that didn't solve it either. Any ideas on a workaround or how to fix this would be much appreciated! 

Thanks,

Ittai Eres

- Muhammad Saad Shamim

- Muhammad Saad Shamim

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/b05653af-d41b-456a-8d37-1b979b43ed58%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAMfeVFskhFR8s28G9bZnt-RArbn-HHZZMBnYp%2BvG4XMTzSV94w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAMfeVFv9u%2B185N7bt2cSWsM%3D9mOtGqK8bd-o9nTKejmLwF%3DZ%2Bg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "3D Genomics" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/3d-genomics/PYWXfjbPr24/unsubscribe.

To unsubscribe from this group and all its topics, send an email to 3d-genomics...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/CAFrDNA%3DCctvWQ9awwGpcRMVe7fFykjwt%2B0xVMcn0UP7k4C9FSw%40mail.gmail.com.

[Ittai Eres]

Just a quick update--I went back and made the hiccups call on a chromosome-by-chromosome basis for this individual, and discovered that all the chromosomes actually work appropriately without this error except for chr16. I have no idea why that may be, however...happy to provide more information on my data on that chromosome to help troubleshoot if it would be useful.

Cheers,

Ittai

[Ittai Eres]

I ran an interactive session on the cluster GPU described above utilizing cuda-memcheck before my call to hiccups. I've provided the error output as well as the call I made below to help with troubleshooting; like I said before I'm also happy to send along my data on this chromosome if that would be useful.

Call: cuda-memcheck ~/path/to/juicer_tools hiccups -m 2048 -c 16 -r 10000 -k KR /path/to/inter_30.hic /path/to/output

Error output:

========= CUDA-MEMCHECK

Reading file: /project2/gilad/ittai/HiC/F-28834/juicer/aligned/inter_30.hic

HiC file version: 8

Using the following configurations for HiCCUPS:

Config res: 10000 peak: 2 window: 5 fdr: 10% radius: 20000

Running HiCCUPS for resolution 10000

========= Invalid __global__ read of size 4

=========     at 0x00003628 in BasicPeakCallingKernel

=========     by thread (3,10,0) in block (98,97,0)

=========     Address 0x2b061c01fc is out of bounds

=========     Saved host backtrace up to driver entry point at kernel launch time

=========     Host Frame:/lib64/libcuda.so.1 (cuLaunchKernel + 0x2cd) [0x23c07d]

=========     Host Frame:/tmp/libJCudaDriver-0.8.0RC-linux-x86_64.so (Java_jcuda_driver_JCudaDriver_cuLaunchKernelNative + 0x1a4) [0x1bb56]

=========     Host Frame:[0x7f55550156d4]

=========

========= Program hit CUDA_ERROR_LAUNCH_FAILED (error 719) due to "unspecified launch failure" on CUDA API call to cuCtxSynchronize. 

=========     Saved host backtrace up to driver entry point at error

=========     Host Frame:/lib64/libcuda.so.1 (cuCtxSynchronize + 0x12d) [0x21ce5d]

=========     Host Frame:/tmp/libJCudaDriver-0.8.0RC-linux-x86_64.so (Java_jcuda_driver_JCudaDriver_cuCtxSynchronizeNative + 0x2b) [0x172a3]

=========     Host Frame:[0x7f55550156d4]

=========

Exception in thread "main" jcuda.CudaException: CUDA_ERROR_LAUNCH_FAILED

        at jcuda.utils.KernelLauncher.checkResult(KernelLauncher.java:1044)

        at jcuda.utils.KernelLauncher.call(KernelLauncher.java:1030)

        at juicebox.tools.utils.juicer.hiccups.GPUController.process(GPUController.java:122)

        at juicebox.tools.clt.juicer.HiCCUPS.runHiccupsProcessing(HiCCUPS.java:484)

        at juicebox.tools.clt.juicer.HiCCUPS.run(HiCCUPS.java:361)

        at juicebox.tools.HiCTools.main(HiCTools.java:86)

========= ERROR SUMMARY: 2 errors

Many thanks,

Ittai 

[David Weisz]

This bug was fixed in the latest version of Juicer tools that can be downloaded from: http://hicfiles.tc4ga.com.s3.amazonaws.com/public/juicer/juicer_tools.1.8.9_jcuda.0.8.jar

David.