HiCCUPS "CUDA_ERROR_OUT_OF_MEMORY" with juicer_tools versions above 1.9.9

[Steve S]

I am running on CentOS 7 with an older Tesla M2090 GPU. I am using juicer 1.5.7 downloaded from master git source on May 20, 2019.

I was originally using CUDA 7.5 with juicer_tools.1.6.2_jcuda.0.7.5.jar and things worked fine when I ran the following test command:

$ java -Djava.library.path=/home/programs/apps/JCuda-All-0.7.5-bin-Linux-x86_64  -jar /home/programs/apps/juicer_tools.1.6.2_jcuda.0.7.5.jar  hiccups -m 1024 -r 5000,10000 -c 22 https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined_30.hic hiccups_results

After successfully running with real inhouse datasets through the juicer pipeline, I started having problems with hiccups crashing on a few larger data sets.

Rather than troubleshoot with a version that I knew to be older, I decided to upgrade to CUDA 8.0 and use the newer juicer_tools.jar versions to see if that solved the problem.

Using versions up to juicer_tools.1.9.9_jcuda.0.8.jar, I got past the errors I previously saw and hiccups completed.

However, now whenever I run the test command with versions of juicer_tools above juicer_tools.1.9.9_jcuda.0.8.jar, I get the following error:

$ java -Djava.library.path=/home/programs/apps/JCuda-All-0.8.0-bin-Linux-x86_64 -jar /home/programs/apps/juicer_tools_1.12.03.jar  hiccups -m 1024 -r 5000,10000 -c 22 https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined_30.hic hiccups_results
WARN [2019-08-21T12:21:20,572]  [Globals.java:138] [main]  Development mode is enabled
Reading file: https://hicfiles.s3.amazonaws.com/hiseq/gm12878/in-situ/combined_30.hic
INFO [2019-08-21T12:21:22,140]  [DirectoryManager.java:179] [main]  IGV Directory: /home/sstelman/igv
INFO [2019-08-21T12:21:22,550]  [HttpUtils.java:937] [main]  Range-byte request succeeded
Using the following configurations for HiCCUPS:
Config res: 10000 peak: 2 window: 5 fdr: 10% radius: 20000
Config res: 5000 peak: 4 window: 7 fdr: 10% radius: 20000
Using 32 CPU threads
Running HiCCUPS for resolution 10000
4%
9%
14%
19%
23%
28%
33%
38%
42%
47%
52%
57%
61%
66%
71%
76%
80%
85%
90%
95%
100%
jcuda.CudaException: Failed to create a context: CUDA_ERROR_OUT_OF_MEMORY
        at jcuda.utils.KernelLauncher.createContext(KernelLauncher.java:650)
        at jcuda.utils.KernelLauncher.initialize(KernelLauncher.java:626)
        at jcuda.utils.KernelLauncher.<init>(KernelLauncher.java:586)
        at jcuda.utils.KernelLauncher.create(KernelLauncher.java:393)
        at jcuda.utils.KernelLauncher.create(KernelLauncher.java:321)
        at jcuda.utils.KernelLauncher.compile(KernelLauncher.java:270)
        at juicebox.tools.utils.juicer.hiccups.GPUController.<init>(GPUController.java:70)
        at juicebox.tools.clt.juicer.HiCCUPS.buildGPUController(HiCCUPS.java:625)
        at juicebox.tools.clt.juicer.HiCCUPS.access$000(HiCCUPS.java:151)
        at juicebox.tools.clt.juicer.HiCCUPS$1.run(HiCCUPS.java:501)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

I do notice when monitoring with "nvidia-smi -l 1" that "GPU Memory Usage" is much higher with the newer versions of juicer_tools, which may be part of the root of the issue.

Any guideance or help in troubleshooting would be appreciated.

[Muhammad Saad Shamim]

Hey Steve,

Can you try version 1.13.01?

We multithreaded several tools starting with version 1.9.9.

The latest version has optimized the memory consumption a bit more.

https://github.com/aidenlab/juicer/wiki/Download#version-11301

You can also run a single-threaded version by passing in the flag --threads 1.

Let us know if this resolves the issue.

Best,

- Muhammad Saad Shamim

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/f415e091-413c-4e69-a67b-3aa0521966fa%40googlegroups.com.

[Steve S]

Hi Muhammad,

The new 1.13.01 version gave the same error. However, the addition of "--threads 1" allowed successful completion of the test hiccups command.

Do you suggest running that way? Can that be passed to juicer somehow, or is there a script that requires an edit?

Also, it it reasonable to just re-run juicer now with "-S postproc" using the new version of juicer_tools, or should I expect to have to re-run any of the prior steps (before arrowhead and hiccups) also?

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-ge...@googlegroups.com.

[Muhammad Saad Shamim]

That's a good point - I've changed the default to now be one thread rather than the number of available threads.

https://github.com/aidenlab/juicer/wiki/Download#version-11302

You should be able to replace the juicer_tools jar with the latest version 1.13.02 (renaming required either in the script or of the file once downloaded).

Running with -S postproc should be fine.

No additional flag or change to the juicer script should be required.

- Muhammad Saad Shamim

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/215fc150-f020-4ba4-a9e9-514e593785ce%40googlegroups.com.

[Steve S]

Thanks Muhammad. Is it a known issue that the progress reporting in hiccups now prints newlines even when the percent complete hasn't incremented? It fills the screen trace buffer in my terminal and makes it difficult to follow progress.

Example:

11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%
11%

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/215fc150-f020-4ba4-a9e9-514e593785ce%40googlegroups.com.