hic file generation with very large file

[Adan Horta]

Hi Neva,

I've run into a problem with generating a .hic for a very deep sequencing library. My merged_nodups.txt file is 1.1TB (>6 billion reads). The SLURM pipeline works great with smaller libraries but fails during the "Writing body" step and generates the error file below. I am using juicer_tools.1.7.6_jcuda.0.8.jar with 23TB of free space.

Have you ever encountered this issue for very large libraries? 

java.io.IOException: No space left on device

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:326)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at juicebox.tools.utils.original.Preprocessor$MatrixZoomDataPP.dumpBlocks(Preprocessor.java:1444)

at juicebox.tools.utils.original.Preprocessor$MatrixZoomDataPP.incrementCount(Preprocessor.java:1392)

at juicebox.tools.utils.original.Preprocessor$MatrixPP.incrementCount(Preprocessor.java:1211)

at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:447)

at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:286)

at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:105)

at juicebox.tools.HiCTools.main(HiCTools.java:98)

Suppressed: java.io.IOException: No space left on device

at java.io.FileOutputStream.writeBytes(Native Method)

at java.io.FileOutputStream.write(FileOutputStream.java:326)

at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)

at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)

at java.io.FilterOutputStream.close(FilterOutputStream.java:158)

at java.io.FilterOutputStream.close(FilterOutputStream.java:159)

... 7 more

Model	HP Enterprise XL170r
CPU	E5-2650v4
Number of CPUs	2
Cores per CPU	12
Total Cores	24
Memory	512 GB
Network	EDR Infiniband

Thank you,

Adan

 

[Neva Durand]

Hello,

Try setting the temporary directory via

 -t <tmpDir> Set a temporary directory for writing

It's possible you are running out of space on /var/tmp or whatever your system temp directory is and not on your home directory.

Best

Neva

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/3b4eefe1-45a0-4e34-a4dd-db01fa6f7a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

[Adan Horta]

Including the -t <tmpDir> flag in juicer_tools pre worked! Thanks Neva