error during creation of .hic file during normalisation

[Raoul RAFFEL]

Hi,

I have problem to generate proper .hic file from Hi-C experiment with juicer pipeline (on SLURM cluster).

Here the output of suspected problem:

hic-246633.out

::::::::::::::

ven. nov.  3 07:10:13 CET 2017

Start preprocess

Writing header

Writing body

............................................................................................................................................................................................................

..........................................................................................................................

Writing footer

Finished preprocess

HiC file version: 8

ERROR [2017-11-03 09:33:01,187]  [DatasetReaderV2.java:236] [main]  Error reading dataset

java.io.EOFException

at htsjdk.tribble.util.LittleEndianInputStream.readString(LittleEndianInputStream.java:119)

at juicebox.data.DatasetReaderV2.read(DatasetReaderV2.java:161)

at juicebox.tools.utils.original.NormalizationVectorUpdater.updateHicFile(NormalizationVectorUpdater.java:68)

at juicebox.tools.utils.original.NormalizationVectorUpdater.updateHicFile(NormalizationVectorUpdater.java:63)

at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:113)

at juicebox.tools.HiCTools.main(HiCTools.java:86)

/home/raffelr/IGH/tools/Juicer//scripts/juicer_tools48g pre -f /home/raffelr/IGH/tools/Juicer/restriction_sites/hg19_DpnII.txt -s /home/raffelr/scratch/Hi-C/CD4_active1/aligned/inter.txt -g /home/raffelr/

scratch/Hi-C/CD4_active1/aligned/inter_hists.m -q 1 /home/raffelr/scratch/Hi-C/CD4_active1/aligned/merged_nodups.txt /home/raffelr/scratch/Hi-C/CD4_active1/aligned/inter.hic /home/raffelr/IGH/bioDB/hg19/g

enome/Homo_sapiens.GRCh37.75.dna_sm.fa.chrom.size 

ven. nov.  3 09:33:02 CET 2017

I succed only with one experiment (on 4) and I don't understant why the other ones fail.

here the output of succed experiment:

hic-243747.out

::::::::::::::

Sat Oct 28 01:32:09 CEST 2017

Start preprocess

Writing header

Writing body

............................................................................................................................................................................................................

..........................................................................................................................

Writing footer

Finished preprocess

HiC file version: 8

Calculating norms for zoom BP_2500000

Calculating norms for zoom BP_1000000

Calculating norms for zoom BP_500000

Calculating norms for zoom BP_250000

Calculating norms for zoom BP_100000

Calculating norms for zoom BP_50000

Calculating norms for zoom BP_25000

Calculating norms for zoom BP_10000

Calculating norms for zoom BP_5000

Calculating norms for zoom FRAG_500

Calculating norms for zoom FRAG_200

Calculating norms for zoom FRAG_100

Calculating norms for zoom FRAG_50

Calculating norms for zoom FRAG_20

Calculating norms for zoom FRAG_5

Calculating norms for zoom FRAG_2

Calculating norms for zoom FRAG_1

Writing expected

Writing norms

Finished writing norms

/home/raffelr/IGH/tools/Juicer//scripts/juicer_tools48g pre -f /home/raffelr/IGH/tools/Juicer/restriction_sites/hg19_DpnII.txt -s /home/raffelr/scratch/Hi-C/HiC1/aligned/inter.txt -g /home/raffelr/scratch

/Hi-C/HiC1/aligned/inter_hists.m -q 1 /home/raffelr/scratch/Hi-C/HiC1/aligned/merged_nodups.txt /home/raffelr/scratch/Hi-C/HiC1/aligned/inter.hic /home/raffelr/IGH/bioDB/hg19/genome/Homo_sapiens.GRCh37.75

.dna_sm.fa.chrom.size 

Do you have any idea ?

Thanks

Cheers,

Raoul RAFFEL

[Neva Durand]

Hello,

That’s a strange error. Are you able to load the hic files in Juicebox?

Best

Neva

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/8e265e24-24be-4d41-9fe6-6c9d4a93b9c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

[Raoul RAFFEL]

Hi,

unfortunately not...

Best,

Raoul

[Neva Durand]

I would try to run those commands again. It looks like there was an error at the end of hic file creation but since it’s inconsistent it is difficult for me to debug. You could try increasing memory or run time in case the cluster ended your job. The other times I’ve seen errors like this, it’s been a filesystem problem. 

Best

Neva

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/12eab330-26a6-473c-9a4c-bb8bb42ec082%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Raoul RAFFEL]

Hi,

I increased memory in the script juicer_tools48g:

java -Djava.awt.headless=true -Djava.library.path=`dirname $0`/lib64 --Xms52480m --Xmx52480m -jar `dirname $0`/juicer_tools.jar $*

and in juicer.sh

#SBATCH --mem-per-cpu=56G  (I've checked RAM with htop on cluster)

and the issue has been solved.

I also checked the memory during the process and I think that there was a lack of very small amount of RAM (maybe less than 1G or 2G) so you should maybe add a little more RAM to juicer.sh (for scheduler) and to juicer_tools48g to prevent this issue for others users.

Many thanks

Cheers,

Raoul RAFFEL

[Raoul RAFFEL]

Hi,

I have still some trouble with one of my experiment. I increased memory up to 100G but I have the same issue with same error message than before (and RAM never increase more than 50G). 

I also tried to use -n option, in this way there is no issue during the processing (no error message) but the .hic cannot be loaded under juicebox.

I also tried to generate .hic file in my server but the issue is still the same (to be sure about the time of processing).

What is weird is that all other files have succed (except one because of few more amount of RAM needed to pass final step).

Do you have any idea ?

Thanks,

Cheers,

Raoul RAFFEL