Juicer on AWS

Richard Gill

unread,

Jan 24, 2018, 8:24:05 PM1/24/18

to 3D Genomics

Hello,

I launched an AWS instance using the Juicer AMI, and followed your documentation to set things up.

When I try to run /opt/juicer/scripts/juicer.sh on the provided sample data, I get these messages:

(-: Looking for fastq files...fastq files exist

LSF daemon (LIM) not responding ... still trying

Please advise.

Thank you!

Richard

Neva Durand

unread,

Jan 24, 2018, 9:46:36 PM1/24/18

to Richard Gill, 3D Genomics

Check your IP. Follow the “first time tweaks” here:

https://github.com/theaidenlab/juicer/wiki/Running-Juicer-on-Amazon-Web-Services#first-time-tweaks

Also the -p flag takes in a file, not a directory. To send in the directory structure for scripts and references, use the -D flag

On Wed, Jan 24, 2018 at 6:35 PM Richard Gill <richar...@gmail.com> wrote:

Hello,

I have set up an AWS instance with the Juicer AMI according to your documentation, and am having trouble getting it to work on the sample files.

For example, from within the directory /opt/juicer/work/HIC003,

I typed:
source /opt/juicer/scripts/juicer.sh -d /opt/juicer/work/HIC003/ -p /opt/juicer/references/

And got these messages:

(-: Looking for fastq files...fastq files exist
LSF daemon (LIM) not responding ... still trying
LSF daemon (LIM) not responding ... still trying
LSF daemon (LIM) not responding ... still trying

Please advise.

Thank you!

Richard

--
You received this message because you are subscribed to the Google Groups "3D Genomics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/c39bacdb-a0e8-4691-8437-7cdea7029c88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Neva Cherniavsky Durand, Ph.D.

Staff Scientist, Aiden Lab

www.aidenlab.org

Richard Gill

unread,

Jan 29, 2018, 2:47:30 PM1/29/18

to 3D Genomics

1) After a day of running the alignment steps, Juicer stalled at a postproc job.

2) I followed the steps provided on the website, killing the jobs and then running these commands:

cat aligned/a1516851053_msplit*_optdups.txt > aligned/opt_dups.txt; cat aligned/a1516851053_msplit*_dups.txt > aligned/dups.txt;cat aligned/a1516851053_msplit*_merged_nodups.txt > aligned/merged_nodups.txt;

juicer.sh -S final

3) Issuing those commands caused two new commands (stats and hic30) to run:

4) But once those commands finished running, Juicer was stalled again, this time at a done step:

5) I tried killing jobs and once again running juicer.sh -S final, which started stats and hic30 jobs with different prefixes:

6) But once they finished, Juicer was once again stalled at the same done step: Inline image 8

Please advise.

Thanks,

Richard

Richard Gill

unread,

Feb 1, 2018, 10:09:00 AM2/1/18

to 3D Genomics

Hi,

Just to clarify my earlier post, the following are the results of typing bjobs -w after Juicer stalled on AWS. I move more than enough (>5TB) of EBS space available and more than enough memory was available (and most of it not used) on instances such as r4.16xlarge, x1.16xlarge, and x1e.32xlarge, all of which led to the same stalling.

1) After a day of running the alignment steps, Juicer stalled at a postproc job.

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME

9504 ubuntu PEND long HEADNODE - a1516851053_postproc Jan 25 03:50

9505 ubuntu PEND normal HEADNODE - a1516851053_done Jan 25 03:50

9859 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge1 Jan 25 15:18

9860 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge Jan 25 15:18

9861 ubuntu PEND Clean3 HEADNODE - a1516885367_clean1 Jan 25 15:18

9862 ubuntu PEND Clean3 HEADNODE - a1516885367_osplit Jan 25 15:18

9863 ubuntu PEND Stat HEADNODE - a1516885367_clean2 Jan 25 15:18

9864 ubuntu PEND Stat HEADNODE - a1516885367_launch Jan 25 15:18

9865 ubuntu PEND Stat HEADNODE - a1516885367_postproc_wrap Jan 25 15:18

9866 ubuntu PEND CleanFnl HEADNODE - a1516885367_prep_done Jan 25 15:18

9867 ubuntu PEND Stat HEADNODE - #!/bin/bash; #BSUB -q Stat; #BSUB -W 1200; #BSUB -o /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.out ; #BSUB -e /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.err; #BSUB -w " done(a151688 Jan 25 15:18

2) Since it stalled, I followed the steps provided on the website, killing the jobs and then running these commands:

cat aligned/a1516851053_msplit*_optdups.txt > aligned/opt_dups.txt; cat aligned/a1516851053_msplit*_dups.txt > aligned/dups.txt;cat aligned/a1516851053_msplit*_merged_nodups.txt > aligned/merged_nodups.txt;
juicer.sh -S final

(-: Looking for fastq files...fastq files exist

Job <12850> is submitted to queue <normal>.

Job <12851> is submitted to queue <Stat>.

Job <12852> is submitted to queue <Stat>.

Job <12853> is submitted to queue <CleanFnl>.

Job <12854> is submitted to queue <Stat>.

(-: Finished adding all jobs... please wait while processing.

3) Issuing those commands caused two new commands (stats and hic30) to run:

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME

12855 ubuntu RUN long HEADNODE HEADNODE a1517075571_stats Jan 27 17:52

12857 ubuntu RUN long HEADNODE HEADNODE a1517075571_hic30 Jan 27 17:52

9504 ubuntu PEND long HEADNODE - a1516851053_postproc Jan 25 03:50

9505 ubuntu PEND normal HEADNODE - a1516851053_done Jan 25 03:50

9859 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge1 Jan 25 15:18

9860 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge Jan 25 15:18

9861 ubuntu PEND Clean3 HEADNODE - a1516885367_clean1 Jan 25 15:18

9862 ubuntu PEND Clean3 HEADNODE - a1516885367_osplit Jan 25 15:18

9863 ubuntu PEND Stat HEADNODE - a1516885367_clean2 Jan 25 15:18

9864 ubuntu PEND Stat HEADNODE - a1516885367_launch Jan 25 15:18

9865 ubuntu PEND Stat HEADNODE - a1516885367_postproc_wrap Jan 25 15:18

9866 ubuntu PEND CleanFnl HEADNODE - a1516885367_prep_done Jan 25 15:18

9867 ubuntu PEND Stat HEADNODE - #!/bin/bash; #BSUB -q Stat; #BSUB -W 1200; #BSUB -o /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.out ; #BSUB -e /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.err; #BSUB -w " done(a151688 Jan 25 15:18

12856 ubuntu PEND long HEADNODE - a1517075571_hic Jan 27 17:52

12858 ubuntu PEND long HEADNODE - a1517075571_postproc Jan 27 17:52

12859 ubuntu PEND normal HEADNODE - a1517075571_clean3 Jan 27 17:52

12860 ubuntu PEND normal HEADNODE - a1517075571_done Jan 27 17:52

4) But once those commands finished running, Juicer was stalled again, this time at a done step:

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME

9505 ubuntu PEND normal HEADNODE - a1516851053_done Jan 25 03:50

9859 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge1 Jan 25 15:18

9860 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge Jan 25 15:18

9861 ubuntu PEND Clean3 HEADNODE - a1516885367_clean1 Jan 25 15:18

9862 ubuntu PEND Clean3 HEADNODE - a1516885367_osplit Jan 25 15:18

9863 ubuntu PEND Stat HEADNODE - a1516885367_clean2 Jan 25 15:18

9864 ubuntu PEND Stat HEADNODE - a1516885367_launch Jan 25 15:18

9865 ubuntu PEND Stat HEADNODE - a1516885367_postproc_wrap Jan 25 15:18

9867 ubuntu PEND Stat HEADNODE - #!/bin/bash; #BSUB -q Stat; #BSUB -W 1200; #BSUB -o /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.out ; #BSUB -e /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.err; #BSUB -w " done(a151688 Jan 25 15:18

12860 ubuntu PEND normal HEADNODE - a1517075571_done Jan 27 17:52

5) I tried killing jobs and once again running juicer.sh -S final, which started stats and hic30 jobs with different prefixes:

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME

12966 ubuntu RUN long HEADNODE HEADNODE a1517097868_stats Jan 28 00:04

12968 ubuntu RUN long HEADNODE HEADNODE a1517097868_hic30 Jan 28 00:04

9505 ubuntu PEND normal HEADNODE - a1516851053_done Jan 25 03:50

9859 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge1 Jan 25 15:18

9860 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge Jan 25 15:18

9861 ubuntu PEND Clean3 HEADNODE - a1516885367_clean1 Jan 25 15:18

9862 ubuntu PEND Clean3 HEADNODE - a1516885367_osplit Jan 25 15:18

9863 ubuntu PEND Stat HEADNODE - a1516885367_clean2 Jan 25 15:18

9864 ubuntu PEND Stat HEADNODE - a1516885367_launch Jan 25 15:18

9865 ubuntu PEND Stat HEADNODE - a1516885367_postproc_wrap Jan 25 15:18

9867 ubuntu PEND Stat HEADNODE - #!/bin/bash; #BSUB -q Stat; #BSUB -W 1200; #BSUB -o /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.out ; #BSUB -e /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.err; #BSUB -w " done(a151688 Jan 25 15:18

12860 ubuntu PEND normal HEADNODE - a1517075571_done Jan 27 17:52

12967 ubuntu PEND long HEADNODE - a1517097868_hic Jan 28 00:04

12969 ubuntu PEND long HEADNODE - a1517097868_postproc Jan 28 00:04

12970 ubuntu PEND normal HEADNODE - a1517097868_clean3 Jan 28 00:04

12971 ubuntu PEND normal HEADNODE - a1517097868_done Jan 28 00:04

6) But once they finished, Juicer was once again stalled at the same done step:

JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME

9505 ubuntu PEND normal HEADNODE - a1516851053_done Jan 25 03:50

9859 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge1 Jan 25 15:18

9860 ubuntu PEND Merge HEADNODE - a1516885367_fragmerge Jan 25 15:18

9861 ubuntu PEND Clean3 HEADNODE - a1516885367_clean1 Jan 25 15:18

9862 ubuntu PEND Clean3 HEADNODE - a1516885367_osplit Jan 25 15:18

9863 ubuntu PEND Stat HEADNODE - a1516885367_clean2 Jan 25 15:18

9864 ubuntu PEND Stat HEADNODE - a1516885367_launch Jan 25 15:18

9865 ubuntu PEND Stat HEADNODE - a1516885367_postproc_wrap Jan 25 15:18

9867 ubuntu PEND Stat HEADNODE - #!/bin/bash; #BSUB -q Stat; #BSUB -W 1200; #BSUB -o /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.out ; #BSUB -e /opt/juicer/work/NHCFV/NHCFV1/debug/finalclean-a1516885367.err; #BSUB -w " done(a151688 Jan 25 15:18

12860 ubuntu PEND normal HEADNODE - a1517075571_done Jan 27 17:52

12971 ubuntu PEND normal HEADNODE - a1517097868_done Jan 28 00:04

Neva Durand

unread,

Feb 1, 2018, 12:35:17 PM2/1/18

to Richard Gill, 3D Genomics

Hello,

It looks like your hic and stats jobs finished. Did you check the logs in the debug folder? You can also try looking at the inter.txt file and loading the inter.hic file in Juicebox. It's possible they finished with errors (which are reported in the logs), in which case the last job would not run since it's waiting for a clean completion.

Best

Neva

To unsubscribe from this group and stop receiving emails from it, send an email to 3d-genomics+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/06ef2dc2-a687-4146-a731-fae6a27104b4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Richard Gill

unread,

Feb 1, 2018, 2:14:11 PM2/1/18

to Neva Durand, 3D Genomics

Hi Neva,

1) Would you please clarify the specific logs I should check? Alignment and merging seemed to work, but there were a fair number generated after these steps:

postproc-a1516898811.err

postproc-a1516898811.out

postproc-a1517075571.err

postproc-a1517075571.out

postproc-a1517097868.err

postproc-a1517097868.out

postproc-a1517122532.err

postproc-a1517122532.out

postproc-a1517153415.err

postproc-a1517153415.out

split-a1516898811.err

split-a1516898811.out

splitmv-a1516898811.err

splitmv-a1516898811.out

stats30hic30-a1517075571.err

stats30hic30-a1517075571.out

stats30hic30-a1517097868.err

stats30hic30-a1517097868.out

stats30hic30-a1517122532.err

stats30hic30-a1517122532.out

stats-a1516898811.err

stats-a1516898811.out

stats-a1517075571.err

stats-a1517075571.out

stats-a1517097868.err

stats-a1517097868.out

stats-a1517122532.err

stats-a1517122532.out

stats-a1517153415.err

stats-a1517153415.out

2) in the aligned folder I was able to find the inter.txt file, with contents below, but no inter.hic file, unless it's somewhere else (I couldn't find it using the find command).

Experiment description: Juicer version 1.5.6; BWA 0.7.12-r1039; 8 threads; splitsize 90000000; java version "1.7.0_79"; /opt/juicer/scripts/juicer.sh -S final

Sequenced Read Pairs: 2,372,999,927

Normal Paired: 802,630,753 (33.82%)

Chimeric Paired: 1,182,415,488 (49.83%)

Chimeric Ambiguous: 374,179,723 (15.77%)

Unmapped: 13,773,963 (0.58%)

Ligation Motif Present: 1,977,990,097 (83.35%)

Alignable (Normal+Chimeric Paired): 1,985,046,241 (83.65%)

Thanks,

Richard

Neva Durand

unread,

Feb 1, 2018, 2:18:29 PM2/1/18

to Richard Gill, 3D Genomics

Hello,

Look at your most recent stats err and stats out files. It looks like the stats job failed.

Best

Neva

Richard Gill

unread,

Feb 1, 2018, 2:35:38 PM2/1/18

to Neva Durand, 3D Genomics

Hi Neva,

1) These are the contents of the most recent stats err files:

stats-a1517122532.err

Exception in thread "main" java.lang.UnsupportedClassVersionError: LibraryComplexity : Unsupported major.minor version 52.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

stats30hic30-a1517122532.err

Picked up _JAVA_OPTIONS: -Xmx16384m

Exception in thread "main" java.lang.UnsupportedClassVersionError: LibraryComplexity : Unsupported major.minor version 52.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:800)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)

at java.net.URLClassLoader.access$100(URLClassLoader.java:71)

at java.net.URLClassLoader$1.run(URLClassLoader.java:361)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

Picked up _JAVA_OPTIONS: -Xmx8192m

Problem with creating fragment-delimited maps, NullPointerException.

This could be due to a null fragment map or to a mismatch in the chromosome name in the fragment map vis-a-vis the input file or chrom.sizes file.

Exiting.

2) These are the contents of the most recent stats out files:

cat stats-a1517122532.out

Sender: LSF System <genome@HEADNODE>

Subject: Job 13073: <a1517122532_launch> Done

Job <a1517122532_launch> was submitted from host <HEADNODE> by user <ubuntu>.

Job was executed on host(s) <HEADNODE>, in queue <Stat>, as user <ubuntu>.

</home/ubuntu> was used as the home directory.

</opt/juicer/work/NHCFV> was used as the working directory.

Started at Sun Jan 28 06:55:33 2018

Results reported at Sun Jan 28 06:55:33 2018

Your job looked like:

------------------------------------------------------------

# LSBATCH: User input

#!/bin/bash

#BSUB -q Stat

#BSUB -W 1200

#BSUB -o /opt/juicer/work/NHCFV/debug/stats-a1517122532.out

#BSUB -e /opt/juicer/work/NHCFV/debug/stats-a1517122532.err

#BSUB -J "a1517122532_launch"

echo "(-: Alignment and merge done, launching other jobs."

bsub -o /opt/juicer/work/NHCFV/debug/stats-a1517122532.out -e /opt/juicer/work/NHCFV/debug/stats-a1517122532.err -q long -W 3600 -R "rusage[mem=16000]" -J "a1517122532_stats" "df -h;_JAVA_OPTIONS=-Xmx16384m; export LC_ALL=en_US.UTF-8; tail -n1 /opt/juicer/work/NHCFV/debug/head-a1517122532.out | awk '{printf\"%-1000s\n\", \$0}' > /opt/juicer/work/NHCFV/aligned/inter.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/stats_dups.txt /opt/juicer/work/NHCFV/aligned/dups.txt; cat /opt/juicer/work/NHCFV/splits/*.res.txt | awk -f /opt/juicer/scripts/stats_sub.awk >> /opt/juicer/work/NHCFV/aligned/inter.txt; java -cp /opt/juicer/scripts/ LibraryComplexity /opt/juicer/work/NHCFV/aligned inter.txt >> /opt/juicer/work/NHCFV/aligned/inter.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/inter.txt -q 1 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt; cat /opt/juicer/work/NHCFV/splits/*_abnorm.sam > /opt/juicer/work/NHCFV/aligned/abnormal.sam; cat /opt/juicer/work/NHCFV/splits/*_unmapped.sam > /opt/juicer/work/NHCFV/aligned/unmapped.sam; awk -f /opt/juicer/scripts/collisions.awk /opt/juicer/work/NHCFV/aligned/abnormal.sam > /opt/juicer/work/NHCFV/aligned/collisions.txt"

bsub -o /opt/juicer/work/NHCFV/debug/hic-a1517122532.out -e /opt/juicer/work/NHCFV/debug/hic-a1517122532.err -q long -W 3600 -R "rusage[mem=16000]" -w "done(a1517122532_stats)" -J "a1517122532_hic" "df -h;export _JAVA_OPTIONS=-Xmx16384m; if [ \"0\" -eq 1 ]; then /opt/juicer/scripts/juicer_tools pre -s /opt/juicer/work/NHCFV/aligned/inter.txt -g /opt/juicer/work/NHCFV/aligned/inter_hists.m -q 1 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter.hic hg38; else /opt/juicer/scripts/juicer_tools pre -f /opt/juicer/restriction_sites/hg38_DpnII.txt -s /opt/juicer/work/NHCFV/aligned/inter.txt -g /opt/juicer/work/NHCFV/aligned/inter_hists.m -q 1 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter.hic hg38; fi ;"

bsub -o /opt/juicer/work/NHCFV/debug/stats30hic30-a1517122532.out -e /opt/juicer/work/NHCFV/debug/stats30hic30-a1517122532.err -q long -W 3600 -R "rusage[mem=16000]" -J "a1517122532_hic30" "df -h;export _JAVA_OPTIONS=-Xmx16384m; export LC_ALL=en_US.UTF-8; tail -n1 /opt/juicer/work/NHCFV/debug/head-a1517122532.out | awk '{printf\"%-1000s\n\", \$0}' > /opt/juicer/work/NHCFV/aligned/inter_30.txt; cat /opt/juicer/work/NHCFV/splits/*.res.txt | awk -f /opt/juicer/scripts/stats_sub.awk >> /opt/juicer/work/NHCFV/aligned/inter_30.txt; java -cp /opt/juicer/scripts/ LibraryComplexity /opt/juicer/work/NHCFV/aligned inter_30.txt >> /opt/juicer/work/NHCFV/aligned/inter_30.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/inter_30.txt -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt; export _JAVA_OPTIONS=-Xmx8192m; if [ \"0\" -eq 1 ]; then /opt/juicer/scripts/juicer_tools pre -s /opt/juicer/work/NHCFV/aligned/inter_30.txt -g /opt/juicer/work/NHCFV/aligned/inter_30_hists.m -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter_30.hic hg38; else /opt/juicer/scripts/juicer_tools pre -f /opt/juicer/restriction_sites/hg38_DpnII.txt -s /opt/juicer/work/NHCFV/aligned/inter_30.txt -g /opt/juicer/work/NHCFV/aligned/inter_30_hists.m -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter_30.hic hg38; fi"

echo

------------------------------------------------------------

Exited

The output (if any) follows:

(-: Alignment and merge done, launching other jobs.

Job <13077> is submitted to queue <long>.

Job <13078> is submitted to queue <long>.

Job <13079> is submitted to queue <long>.

PS:

Read file </opt/juicer/work/NHCFV/debug/stats-a1517122532.err> for stderr output of this job.

Sender: LSF System <genome@HEADNODE>

Subject: Job 13077: <a1517122532_stats> Exited

Job <a1517122532_stats> was submitted from host <HEADNODE> by user <ubuntu>.

Job was executed on host(s) <HEADNODE>, in queue <long>, as user <ubuntu>.

</home/ubuntu> was used as the home directory.

</opt/juicer/work/NHCFV> was used as the working directory.

Started at Sun Jan 28 06:55:34 2018

Results reported at Sun Jan 28 12:39:19 2018

Your job looked like:

------------------------------------------------------------

# LSBATCH: User input

df -h;_JAVA_OPTIONS=-Xmx16384m; export LC_ALL=en_US.UTF-8; tail -n1 /opt/juicer/work/NHCFV/debug/head-a1517122532.out | awk '{printf"%-1000s\n", $0}' > /opt/juicer/work/NHCFV/aligned/inter.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/stats_dups.txt /opt/juicer/work/NHCFV/aligned/dups.txt; cat /opt/juicer/work/NHCFV/splits/*.res.txt | awk -f /opt/juicer/scripts/stats_sub.awk >> /opt/juicer/work/NHCFV/aligned/inter.txt; java -cp /opt/juicer/scripts/ LibraryComplexity /opt/juicer/work/NHCFV/aligned inter.txt >> /opt/juicer/work/NHCFV/aligned/inter.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/inter.txt -q 1 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt; cat /opt/juicer/work/NHCFV/splits/*_abnorm.sam > /opt/juicer/work/NHCFV/aligned/abnormal.sam; cat /opt/juicer/work/NHCFV/splits/*_unmapped.sam > /opt/juicer/work/NHCFV/aligned/unmapped.sam; awk -f /opt/juicer/scripts/collisions.awk /opt/juicer/work/NHCFV/aligned/abnormal.sam > /opt/juicer/work/NHCFV/aligned/collisions.txt

------------------------------------------------------------

Exited with signal termination: Killed.

Resource usage summary:

CPU time : 0.01 sec.

Max Memory : 770 MB

Max Swap : 808 MB

Max Processes : 3

The output (if any) follows:

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 20G 5.6G 14G 30% /

udev 30G 12K 30G 1% /dev

tmpfs 6.0G 256K 6.0G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 30G 0 30G 0% /run/shm

/dev/xvdb 15T 8.9T 4.9T 65% /opt

PS:

Read file </opt/juicer/work/NHCFV/debug/stats-a1517122532.err> for stderr output of this job.

stats30hic30-a1517122532.out

Sender: LSF System <genome@HEADNODE>

Subject: Job 13079: <a1517122532_hic30> Exited

Job <a1517122532_hic30> was submitted from host <HEADNODE> by user <ubuntu>.

Job was executed on host(s) <HEADNODE>, in queue <long>, as user <ubuntu>.

</home/ubuntu> was used as the home directory.

</opt/juicer/work/NHCFV> was used as the working directory.

Started at Sun Jan 28 06:55:34 2018

Results reported at Sun Jan 28 12:38:31 2018

Your job looked like:

------------------------------------------------------------

# LSBATCH: User input

df -h;export _JAVA_OPTIONS=-Xmx16384m; export LC_ALL=en_US.UTF-8; tail -n1 /opt/juicer/work/NHCFV/debug/head-a1517122532.out | awk '{printf"%-1000s\n", $0}' > /opt/juicer/work/NHCFV/aligned/inter_30.txt; cat /opt/juicer/work/NHCFV/splits/*.res.txt | awk -f /opt/juicer/scripts/stats_sub.awk >> /opt/juicer/work/NHCFV/aligned/inter_30.txt; java -cp /opt/juicer/scripts/ LibraryComplexity /opt/juicer/work/NHCFV/aligned inter_30.txt >> /opt/juicer/work/NHCFV/aligned/inter_30.txt; /opt/juicer/scripts/statistics.pl -s /opt/juicer/restriction_sites/hg38_DpnII.txt -l GATCGATC -o /opt/juicer/work/NHCFV/aligned/inter_30.txt -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt; export _JAVA_OPTIONS=-Xmx8192m; if [ "0" -eq 1 ]; then /opt/juicer/scripts/juicer_tools pre -s /opt/juicer/work/NHCFV/aligned/inter_30.txt -g /opt/juicer/work/NHCFV/aligned/inter_30_hists.m -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter_30.hic hg38; else /opt/juicer/scripts/juicer_tools pre -f /opt/juicer/restriction_sites/hg38_DpnII.txt -s /opt/juicer/work/NHCFV/aligned/inter_30.txt -g /opt/juicer/work/NHCFV/aligned/inter_30_hists.m -q 30 /opt/juicer/work/NHCFV/aligned/merged_nodups.txt /opt/juicer/work/NHCFV/aligned/inter_30.hic hg38; fi

------------------------------------------------------------

Exited with exit code 63.

Resource usage summary:

CPU time : 20418.05 sec.

Max Memory : 768 MB

Max Swap : 806 MB

Max Processes : 3

The output (if any) follows:

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 20G 5.6G 14G 30% /

udev 30G 12K 30G 1% /dev

tmpfs 6.0G 256K 6.0G 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 30G 0 30G 0% /run/shm

/dev/xvdb 15T 8.9T 4.9T 65% /opt

PS:

Read file </opt/juicer/work/NHCFV/debug/stats30hic30-a1517122532.err> for stderr output of this job.

Please let me know if there are steps I can take to finish the run.

Thanks,

Richard

David Weisz

unread,

Feb 6, 2018, 7:33:14 PM2/6/18

to 3D Genomics

Hi,

Solving the problem related to "Unsupported major.minor version 52.0" can be done by simply updating to the latest java version.

Follow these steps:

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java8-installer

Best,

David.

To view this discussion on the web visit https://groups.google.com/d/msgid/3d-genomics/06ef2dc2-a687-4146-a731-fae6a27104b4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Neva Cherniavsky Durand, Ph.D.
Staff Scientist, Aiden Lab
www.aidenlab.org

Richard Gill

unread,

Feb 7, 2018, 6:13:51 PM2/7/18

to David Weisz, 3D Genomics

Thanks David. I will give that a try. I updated the juicer_tools jar and the pipeline ran until the done step and generated a .hic file.

Richard

...

[Message clipped]

Richard Gill

unread,

Feb 9, 2018, 12:54:42 AM2/9/18

to 3D Genomics

Hi David,

sudo apt-get update gets hung up because it cannot update R. It turns out most mirrors no longer host R for Ubuntu 12.04 (Precise).

Without getting past this hurdle, I can't update Java on the Juicer AMI.

Please advise.

Thanks,

Richard

Richard Gill

unread,

Feb 23, 2018, 10:50:29 AM2/23/18

to 3D Genomics

Hi,

I updated Java and re-ran, and there is now an out-of-memory error in my stats30hic30*.err file:

Picked up _JAVA_OPTIONS: -Xmx16384m

Picked up _JAVA_OPTIONS: -Xmx8192m

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects

at java.util.Arrays.copyOf(Arrays.java:3181)

at java.util.ArrayList.grow(ArrayList.java:265)

at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)

at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)

at java.util.ArrayList.add(ArrayList.java:462)

at com.google.common.base.Splitter.splitToList(Splitter.java:422)

at juicebox.tools.utils.original.AsciiPairIterator.advance(AsciiPairIterator.java:106)

at juicebox.tools.utils.original.AsciiPairIterator.next(AsciiPairIterator.java:247)

at juicebox.tools.utils.original.Preprocessor.writeBody(Preprocessor.java:392)

at juicebox.tools.utils.original.Preprocessor.preprocess(Preprocessor.java:286)

at juicebox.tools.clt.old.PreProcessing.run(PreProcessing.java:105)

at juicebox.tools.HiCTools.main(HiCTools.java:98)

Please advise.

Thanks,

Richard

On Friday, February 9, 2018 at 12:54:42 AM UTC-5, Richard Gill wrote:

Hi David,

sudo apt-get update gets hung up because it cannot update R. It turns out most mirrors no longer host R for Ubuntu 12.04 (Precise).
Without getting past this hurdle, I can't update Java on the Juicer AMI.

Please advise.

Thanks,

Richard

On Wednesday, February 7, 2018 at 6:13:51 PM UTC-5, Richard Gill wrote:

Thanks David. I will give that a try. I updated the juicer_tools jar and the pipeline ran until the done step and generated a .hic file.

Richard

David Weisz

unread,

Feb 23, 2018, 4:46:09 PM2/23/18

to 3D Genomics

Hi,

How much memory is allocated to the AWS instance you are running juicer_tools?

Richard Gill

unread,

Feb 23, 2018, 4:50:04 PM2/23/18

to 3D Genomics

Hi David,

This is occurring on the most powerful AWS instance x1e.32xlarge (3904.0 GB RAM, 128 vCPUs). My data is high-resolution (~2-kb) but when I monitor processes, not all of the RAM is being used so I suspect there is a different issue going on.

Thanks,

Richard

Neva Durand

unread,

Feb 23, 2018, 5:01:51 PM2/23/18

to Richard Gill, 3D Genomics

You should probably change the Java options to load more initial and maximum heap space.

Please advise.

<div style="font-family:arial

Richard Gill

unread,

Feb 23, 2018, 5:17:06 PM2/23/18

to Neva Durand, 3D Genomics

Hi Neva,

In the juicer.sh file I see the following potentially relevant lines:

bsub -o ${debugdir}/stats-${groupname}.out -e ${debugdir}/stats-${groupname}.err -q $long_queue -W $long_queue_time -R "rusage[mem=16000]" $waitstring22 -J "${groupname}_stats" "df -h;_JAVA_OPTIONS=-Xmx16384m; export LC_ALL=en_U$

bsub -o ${debugdir}/hic-${groupname}.out -e ${debugdir}/hic-${groupname}.err -q $long_queue -W $long_queue_time -R "rusage[mem=16000]" -w "done(${groupname}_stats)" -J "${groupname}_hic" "df -h;export _JAVA_OPTIONS=-Xmx16384m; i$

bsub -o ${debugdir}/stats30hic30-${groupname}.out -e ${debugdir}/stats30hic30-${groupname}.err -q $long_queue -W $long_queue_time -R "rusage[mem=16000]" $waitstring22 -J "${groupname}_hic30" "df -h;export _JAVA_OPTIONS=-Xmx16384$

1) Please confirm

a) whether these three are the lines I should adjust,

b) the exact variables I should change

c) the recommended values.

2) Are there any additional changes needed?

Thanks,

Richard

Neva Durand

unread,

Feb 23, 2018, 5:23:20 PM2/23/18

to Richard Gill, 3D Genomics

Change

mem=16000

To

mem=40000

And

Xmx16384m

To

Xmx40000m

It is surprising, however, that this is failing in stats.

--

Richard Gill

unread,

Feb 23, 2018, 5:30:16 PM2/23/18

to Neva Durand, 3D Genomics

Thanks Neva. I'll give this a try.

To give you a sense of my data's size, and perhaps why this is failing, these are my file sizes after the post-stalling cat steps on AWS:

opt_dups.txt - 4.7 GB

dups.txt - 142 GB

merged_nodups.txt - 629 GB

Richard

Kayleen

unread,

Apr 5, 2020, 10:44:04 PM4/5/20

to 3D Genomics

Dear all,

Sorry for bringing up this old thread, but I am facing the exact same error.

I tried sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer as recommended, however it appears that the java version is outdated. Please view the attached files for the error.