TASSEL out of memory problem

[Feseha Akele]

Hello TASSEL user group
I am running TASSEL using a reference genome composed of 247 contigs. I could run the -FastqToTagCountPlugin
-MergeMultipleTagCountPlugin and the bowtie2 alignment  but the pipeline fails on the -FastqToTBTPlugin.
 I am using a 16 core 32Gig RAM machine. I changed the memory to 15g and 30g but I still get the out of
memory error shown below. What can I do to resolve this problem?

Tassel Pipeline Arguments: -fork1 -FastqToTBTPlugin -i ../../fastq -k ../../fastq/DavisGBSkey.txt -e PstI-MspI -o tbt -t mergedTagCounts/Davis_GBS_MasterTags.cnt.fq -y -endPlugin -runfork1
[main] INFO net.maizegenetics.pipeline.TasselPipeline - Tassel Version: 3.0.157  Date: June 27, 2013
[main] INFO net.maizegenetics.gbs.pipeline.FastqToTBTPlugin - FastqToTBTPlugin: setParameters: Using the following fastq files:
[main] INFO net.maizegenetics.gbs.pipeline.FastqToTBTPlugin - /data/davis/DavisGBS_June2013/contigs_247_based_analysis/../../fastq/H0WE7ADXX_1_Davis.fastq
Reading Haplotypes distribution from:mergedTagCounts/Davis_GBS_MasterTags.cnt.fq
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at net.maizegenetics.gbs.tagdist.TagCounts.initMatrices(TagCounts.java:62)
        at net.maizegenetics.gbs.tagdist.TagCounts.readBinaryTagCountFile(TagCounts.java:179)
        at net.maizegenetics.gbs.tagdist.TagCounts.readDistFile(TagCounts.java:56)
        at net.maizegenetics.gbs.tagdist.TagCounts.<init>(TagCounts.java:45)
        at net.maizegenetics.gbs.pipeline.FastqToTBTPlugin.setParameters(FastqToTBTPlugin.java:195)
        at net.maizegenetics.pipeline.TasselPipeline.parseArgs(TasselPipeline.java:1219)
        at net.maizegenetics.pipeline.TasselPipeline.<init>(TasselPipeline.java:121)
        at net.maizegenetics.pipeline.TasselPipeline.main(TasselPipeline.java:161)

Any suggestion would be appreciated.
Feseha

[Terry Casstevens]

Please see this document how to increase heap size.   Send the command you are running if you still have problems.

http://www.maizegenetics.net/tassel/docs/ExecutingTassel.pdf

Cheers,

Terry

--
You received this message because you are subscribed to the Google Groups "TASSEL - Trait Analysis by Association, Evolution and Linkage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tassel+un...@googlegroups.com.
To post to this group, send email to tas...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tassel/e99be04d-cc0e-45e2-8073-e9583fca2dc2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Feseha Akele]

Hi Terry;
I set the memory in the run_pipeline.pl script and as you can see below it is set for a minimum of 20 and a maximum of 32g RAM
I also tried a minumum 15g amd max 30g combination.
As per your suggestion I have posted the command as well as the entire output of the command ..

fesehat@cobb contigs_247_based_analysis]# /usr/local/src/tassel3-standalone/run_pipeline.pl -fork1 -FastqToTBTPlugin -i ../../fastq -k ../../fastq/DavisGBSkey.txt -e PstI-MspI -o tbt -t mergedTagCounts/Davis_GBS_MasterTags.cnt.fq -y -endPlugin  -runfork1
/usr/local/src/tassel3-standalone/lib/poi-3.0.1-FINAL-20070705.jar:/usr/local/src/tassel3-standalone/lib/commons-lang-2.1.jar:/usr/local/src/tassel3-standalone/lib/commons-configuration-1.2.jar:/usr/local/src/tassel3-standalone/lib/batik-css
.jar:/usr/local/src/tassel3-standalone/lib/jung-1.7.5.jar:/usr/local/src/tassel3-standalone/lib/axiom-impl-1.2.4.jar:/usr/local/src/tassel3-standalone/lib/jcommon-1.0.6.jar:/usr/local/src/tassel3-standalone/lib/xmlParserAPIs.jar:/usr/local/s
rc/tassel3-standalone/lib/batik-dom.jar:/usr/local/src/tassel3-standalone/lib/biojava3-alignment-3.0.jar:/usr/local/src/tassel3-standalone/lib/ssj.jar:/usr/local/src/tassel3-standalone/lib/commons-httpclient-3.0.1.jar:/usr/local/src/tassel3-
standalone/lib/commons-codec-1.3.jar:/usr/local/src/tassel3-standalone/lib/batik-parser.jar:/usr/local/src/tassel3-standalone/lib/ejml-0.13.jar:/usr/local/src/tassel3-standalone/lib/commons-dbcp-1.2.1.jar:/usr/local/src/tassel3-standalone/li
b/commons-pool-1.3.jar:/usr/local/src/tassel3-standalone/lib/axis2-kernel-1.2.jar:/usr/local/src/tassel3-standalone/lib/batik-ext.jar:/usr/local/src/tassel3-standalone/lib/jfreechart-1.0.3.jar:/usr/local/src/tassel3-standalone/lib/batik-xml.
jar:/usr/local/src/tassel3-standalone/lib/colt.jar:/usr/local/src/tassel3-standalone/lib/XmlSchema-1.3.1.jar:/usr/local/src/tassel3-standalone/lib/lzma-4.63-jio-0.93.jar:/usr/local/src/tassel3-standalone/lib/LiuExt.jar:/usr/local/src/tassel3
-standalone/lib/xml.jar:/usr/local/src/tassel3-standalone/lib/batik-gui-util.jar:/usr/local/src/tassel3-standalone/lib/log4j-1.2.13.jar:/usr/local/src/tassel3-standalone/lib/biojava3-core-3.0.jar:/usr/local/src/tassel3-standalone/lib/batik-s
vggen.jar:/usr/local/src/tassel3-standalone/lib/batik-util.jar:/usr/local/src/tassel3-standalone/lib/biojava3-phylo-3.0.jar:/usr/local/src/tassel3-standalone/lib/batik-svg-dom.jar:/usr/local/src/tassel3-standalone/lib/commons-collections-3.1
.jar:/usr/local/src/tassel3-standalone/lib/batik-gvt.jar:/usr/local/src/tassel3-standalone/lib/commons-math-2.2.jar:/usr/local/src/tassel3-standalone/lib/crimson-parser.jar:/usr/local/src/tassel3-standalone/lib/batik-awt-util.jar:/usr/local/
src/tassel3-standalone/lib/commons-logging-1.1.jar:/usr/local/src/tassel3-standalone/lib/gdpc.jar:/usr/local/src/tassel3-standalone/lib/mail-1.4.jar:/usr/local/src/tassel3-standalone/lib/cisd-jhdf5-batteries_included_lin_win_mac.jar:/usr/loc
al/src/tassel3-standalone/lib/wsdl4j-1.6.2.jar:/usr/local/src/tassel3-standalone/lib/forester.jar:/usr/local/src/tassel3-standalone/lib/jgrapht-0.5.3.jar:/usr/local/src/tassel3-standalone/lib/mysql-connector-java-3.1.13-bin.jar:/usr/local/sr
c/tassel3-standalone/lib/commons-discovery.jar:/usr/local/src/tassel3-standalone/lib/xercesImpl.jar:/usr/local/src/tassel3-standalone/lib/junit-4.1.jar:/usr/local/src/tassel3-standalone/lib/axiom-api-1.2.4.jar:/usr/local/src/tassel3-standalo
ne/lib/geronimo-spec-activation-1.0.2-rc4.jar:/usr/local/src/tassel3-standalone/sTASSEL.jar
Memory Settings: -Xms20g -Xmx32g

Tassel Pipeline Arguments: -fork1 -FastqToTBTPlugin -i ../../fastq -k ../../fastq/DavisGBSkey.txt -e PstI-MspI -o tbt -t mergedTagCounts/Davis_GBS_MasterTags.cnt.fq -y -endPlugin -runfork1
[main] INFO net.maizegenetics.pipeline.TasselPipeline - Tassel Version: 3.0.157  Date: June 27, 2013
[main] INFO net.maizegenetics.gbs.pipeline.FastqToTBTPlugin - FastqToTBTPlugin: setParameters: Using the following fastq files:
[main] INFO net.maizegenetics.gbs.pipeline.FastqToTBTPlugin - /data/davis/DavisGBS_June2013/contigs_247_based_analysis/../../fastq/H0WE7ADXX_1_Davis.fastq
Reading Haplotypes distribution from:mergedTagCounts/Davis_GBS_MasterTags.cnt.fq
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at net.maizegenetics.gbs.tagdist.TagCounts.initMatrices(TagCounts.java:62)
        at net.maizegenetics.gbs.tagdist.TagCounts.readBinaryTagCountFile(TagCounts.java:179)
        at net.maizegenetics.gbs.tagdist.TagCounts.readDistFile(TagCounts.java:56)
        at net.maizegenetics.gbs.tagdist.TagCounts.<init>(TagCounts.java:45)
        at net.maizegenetics.gbs.pipeline.FastqToTBTPlugin.setParameters(FastqToTBTPlugin.java:195)
        at net.maizegenetics.pipeline.TasselPipeline.parseArgs(TasselPipeline.java:1219)
        at net.maizegenetics.pipeline.TasselPipeline.<init>(TasselPipeline.java:121)
        at net.maizegenetics.pipeline.TasselPipeline.main(TasselPipeline.java:161)

[Feseha Akele]

[Terry Casstevens]

Please try 20g as the max instead of maxing out the machine.  Also consider updating to the latest build of Tassel 3.

Cheers,

Terry

--

You received this message because you are subscribed to the Google Groups "TASSEL - Trait Analysis by Association, Evolution and Linkage" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tassel+un...@googlegroups.com.
To post to this group, send email to tas...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tassel/dab07400-d2ca-4b5f-86fc-013409f95db2%40googlegroups.com.

[Jeff Glaubitz]

Hi Feseha,

 

The problem is NOT the memory settings (a max of 16g should suffice).  It is the format of the master tag list file (Davis_GBS_MasterTags.cnt.fq).  The *.fq version is a text file (for use as input for BWA or bowtie2).  The FastqToTBTPlugin  needs to use a binary (*.cnt) file for the -t (“tag list”) option.  If you try the unix command:

>head  Davis_GBS_MasterTags.cnt

the output should look like gibberish.

 

This is all explained in the documentation (under FastqToTBTPlugin):

http://www.maizegenetics.net/tassel/docs/TasselPipelineGBS.pdf

 

Best,

 

Jeff

 

--

Jeff Glaubitz

Project Manager

Genetic Architecture of Maize and Teosinte

National Science Foundation award 0820619

http://www.panzea.org

Institute for Genomic Diversity

Cornell University

175 Biotechnology Bldg

Ithaca, NY 14853

Phone: 607-255-1386

jcg...@cornell.edu

To view this discussion on the web visit https://groups.google.com/d/msgid/tassel/CACHsrTseMYD9tawgd2FJgSi6kUpoCWO-tmDGkmW1HQgmWd8DXg%40mail.gmail.com.

[Feseha Akele]

Hi Terry;
Thanks for the conversation. I took ouy the -t option from the -MergeMultipleTagCountPlugin command and the *.cnt file became binary
instead of a *.cnt.fq text file. And now it is running as expected.

Thanks
Feseha

To post to this group, send email to ta...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tassel/dab07400-d2ca-4b5f-86fc-013409f95db2%40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.