Jellyfish error

[Jonathan Featherston]

Dear Trinity Support 

I'm getting the following error while running trinity 2-0.6:

terminate called after throwing an instance of 'jellyfish::thread_exec::Error'

  what():  Can't create thread: Resource temporarily unavailable

I can't find the above error on other threads. 

Below is my output. I've tried a lot of different things up to this point. First I had to deal with Java given that we have only Java 1.6 installed. Basically resolved by installing a local copy of 1.7 and exporting the path to 1.7 in my bash_profile. I've attempted running with fewer CPUs. Originally with 16 then down to just 1. The node has 64GB of memory but I'm only working here with about 50million reads. I began by running without normalise_reads but got core_dump errors. Then learnt that I had to un limit the stack so have done so and I included normalize-reads. I'm not sure if the unlimit stack size command is appropriate to Trinity 2 but I have tried with and without it to no avail.     

I've run a few commands but I believe this was the last one (and is representative):

./Trinity --seqType fq --left ../tetraTR1.fastq --right ../tetraTR2.fastq --SS_lib_type RF --CPU 10 --max_memory 62G --verbose --normalize_reads --output trinity_tetra3

I have run the command on its own with the same error: CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/jellyfish/bin/jellyfish count -t 10 -m 25 -s 8661787384  both.fa

Your assistance would be greatly appreciated.

Kind Regards

Jonathan

 

Trinity version: v2.0.6

-ERROR: couldn't run the network check to confirm latest Trinity software version.

Paired mode requires bowtie. Found bowtie at: /scratch/sysusers/Jonathan/src/bowtie-master/bowtie

 and bowtie-build at /scratch/sysusers/Jonathan/src/bowtie-master/bowtie-build

-since butterfly will eventually be run, lets test for proper execution of java

Found samtools at: /scratch/bin/samtools

#######################################

Running Java Tests

Tuesday, April 28, 2015: 12:56:49 CMD: java -Xmx64m -jar /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/util/support_scripts/ExitTester.jar 0

CMD finished (0 seconds)

Tuesday, April 28, 2015: 12:56:49 CMD: java -Xmx64m -jar /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/util/support_scripts/ExitTester.jar 1

-we properly captured the java failure status, as needed.  Looking good.

Java tests succeeded.

###################################

Tuesday, April 28, 2015: 12:56:50 CMD: mkdir -p /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity_tetra3

CMD finished (0 seconds)

Tuesday, April 28, 2015: 12:56:50 CMD: mkdir -p /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity_tetra3/chrysalis

CMD finished (0 seconds)

----------------------------------------------------------------------------------

-------------- Trinity Phase 1: Clustering of RNA-Seq Reads  ---------------------

----------------------------------------------------------------------------------

Converting input files. (in parallel)Tuesday, April 28, 2015: 12:56:50 CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/fastool/fastool --rev  --illumina-trinity --to-fasta /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR1.fastq >> left.fa 2> /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR1.fastq.readcount 

Tuesday, April 28, 2015: 12:56:50 CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/fastool/fastool --illumina-trinity --to-fasta /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR2.fastq >> right.fa 2> /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR2.fastq.readcount 

CMD finished (73 seconds)

CMD finished (97 seconds)

Tuesday, April 28, 2015: 12:58:27 CMD: cat left.fa right.fa > both.fa

Done converting input files.CMD finished (20 seconds)

-------------------------------------------

----------- Jellyfish  --------------------

-- (building a k-mer catalog from reads) --

-------------------------------------------

Tuesday, April 28, 2015: 12:58:53 CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/jellyfish/bin/jellyfish count -t 10 -m 25 -s 8661787384  both.fa

terminate called after throwing an instance of 'jellyfish::thread_exec::Error'

  what():  Can't create thread: Resource temporarily unavailable

Error, cmd: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/jellyfish/bin/jellyfish count -t 10 -m 25 -s 8661787384  both.fa died with ret 134 at ./Trinity line 2116.

Trinity run failed. Must investigate error above.

[Tiago Hori]

What version of Java Virtual Machine are you using?

T.

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Featherston]

I'm using a JRE. 1.7.

[Tiago Hori]

Try Oracle 1.8. 1.7 crashes the new version every now and then, not sure why.

T.

Sent from my iPhone

[Jonathan Featherston]

Ok thank you I will try that!

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Jonathan Featherston]

Hi Tiago

I have tried this with the JDK 1.8 and get the same error I'm afraid.

J

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Tiago Hori]

There are a few things that sometimes result in a problem with resource allocation. We seen number of files limitation brake Jellyfish, sometimes people think they have enough disk space, but that what they can actually access has been limited. Lastly, sometimes people run out of RAM, but I am not sure that is your case. How many reads do you have?

T.

Ultimate stresses sportsmanship and fair play. Competitive play is encouraged, but NEVER AT THE EXPENSE OF RESPECT BETWEEN PLAYERS, adherence to the rules and the BASIC JOY OF PLAY.

Dear Trinity Support 

Tuesday, April 28, 2015: 12:56:49CMD: java -Xmx64m -jar /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/util/support_scripts/ExitTester.jar 0

CMD finished (0 seconds)

Tuesday, April 28, 2015: 12:56:49CMD: java -Xmx64m -jar /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/util/support_scripts/ExitTester.jar 1

-we properly captured the java failure status, as needed.  Looking good.

Java tests succeeded.

###################################

Tuesday, April 28, 2015: 12:56:50CMD: mkdir -p /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity_tetra3

CMD finished (0 seconds)

Tuesday, April 28, 2015: 12:56:50CMD: mkdir -p /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity_tetra3/chrysalis

CMD finished (0 seconds)

----------------------------------------------------------------------------------

-------------- Trinity Phase 1: Clustering of RNA-Seq Reads  ---------------------

----------------------------------------------------------------------------------

Converting input files. (in parallel)Tuesday, April 28, 2015: 12:56:50CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/fastool/fastool --rev  --illumina-trinity --to-fasta /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR1.fastq >> left.fa 2> /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR1.fastq.readcount 

Tuesday, April 28, 2015: 12:56:50CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/fastool/fastool --illumina-trinity --to-fasta /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR2.fastq >> right.fa 2> /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/../tetraTR2.fastq.readcount 

CMD finished (73 seconds)

CMD finished (97 seconds)

Tuesday, April 28, 2015: 12:58:27CMD: cat left.fa right.fa > both.fa

Done converting input files.CMD finished (20 seconds)

-------------------------------------------

----------- Jellyfish  --------------------

-- (building a k-mer catalog from reads) --

-------------------------------------------

Tuesday, April 28, 2015: 12:58:53CMD: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/jellyfish/bin/jellyfish count -t 10 -m 25 -s 8661787384  both.fa

terminate called after throwing an instance of 'jellyfish::thread_exec::Error'

  what():  Can't create thread: Resource temporarily unavailable

Error, cmd: /scratch/sysusers/Jonathan/Transcriptome/trinityrnaseq-2.0.6/trinity-plugins/jellyfish/bin/jellyfish count -t 10 -m 25 -s 8661787384  both.fa died with ret 134 at ./Trinity line 2116.

Trinity run failed. Must investigate error above.

[Jonathan Featherston]

Only 40million. The node has 62g memory and the cluster has nearly 200tb still free

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Jonathan Featherston]

Is the ulimited stack size needed for Trinity 2?

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Will Holtz]

What is the output from the command ulimit?

-Will

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed.  If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited.  If you receive this e-mail message in error, please e-mail the sender at who...@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers).

Please consider the environment before printing this email.

[Jonathan Featherston]

Hi. Well I guess the same either way actually.

[Jonathan Featherston]

I'm going to try recompiling trinity again. I realise now that I didn't source the bash profile or restart after I added the export to java 8. It may not have compiled properly I suppose although no errors were given.

[Jonathan Featherston]

Unfortunately this didn't work. I get the same error.

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Will Holtz]

I'm not sure if you were replying to my message. I don't think the ulimited stack size is your problem, but I do belive the ulimited max user processes is the source of your problem. You can try removing this limit with: ulimit -u unlimited.

-Will

On Tue, Apr 28, 2015 at 10:07 AM, Jonathan Featherston <feathe...@arc.agric.za> wrote:

Hi. Well I guess the same either way actually.

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Featherston]

Hi Will

Thanks. Yes sorry that's what I have done. Google let me believe that the ulimit command was affecting the stack size (behind the screen issue). Either way I have run it with and without this command and still get the jellyfish error.

Regards

Jonathan 

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Jonathan Featherston]

Wait sorry. Yes I have run ulimit -s unlimited. Will try again with u instead. 

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Jonathan Featherston]

But of course I need root access to do this. Will try with admin guys tomorrow. Thanks. a Resource error might hint at something like this.

On Tuesday, April 28, 2015 at 4:20:13 PM UTC+2, Jonathan Featherston wrote:

[Brian Haas]

sorry - getting into the conversation late.  This is a thread-related issue.  Try reducing the number of threads you're asking for (try --CPU 4 instead of 10) and see if that helps.

If your machine has a lot of cores, then I'm not sure why the multithreading would be an issue, unless you're competing with a lot of others for the same resources.

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas