How to coordinate --JM and -CPU in Trinity with memory, cpu and nodes request from HPC?

[Chun Su]

Hi,

I have tested Trinity with different grid systems with varied settings. Some works but some does not. It looks like only Butterfly can run across different nodes which can be defined by conf_file.txt, while inchworm and chrysalis can only run on one node. 

So for example, I have a fastq input with ~ 200M reads. I need 200G for -JM. On SLURM grid, each node has 16 cores. Each core has 16G RAM. So my script would be

#!/bin/bash

#SBATCH -n 16

#SBATCH -N 1

#SBATCH --cpus-per-task=1

#SBATCH -p gpu

#SBATCH -t 2-00:00:00

#SBATCH --account='A'

#SBATCH --job-name=Trinity

/home/cs3jd/software/trinityrnaseq_r20140717/Trinity --seqType fq \

        --JM 200G \

        --single all.trimmed.fasta  \

        --SS_lib_type R  \

         --CPU 16 \

        --output Trinity_output \

        --grid_conf conf_file.txt

The job was killed because "terminate called after throwing an instance of 'std::bad_alloc'   what():  std::bad_alloc" at jellyfish step. I assumed this is out of memory. 

I remember some mailing tread mentioned --JM * -CPU is what physical computer should have. Then I change --CPU 16 to 1 while keep nothing else unchanged. Error report " terminate called after throwing an instance of 'jellyfish::thread_exec::Error'

  what():  Can't create thread: Resource temporarily unavailable" . Does it mean HPC cannot allocate resource my me just at this moment? If I try later, maybe it gonna work?

However, when setting --CPU 1, there is another problem. When it comes to chrysalis that taking most of running time, it will only run on core. It takes forever to finish. I had to change -CPU back to 16 after inchworm was done. It still took 4 days on this phase.

So I struggled on coordinating Trinity with HPC setting. I become really confused. I have submitted the same job to XSEDE blacklight. Although it is still in queue, I am wondering that the following setting will get my job killed or not.

#!/bin/bash

#PBS -l ncpus=32

#PBS -l walltime=96:00:00

#PBS -j oe Trinity-PBS-output-file

#PBS -q batch

ja

module load trinity/r2014-07-17

ulimit -s unlimited

cd $SCRATCH

Trinity --seqType fq --JM 200G \

        --single $SCRATCH/all.trimmed.fasta  \

        --SS_lib_type R 

        --CPU 32

        --output $SCRATCH/Trinity_output

# I did not put conf_file.txt since XSEDE has only 2 nodes, but each nodes have high RAM. no need to let butterfly running across nodes.

Thank you,

Chun

[Will Holtz]

The error messages "terminate called after throwing an instance of 'jellyfish::thread_exec::Error'  what():  Can't create thread: Resource temporarily unavailable" indicates that you have reached the maximum number of processes (threads) that you are allowed to have running at once. You may be able to raise this limit with 'ulimit -u unlimited', however this may require additional privileges on some systems.

-Will

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

The information contained in this e-mail message or any attachment(s) may be confidential and/or privileged and is intended for use only by the individual(s) to whom this message is addressed.  If you are not the intended recipient, any dissemination, distribution, copying, or use is strictly prohibited.  If you receive this e-mail message in error, please e-mail the sender at who...@lygos.com and destroy this message and remove the transmission from all computer directories (including e-mail servers).

Please consider the environment before printing this email.

[Philip Blood]

Hi Chun,

Check out the documentation here for an example of how to run Trinity correctly on Blacklight (this is for 2014 versions of Trinity):

http://www.psc.edu/index.php/user-resources/software/trinity

As documented there, you should load modules for both samtools (use 0.1.19) and bowtie and also set

ulimit -s unlimited

You should also set --bflyGCThreads=16, as some versions of Trinity have not limited the java GC threads, so java tries to launch GC on all cores of Blacklight (2048 per node).  

If you run Trinity 2.0+, you'll just need to change --JM to --max_memory.

Feel free to send email to rem...@psc.edu if you run into problems.

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.