grid=LSF
cmd=bsub -q regevlab -R "rusage[mem=10]"
mount_test=T
max_nodes=500
cmds_per_node=100
cmd=bsub -I -n 20 -R "select[nxt] span[ptile=20] rusage[mem=60000]" -M 3000
--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.
<2015.03.17_Trinity_Grid_LSF_Testing_1.txt>
cmd=bsub -n 20 -R "select[nxt] span[ptile=20] rusage[mem=60000]" -M 5000And that worked as well!
Brian and Dan: Thanks for having this discussion. This post has been most helpful for speeding up Phase 2 (Butterfly) on our cluster. We are struggling to get our metatranscriptome assembly to finish before getting kicked off the "week" queue. So want to utilize the massively parallel option. I'm trying to figure out how to tune it to work in our system.Following along with this post, I have been able to get our cluster to accept the --grid_conf configuration file for our data.grid=LSFcmd=bsub -q hour -M 10 -n 8 -R "span[ptile=8]"mount_test=Tmax_nodes=20cmds_per_node=1000This is the overarching job,bsub -q week -n 8 -R "span[hosts=1]" -M 128 Trinity --seqType fq --max_memory 128G \--normalize_reads --left ${R1} --right ${R2} \--grid_conf ./grid.conf --CPU 8I thought that this would run at least 8 commands from the generated shell (with 1000 commands, e.g. J13795.S0.sh) at a given time on one node. However, using top on the given node, it shows one command running and never see other CPU/core utilization. It is a little confusing because it is one bsub job firing off other bsubs.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND27685 haines 25 0 89932 8148 1952 S 1.7 0.0 0:00.05 perl27786 haines 25 0 402m 60m 1356 R 0.3 0.1 0:00.01 GraphFromFasta10308 haines 15 0 19780 2316 1340 S 0.0 0.0 0:00.00 res10321 haines 17 0 65964 1164 964 S 0.0 0.0 0:00.00 1429982431.350810323 haines 16 0 65968 1236 988 S 0.0 0.0 0:00.24 J13795.S0.sh10465 haines 15 0 91044 1812 1036 S 0.0 0.0 0:00.04 sshd10469 haines 18 0 68320 1800 1300 S 0.0 0.0 0:00.01 bash10610 haines 15 0 13032 1388 848 R 0.0 0.0 0:00.23 top27785 haines 25 0 65964 1152 948 S 0.0 0.0 0:00.00 shWhereas when we run without --grid_conf on a given node, it utilizes more of the CPUs requested
Also, how does grid_conf help chrysalis? (I started this on our data after getting thru chrysalis.)
Dan wrote:cmd=bsub -n 20 -R "select[nxt] span[ptile=20] rusage[mem=60000]" -M 5000And that worked as well!Dan: How did you get yours to run parallel CPUs on a given node?BTW, we are using Trinity 2.0.6.Thanks so much again for all the previous details. We are so much farther along because them.s
--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.
grid=LSFcmd=bsub -q hour -M 10 -n 5 -R "span[ptile=5]"mount_test=Tmax_nodes=500cmds_per_node=20
Trinity --seqType fq --max_memory 128G --normalize_reads --left ${R1} --right ${R2} --CPU 8 --grid_conf ./grid.conf --grid_node_CPU 5 --grid_node_max_memory 10G
--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.
${TRINITY_HOME}/Trinity --seqType fq --max_memory 2G --left reads.left.fq.gz --right reads.right.fq.gz --SS_lib_type RF --CPU 8 --grid_exec "$GRID_SCRIPT" --output /anno/sanyalab/TESTAREA/trinity_test --full_cleanup > test_trinity.out 2>test_trinity.errGRID_SCRIPT="$TRINITY_HOME/trinity-plugins/HpcGridRunner-1.0.2/hpc_cmds_GridRunner.pl --grid_conf $TRINITY_HOME/trinity-plugins/HpcGridRunner-1.0.2/hpc_conf/Phi_LSF.100.conf -c"# grid type:
grid=LSF
# template for a grid submission
cmd=bsub -q prod -P ti-assembly-txpt -M 20 -R "rusage[mem=20,scr=100]"
# note -e error.file -o out.file are set internally, so dont set them in the above cmd.
# uses the LSF feature to pre-exec and check that the file system is mounted before executing.
# this helps when you have some misbehaving grid nodes that lost certain file mounts.
mount_test=T
##########################################################################################
# settings below configure the Trinity job submission system, not tied to the grid itself.
##########################################################################################
# number of grid submissions to be maintained at steady state by the Trinity submission system
max_nodes=200
# number of commands that are batched into a single grid submission job.
cmds_per_node=50Dan - it seems you've got this working pretty well. In case you weren't aware, we have:so you can use the grid dispatch system for all sorts of general computes.best,~brian
On Wed, Jun 10, 2015 at 11:03 AM, Dan Browne <dbrow...@gmail.com> wrote:
Hey Sara,
Sorry for the really delayed response! Maybe you've already figured out some of these questions, but I'll chime in anyways just in case.
The --grid_conf option isn't going to make a huge difference to the processes of the main job that you're running, it's going to farm out the processing for stage 2 in lots of little chunks, submitting independent jobs to the cluster through LSF. You'll be able to see this when it's happening with the "bjobs" command.
You'll want to be sure to coordinate the cmd template in your grid.conf file with the --grid_node_CPU and --grid_node_max_memory options in the overarching command. For example, if your grid.conf file looks like:grid=LSFcmd=bsub -q hour -M 10 -n 5 -R "span[ptile=5]"mount_test=Tmax_nodes=500cmds_per_node=20
Then you'll want your overarching Trinity command to look like:Trinity --seqType fq --max_memory 128G --normalize_reads --left ${R1} --right ${R2} --CPU 8 --grid_conf ./grid.conf --grid_node_CPU 5 --grid_node_max_memory 10G
As Brian mentioned, it's better than have more max_nodes and fewer cmds_per_node. This will really take advantage of the parallel nature of the cluster and dramatically reduce your assembly time. I can start and finish assemblies in 5-8 hours utilizing the --grid_conf option on our cluster. I'm not sure how big your cluster is, but you should try to use as many nodes as you can. For example, our cluster has about 700 nodes with 20 cores each. I usually have my grid.conf file set very similarly to what I've described above. The command template submits jobs requesting 5 cores at a time, running 20 commands per job with the 5 cores running in parallel. Many 5-core jobs are submitted in parallel - I think the most cores I've been using in parallel at one time was about 4,000 (out of the roughly 14,000 available on our cluster).
Hope this helps!
Dan
--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.