Trinity resource usage monitoring in v2.4.0

[Brian Haas]

Greetings all,

For those interested in exploring Trinity resource usage, we have some additional monitoring capabilities that we've made more accessible in the latest release.  You can explore this here:

 https://github.com/trinityrnaseq/trinityrnaseq/wiki/Trinity-Runtime-Profiling

best,

~brian

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Mikhail]

Hi Brian!

I am currently trying to optimize Trinity and need to check how the resources are being used. So I just tried to use this runtime profiling system. I get really weird results (please see attached PDF file generated by examine_resource_usage_profiling.pl). Looks like no all the relevant processes are captured.

If necessary please find also trinity run output and .dat file generated by collectl

All the best,

Mikhail

[Brian Haas]

Hi Mikhail,

The reason the plot looks weird here is because the whole job only took a couple of minutes to run, and the monitoring is set to poll the system once a minute.

You can change the polling time via:

      --monitor_sec

parameter.  For this super short running job, you can set it at one second

      --monitor_sec  1

and hopefully the plots will make more sense.

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.
To post to this group, send email to trinityrnaseq-users@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Mikhail]

That was a test run involving very small assembly. So I created another test subset which takes couple of hours to assemble. 

This is how the resource usage looks like (attached).

What disturbs me is that though I requested 24 CPU and 100 GB of RAM there is no point when the resources are fully utilized. Can this mean that the assembly can be potentially sped up? Also my cluster quota is counted on the usage of 24 CPUs in this case  : - )) which is not cost-effective.

What if I use grid parallelization by providing --grid_exec script? Can this give any boost in performance?

To post to this group, send email to trinityrn...@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

The grid exec should make the 2nd phase run much faster.   Also, I think there are some processes that are not being captured that should be accounted for in the reporting.  I'll look into that shortly.

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Mikhail]

If there are some debug runs I can make on my system I am ready to help.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

Sounds good.  

I'd suggest cloning the 'devel' branch of Trinity, and then you can just do a git pull to update it once I've got further updates for you.

   git clone https://github.com/trinityrnaseq/trinityrnaseq.git

   git checkout devel

more later,

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

Hi Mikhail,

ParaFly is now included in the monitoring output, and most of what it's doing is running the 2nd phase trinity process, so it should reflect the number of 'active' 2nd phase trinity processes running concurrently.

I'll run a few tests on our end to see how things look with our test data sets.

more later,

~brian

[Mikhail]

Great! Now it is clear that ParaFly actually fully exploits the CPU resources.

It also clear that memory demands of GG assembly are really low and CPU is the bottleneck. I should try to use grid execution. I have a question concerning it though. Where the amount of grid jobs which will be submitted is specified? Is it related to --CPU parameter? 

-Mikhail

[Brian Haas]

Not bad.  Mine is attached w/ --CPU 10.  It didn't quite get to 10 - mostly around 5.  I'm trying again w/ --CPU 20 for comparison.   It might be that ParaFly could use some optimization wrt the multithreading settings.

Our info on configuring execution for a compute farm (LSF, SGE, SLURM, or PBS) is here:

https://github.com/trinityrnaseq/trinityrnaseq/wiki/Running%20Trinity#grid_conf

The --CPU settings won't matter when that's set, since the grid exec configuration will determine how jobs are batched, and your throughput on your compute farm will ultimately determine how much parallelization you can achieve.

~b

[Brian Haas]

I made a few more updates to better improve on the monitoring.

Here's what the latest (next release) reports will look like:

    https://github.com/trinityrnaseq/trinityrnaseq/wiki/Trinity-Runtime-Profiling

I'll put a couple more examples together for comparison.

best,

~b

[Brian Haas]

I added a plot for running our 50 M PE mouse data set through:

https://github.com/trinityrnaseq/trinityrnaseq/wiki/Trinity-Runtime-Profiling

There are still a few tweaks we should make to it, like putting the I/O on a log scale, but overall I think it's looking pretty good now.

best,

~brian

[Mikhail]

Here are the results for the big dataset. 

(Redundant lines in log file are removed to reduce size)

Look good. Here is what I noticed though:

1. Now very low level processes (like touch, which etc) are captured. They are not very informative.

2. In this genome guided run the first phase (which actually takes most of time to run) still does not seem to use the whole memory and CPU. Is there anything I can do to speed it up? 

BAM file parsing seems to be parallelizable process. 

Thanks,

Mikhail 

[Brian Haas]

Thanks, Mikhail!

The latest devel branch code for the monitoring should capture just the subset of trinity processes and not everything as shown here.  (Note, I don't recommend using the rest of the devel trinity code right now... just the monitoring code if you want it.  The other pieces there are under various stages of development).

The genome-guided setup needs to be optimized... it just hasn't been a huge priority and resources are tight.

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.
To post to this group, send email to trinityrnaseq-users@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

I should be able to get the parallel genome-guided prep step integrated in the next release.

~b