restart trinity after wall time ends

[LK]

Hi, 

Is it possible for Trinity to pick up where it left off? We maxed out on wall time on our local server before Trinity was finished. 

The last completed tasks were:

-rw-r--r-- 1  190M Jul 11 12:48 recursive_trinity.cmds.completed

-rw-r--r-- 1  286M Jul  3 16:38 recursive_trinity.cmds

Thanks!

[Tiago Hori]

Not exactly where it stopped. There are several check point files that Trinity created, including the ones you list. Trinity will pick-up at the latest compete check point if you run the same command pointing to the same output folder.

T.

"Profanity the is the only language all programmers understand" 

Sent from my iPhone, the universal excuse for my poor spelling.

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Lydia Kapsenberg]

Hi Tiago,

Thanks for your quick reply!

Should we delete this file: 

-rw-r--r-- 1  190M Jul 11 12:48 recursive_trinity.cmds.completed

Since it is not the same size at the last completed check point:

rw-r--r-- 1  286M Jul  3 16:38 recursive_trinity.cmds

...before submitting the original job file again?

Lydia

[Tiago Hori]

No need. It will replace everything following the check point it uses. 

Unless something died in the middle of a very peculiar point, which sometimes can create problems. Most times, just re-running it does the trick.

T.

"Profanity the is the only language all programmers understand" 

Sent from my iPhone, the universal excuse for my poor spelling.

[Lydia Kapsenberg]

We'll give it a go. Thank you!

Lydia

[Lydia]

It looks like starting Trinity from where it left on did not work for us. 

Specifically, in: Trinity Phase 2: Assembling Clusters of Reads

We got the following error:

WARNING, cannot remove output directory /programs/trinityrnaseq-2.0.6/trinity_good_out/read_partitions/Fb_4/CBin_4580/c458084.trinity.reads.fa.out, since not created in this run. (safety precaution)

[Brian Haas]

Hi Lydia

That's just a warning. It should be fine.  You'll just have a few extra files left over when it finishes, which you can delete yourself afterwards.

best,

~brian

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at http://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Lydia Kapsenberg]

Hi Brian, 

I think the issue had to do with restarting trinity. When we resubmitted the job, it looks like the total number of CPUs being used was doubled from before. Trinity was running very slow.

The original run ended by our computing cluster (running out of time) at:

-rw-r--r-- 1  190M Jul 11 12:48 recursive_trinity.cmds.completed

-rw-r--r-- 1  286M Jul  3 16:38 recursive_trinity.cmds

We resubmitted the same job but removed the last incomplete step (recursive_trinity.cmds.completed). When trinity started again it got to 190 M in about 1 week (which is what we expected). Following that, it slowed down to about 1 M every 48 hours. Looking at the cores being used for this job, trinity was using between 60 and 80 cores, but we only have 32 available. Is there a way we can control how many CPUs are used when we restart? 

Our code is:

perl Trinity --seqType fq --max_memory 1000G --left ~/mcali_1.fq  --right ~/mcali_2.fq --SS_lib_type RF --CPU 32 -output trinity_out/    

Thanks for your help.

Lydia