Re: Trinity run failed

68 views
Skip to first unread message

Baradwaj

unread,
Aug 9, 2024, 8:30:03 PM8/9/24
to trinityrn...@googlegroups.com, Brian Haas
Hello Brian.

The trinity run failed during phase 2.

There are 3 logs files attached with this email.
The first log was generated when the Trinity run was ended by mistake.
Trinity_Run1.log

The second log shows some errors generated during Phase 2, but the run continued and finally failed.
Trinity_Run2.log

The last few lines of the log are as follows.
---------------------------------------------------------------------------------------------------
succeeded(15087), failed(1)   99.9969% completed.    
succeeded(15088), failed(1)   99.9974% completed.    
succeeded(15089), failed(1)   99.9978% completed.    
succeeded(15090), failed(1)   99.9982% completed.    
succeeded(15091), failed(1)   99.9987% completed.    
succeeded(15092), failed(1)   99.9991% completed.    
succeeded(15093), failed(1)   99.9996% completed.    
succeeded(15094), failed(1)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 6 -v -shuffle  died with ret 256 at /usr/local/bin/Trinity line 2919.
main::process_cmd("/usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trini"...) called at /usr/local/bin/Trinity line 3653
main::run_partitioned_cmds("recursive_trinity.cmds") called at /usr/local/bin/Trinity line 2528
main::run_recursive_trinity("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/chrysa"...) called at /usr/local/bin/Trinity line 2271
main::run_chrysalis("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/inchwo"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", 200, 500, undef, "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa") called at /usr/local/bin/Trinity line 1895
main::run_Trinity() called at /usr/local/bin/Trinity line 1500
eval {...} called at /usr/local/bin/Trinity line 1499

Trinity run failed. Must investigate error above.
---------------------------------------------------------------------------------------------------



The 3rd log was generated when the run was restarted. 
Trinity_Run3.log

The last few lines of the log are as follows.
---------------------------------------------------------------------------------------------------------
Error encountered::  <!----
CMD: /usr/local/bin/Inchworm/bin//inchworm --reads /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa --run_inchworm -K 18 --monitor 1   --DS  --num_threads 1  -L 50  --make_super_reads  > /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/inchworm.DS.fa.SR.18.tmp 2>tmp.657097.1723171038.stderr

Errmsg:
Kmer length set to: 18
Min assembly length set to: 50
Running Inchworm in Super-Reads mode
Monitor turned on, set to: 1
double stranded mode set
setting number of threads to: 1
-storing Kmers...
 <stacktrace off>

Error, cannot open file /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa

--->

Error, cmd: /usr/local/bin/Inchworm/bin//inchworm --reads /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa --run_inchworm -K 18 --monitor 1   --DS  --num_threads 1  -L 50  --make_super_reads  > /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/inchworm.DS.fa.SR.18.tmp 2>tmp.657097.1723171038.stderr died with ret 256  at /usr/local/bin/PerlLib/Pipeliner.pm line 187.
Pipeliner::run(Pipeliner=HASH(0x556f155e54c0)) called at /usr/local/bin/util/support_scripts/../../Trinity line 2729
eval {...} called at /usr/local/bin/util/support_scripts/../../Trinity line 2719
main::run_inchworm("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_p"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_p"..., undef, "", 18, 1) called at /usr/local/bin/util/support_scripts/../../Trinity line 1784
main::run_Trinity() called at /usr/local/bin/util/support_scripts/../../Trinity line 1500
eval {...} called at /usr/local/bin/util/support_scripts/../../Trinity line 1499



If it indicates bad_alloc(), then Inchworm ran out of memory.  You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.

** The inchworm process failed.warning, cmd: /usr/local/bin/util/support_scripts/../../Trinity --single "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa" --output "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanup --no_salmon   failed with ret: 256, going to retry.

succeeded(0), failed(1)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 6 -v -shuffle  died with ret 256 at /usr/local/bin/Trinity line 2919.
main::process_cmd("/usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trini"...) called at /usr/local/bin/Trinity line 3653
main::run_partitioned_cmds("recursive_trinity.cmds") called at /usr/local/bin/Trinity line 2528
main::run_recursive_trinity("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/chrysa"...) called at /usr/local/bin/Trinity line 2271
main::run_chrysalis("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/inchwo"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", 200, 500, undef, "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa") called at /usr/local/bin/Trinity line 1895
main::run_Trinity() called at /usr/local/bin/Trinity line 1500
eval {...} called at /usr/local/bin/Trinity line 1499

Trinity run failed. Must investigate error above.
----------------------------------------------------------------------------------------------------------


kindly help!

With best regards,

Baradwaj

    

    





On Thu, Aug 8, 2024 at 11:08 PM Baradwaj <viveka...@gmail.com> wrote:
Hello Brian.

The trinity run failed during phase 2.

There are 3 logs files attached with this email.
The first log was generated when the Trinity run was ended by mistake.
Trinity_Run1.log

The second log shows some errors generated during Phase 2, but the run continued and finally failed.
Trinity_Run2.log

The last few lines of the log are as follows.
---------------------------------------------------------------------------------------------------
succeeded(15087), failed(1)   99.9969% completed.    
succeeded(15088), failed(1)   99.9974% completed.    
succeeded(15089), failed(1)   99.9978% completed.    
succeeded(15090), failed(1)   99.9982% completed.    
succeeded(15091), failed(1)   99.9987% completed.    
succeeded(15092), failed(1)   99.9991% completed.    
succeeded(15093), failed(1)   99.9996% completed.    
succeeded(15094), failed(1)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 6 -v -shuffle  died with ret 256 at /usr/local/bin/Trinity line 2919.
main::process_cmd("/usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trini"...) called at /usr/local/bin/Trinity line 3653
main::run_partitioned_cmds("recursive_trinity.cmds") called at /usr/local/bin/Trinity line 2528
main::run_recursive_trinity("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/chrysa"...) called at /usr/local/bin/Trinity line 2271
main::run_chrysalis("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/inchwo"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", 200, 500, undef, "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa") called at /usr/local/bin/Trinity line 1895
main::run_Trinity() called at /usr/local/bin/Trinity line 1500
eval {...} called at /usr/local/bin/Trinity line 1499

Trinity run failed. Must investigate error above.
---------------------------------------------------------------------------------------------------



The 3rd log was generated when the run was restarted. 
Trinity_Run3.log

The last few lines of the log are as follows.
---------------------------------------------------------------------------------------------------------
Error encountered::  <!----
CMD: /usr/local/bin/Inchworm/bin//inchworm --reads /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa --run_inchworm -K 18 --monitor 1   --DS  --num_threads 1  -L 50  --make_super_reads  > /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/inchworm.DS.fa.SR.18.tmp 2>tmp.657097.1723171038.stderr

Errmsg:
Kmer length set to: 18
Min assembly length set to: 50
Running Inchworm in Super-Reads mode
Monitor turned on, set to: 1
double stranded mode set
setting number of threads to: 1
-storing Kmers...
 <stacktrace off>

Error, cannot open file /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa

--->

Error, cmd: /usr/local/bin/Inchworm/bin//inchworm --reads /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/single.fa --run_inchworm -K 18 --monitor 1   --DS  --num_threads 1  -L 50  --make_super_reads  > /media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/inchworm.DS.fa.SR.18.tmp 2>tmp.657097.1723171038.stderr died with ret 256  at /usr/local/bin/PerlLib/Pipeliner.pm line 187.
Pipeliner::run(Pipeliner=HASH(0x556f155e54c0)) called at /usr/local/bin/util/support_scripts/../../Trinity line 2729
eval {...} called at /usr/local/bin/util/support_scripts/../../Trinity line 2719
main::run_inchworm("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_p"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_p"..., undef, "", 18, 1) called at /usr/local/bin/util/support_scripts/../../Trinity line 1784
main::run_Trinity() called at /usr/local/bin/util/support_scripts/../../Trinity line 1500
eval {...} called at /usr/local/bin/util/support_scripts/../../Trinity line 1499



If it indicates bad_alloc(), then Inchworm ran out of memory.  You'll need to either reduce the size of your data set or run Trinity on a server with more memory available.

** The inchworm process failed.warning, cmd: /usr/local/bin/util/support_scripts/../../Trinity --single "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa" --output "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanup --no_salmon   failed with ret: 256, going to retry.

succeeded(0), failed(1)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trinity.cmds -CPU 6 -v -shuffle  died with ret 256 at /usr/local/bin/Trinity line 2919.
main::process_cmd("/usr/local/bin/trinity-plugins/BIN/ParaFly -c recursive_trini"...) called at /usr/local/bin/Trinity line 3653
main::run_partitioned_cmds("recursive_trinity.cmds") called at /usr/local/bin/Trinity line 2528
main::run_recursive_trinity("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/chrysa"...) called at /usr/local/bin/Trinity line 2271
main::run_chrysalis("/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/inchwo"..., "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", 200, 500, undef, "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa", "/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/both.fa") called at /usr/local/bin/Trinity line 1895
main::run_Trinity() called at /usr/local/bin/Trinity line 1500
eval {...} called at /usr/local/bin/Trinity line 1499

Trinity run failed. Must investigate error above.
----------------------------------------------------------------------------------------------------------


kindly help!

With best regards,

Baradwaj

    

    



Trinity_Run_logs.zip

Brian Haas

unread,
Aug 9, 2024, 8:30:19 PM8/9/24
to Baradwaj, trinityrn...@googlegroups.com
Hi,

It looks like there might have been a hiccup somewhere that corrupted one of the sub-jobs.

If you remover this directory:
/media/baradwaj/My_Volume/TRINITY/Gymnema/Trinity_Runs/read_partitions/Fb_1/CBin_1449/c146444.trinity.reads.fa.out/

and then rerun your original Trinity command, it should retry that one job from scratch. If it completes ok, it'll finish the rest of the job and give you your final Trinity fasta file.

best,

Brian
--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

Baradwaj

unread,
Aug 10, 2024, 8:15:24 PM8/10/24
to Brian Haas, trinityrn...@googlegroups.com
Grear! Thank you, Brian. It worked like a charm :)

With best regards,

Baradwaj

    

    





Brian Haas

unread,
Aug 10, 2024, 8:15:44 PM8/10/24
to Baradwaj, trinityrn...@googlegroups.com
excellent!!
Reply all
Reply to author
Forward
0 new messages