ParaFly: died with ret 256 Trinity line 2116

[thomas...@colorado.edu]

Hi all,

I am running into an issue at the end of Trinity. Things look like they are about to finish correctly, but then fails with threading or something. I am getting the same problem with another run, and I have tried with the default bflyGCThreads and HeapSpaceMax. Here is the command and the last bit of the log:

Command:

/home1/03564/tg828633/trinityrnaseq-2.0.6/Trinity --verbose --seqType fq --max_memory 700G --left /scratch/03564/tg828633/ISG_FB_assemblies/rhe/rf_paired_qual_1.fastq.normalized_K25_C50_pctSD200.fq --right /scratch/03564/tg828633/ISG_FB_assemblies/rhe/rf_paired_qual_2.fastq.normalized_K25_C50_pctSD200.fq --CPU 32 --bflyGCThreads 7 --bflyHeapSpaceMax 50G --output /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/

log:

-- (Integrate mapped reads into de Bruijn graph) --

---------------------------------------------------

Fully cleaning up.

succeeded(499055), failed(12)   99.9998% completed.    CMD finished (0 seconds)

CMD finished (0 seconds)

Butterfly_cmds: /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4990/c499067.trinity.reads.fa.out/chrysalis/butterfly_commands

Inchworm and Chrysalis complete.  Butterfly commands to execute are provided here:

/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4990/c499067.trinity.reads.fa.out/chrysalis/butterfly_commands

---------------------------------------------------------------

-------------------- Butterfly --------------------------------

-- (Reconstruct transcripts from reads and de Bruijn graphs) --

---------------------------------------------------------------

CMD finished (0 seconds)

CMD finished (0 seconds)

Fully cleaning up.

succeeded(499056), failed(12)   100% completed.    

We are sorry, commands in file: [FailedCommands] failed.  :-( 

Error, cmd: /home1/03564/tg828633/trinityrnaseq-2.0.6/trinity-plugins/parafly/bin/ParaFly -c recursive_trinity.cmds -CPU 32 -v  died with ret 256 at /home1/03564/tg828633/trinityrnaseq-2.0.6/Trinity line 2116.

Trinity run failed. Must investigate error above.

Failed Commands:

/home1/03564/tg828633/trinityrnaseq-2.0.6/util/support_scripts/../../Trinity --single "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa" --output "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out" --CPU 1 --max_memory 1G --full_cleanup --seqType fa --trinity_complete --verbose --bflyGCThreads 7 --bflyHeapSpaceMax 50G  

/home1/03564/tg828633/trinityrnaseq-2.0.6/util/support_scripts/../../Trinity --single "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475699.trinity.reads.fa" --output "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475699.trinity.reads.fa.out" --CPU 1 --max_memory 1G --full_cleanup --seqType fa --trinity_complete --verbose --bflyGCThreads 7 --bflyHeapSpaceMax 50G  

/home1/03564/tg828633/trinityrnaseq-2.0.6/util/support_scripts/../../Trinity --single "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475695.trinity.reads.fa" --output "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475695.trinity.reads.fa.out" --CPU 1 --max_memory 1G --full_cleanup --seqType fa --trinity_complete --verbose --bflyGCThreads 7 --bflyHeapSpaceMax 50G...

If anyone knows what might be causing the trouble, your help would be greatly appreciated.

Best,

Tommy

[Fulton, Ben]

Have you tried simply restarting the run? If there is an issue with threading or memory, restarting should pick up with only the failed commands and try to run them again.

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[thomas...@colorado.edu]

Thanks for the Reply, Ben.

I already tried restarting both runs that gave me this error. That did not fix the issue. The output is identical to the one above (aside from skipping the steps it already performed and the number of successful/failed reads).

Tommy

[Brian Haas]

Hi

When we've encountered this most recently it involved control-M chars in the reads file.

Can you send us the file

/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa

so we can take a peek!

Thx!

-Brian
(by iPhone)


> On Jan 21, 2016, at 12:48 AM, thomas...@colorado.edu wrote:
>
> /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa

[Fulton, Ben]

Try running one of the failed commands by itself. It would look like

 

Trinity --single "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa" --output "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out" --CPU 1 --max_memory 1G --full_cleanup --seqType fa --trinity_complete --verbose --bflyGCThreads 7 --bflyHeapSpaceMax 50G

 

I’m guessing it will fail again, but it may give some useful output explaining why.

 

--

Ben Fulton

Research Technologies

Scientific Applications and Performance Tuning

Indiana University

E-Mail: befu...@iu.edu

--

[thomas...@colorado.edu]

This should be what you are looking for.

Tommy

[thomas...@colorado.edu]

here is the command/output:

/home1/03564/tg828633/trinityrnaseq-2.0.6/Trinity --single "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa" --output "/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out" --CPU 1 --max_memory 1G --full_cleanup --seqType fa --trinity_complete --verbose --bflyGCThreads 7 --bflyHeapSpaceMax 50G 

-since butterfly will eventually be run, lets test for proper execution of java

#######################################

Running Java Tests

CMD finished (0 seconds)

-we properly captured the java failure status, as needed.  Looking good.

Java tests succeeded.

###################################

#######################################################################

Inchworm file: /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out/inchworm.K25.L25.DS.fa detected.

Skipping Inchworm Step, Using Previous Inchworm Assembly

#######################################################################

Butterfly_cmds: /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out/chrysalis/butterfly_commands

Inchworm and Chrysalis complete.  Butterfly commands to execute are provided here:

/scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out/chrysalis/butterfly_commands

---------------------------------------------------------------

-------------------- Butterfly --------------------------------

-- (Reconstruct transcripts from reads and de Bruijn graphs) --

---------------------------------------------------------------

CMD finished (0 seconds)

CMD finished (0 seconds)

Fully cleaning up.

WARNING, cannot remove output directory /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa.out, since not created in this run. (safety precaution)

Log file:

drwx------  3 tg828633 G-816421  4096 Jan 22 18:39 ./

drwx------ 14 tg828633 G-816421 20480 Jan 22 18:39 ../

drwx------  3 tg828633 G-816421  4096 Jan 22 17:48 chrysalis/

-rw-------  1 tg828633 G-816421   335 Jan 20 10:48 inchworm.K25.L25.DS.fa

-rw-------  1 tg828633 G-816421     0 Jan 20 10:48 inchworm.K25.L25.DS.fa.finished

lrwxrwxrwx  1 tg828633 G-816421   109 Jan 20 10:48 single.fa -> /scratch/03564/tg828633/ISG_FB_assemblies/rhe/trinity/read_partitions/Fb_4/CBin_4756/c475677.trinity.reads.fa

-rw-------  1 tg828633 G-816421     2 Jan 20 10:48 single.fa.read_count

-rw-------  1 tg828633 G-816421   289 Jan 20 10:48 tmp.iworm.fa.pid_92633.thread_0

Thank you again for your help

Tommy

[Brian Haas]

Yes, it's the cntrl-M problem (screenshot attached from cat -te file.fq | less)

There is a recent thread on our google group that describes how to get around this:

https://groups.google.com/forum/#!searchin/trinityrnaseq-users/completed$203$20days/trinityrnaseq-users/1OuDRTPv8eo/7xY8OyJiEAAJ

best,

~b 

[thomas...@colorado.edu]

I followed the instructions on the linked thread and everything worked! 

Thanks,

Tommy

[Brian Haas]

Fantastic!  Thanks for the update!

~brian

--

You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Kyle Martin]

Hi All,

I seem to have encountered a similar issue.  In my case, two commands failed originally:

[--verbose output]

succeeded(333365), failed(2)   100% completed.

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /home/bo1xkjm/Programs/trinityrnaseq-2.2.0/trinity-plugins/parafly/bin/ParaFly -c recursive_trinity.cmds -CPU 10 -v  died with ret 256 at /home/bo1xkjm/Programs/trinityrnaseq-2.2.0/Trinity line 2206.

Trinity run failed. Must investigate error above.

------------------

The ctrlM fix reported here solved one of them.  Now, only one command has failed:

From --verbose output following rerun after CtrlM fix:

Trinity run failed. Must investigate error above.

succeeded(1), failed(1)   100% completed.

We are sorry, commands in file: [FailedCommands] failed.  :-(

Error, cmd: /home/bo1xkjm/Programs/trinityrnaseq-2.2.0/trinity-plugins/parafly/bin/ParaFly -c recursive_trinity.cmds -CPU 10 -v  died with ret 256 at /home/bo1xkjm/Programs/trinityrnaseq-2.2.0/Trinity line 2206.

Trinity run failed. Must investigate error above.

---------------------------

From FailedCommands File:

---------------------------

/home/bo1xkjm/Programs/trinityrnaseq-2.2.0/util/support_scripts/../../Trinity --single "/mnt/fastdata/bo1xkjm/SDGS_sequencing_data/Trin_Test_Full_molecol/Scan_full_assembly_trinity_Q20_Dust7_derep14_v3/read_partitions/Fb_0/CBin_766/c76663.trinity.reads.fa" --output "/mnt/fastdata/bo1xkjm/SDGS_sequencing_data/Trin_Test_Full_molecol/Scan_full_assembly_trinity_Q20_Dust7_derep14_v3/read_partitions/Fb_0/CBin_766/c76663.trinity.reads.fa.out" --CPU 1 --max_memory 1G --seqType fa --trinity_complete --full_cleanup --verbose  

-------------------------

I believe that I applied the fix faithfully to both of the subjobs that failed originally. 

Can anyone suggest what I should try next to discover the problem?

Thanks,

Kyle 

On Friday, January 29, 2016 at 6:57:56 PM UTC, Brian Haas wrote:

Fantastic!  Thanks for the update!

~brian

On Fri, Jan 29, 2016 at 1:56 PM, <thomas...@colorado.edu> wrote:

I followed the instructions on the linked thread and everything worked! 

Thanks,

Tommy

On Sunday, January 24, 2016 at 6:12:10 AM UTC-7, Brian Haas wrote:

Yes, it's the cntrl-M problem (screenshot attached from cat -te file.fq | less)

There is a recent thread on our google group that describes how to get around this:

https://groups.google.com/forum/#!searchin/trinityrnaseq-users/completed$203$20days/trinityrnaseq-users/1OuDRTPv8eo/7xY8OyJiEAAJ

best,

~b 

--

You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

Hi Kyle,

If you just rerun your original Trinity command, will it complete those two failed commands and wrap up from there?

best,

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--

You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Adriana Romero]

Worked for me too!!! I'm using version 2.4.0. 

At the end of first run I had 42 errors. I re-run it without making any changes. It corrected 41 errors but was still left with one. I was able to fix this one error following instructions from linked thread: 

# remove the existing output directory for just this one failed mini-job:

    rm -r /new-data/xs253/trinity_out_dir/read_partitions/Fb_2/CBin_2428/c242885.trinity.reads.fa.out

# remove the cntrl-M chars, creating a temp file

    cat /new-data/xs253/trinity_out_dir/read_partitions/Fb_2/CBin_2428/c242885.trinity.reads.fa | perl -lane 's/\cM//g; print;' > test.output.fa

# replace the original small read file with your temp file

    mv test.output.fa /new-data/xs253/trinity_out_dir/read_partitions/Fb_2/CBin_2428/c242885.trinity.reads.fa

[hanyan li]

Hi Brain, 

I got the similar errors that I have three files that couldn't go through and generate fa.out files. However, No matter I tried to remove them or tackle the cntrl-M problem, it doesn't work for me and keep generating the same output file. 

Here is the error command:/usr/local/miniconda3/opt/trinity-2.8.5/util/support_scripts/../../Trinity --sin

gle "/home/hanyan/transcript/insilf/test_trinity_outdir/read_partitions/Fb_2/CBi

n_2662/c266260.trinity.reads.fa" --output "/home/hanyan/transcript/insilf/test_t

rinity_outdir/read_partitions/Fb_2/CBin_2662/c266260.trinity.reads.fa.out" --CPU

 1 --max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanu

p --bflyCPU 20 --bflyHeapSpaceMax 10G  

/usr/local/miniconda3/opt/trinity-2.8.5/util/support_scripts/../../Trinity --sin

gle "/home/hanyan/transcript/insilf/test_trinity_outdir/read_partitions/Fb_0/CBi

n_217/c21782.trinity.reads.fa" --output "/home/hanyan/transcript/insilf/test_tri

nity_outdir/read_partitions/Fb_0/CBin_217/c21782.trinity.reads.fa.out" --CPU 1 -

-max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanup --

bflyCPU 20 --bflyHeapSpaceMax 10G  

And the three files. 

Thanks a lot, 

Hanyan

在 2016年1月24日星期日 UTC-7上午6:12:10，Brian Haas写道：

[Brian Haas]

Hi - thanks for sending the example.  The read cluster looks pretty large.    Can you share your original Trinity command?

--

You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/fbfdf006-0574-4916-806f-ca9c40782e6d%40googlegroups.com.

[Brian Haas]

This is the cause of the problem:  --no_normalize_reads

I definitely do not recommend this, unless it's under some special circumstances.  I should make this more clear in the documentation.  I'll do something about it before the next release.

Try rerunning your Trinity assembly job in a new output directory (so it doesn't try to reuse any earlier intermediate outputs).  Let's see how that goes.

best,

~brian

On Wed, Aug 21, 2019 at 6:11 PM hanyan li <liha...@gmail.com> wrote:

Hi Brian，

Thanks a lot for your reply! Here is my original Trinity command: 

Trinity --left left.norm.fq --right right.norm.fq --CPU 20 --max_memory 150G --seqType fq --no_normalize_reads --bflyCPU 20 --bflyHeapSpaceMax 10G --output test_trinity_outdir &

I have conducted normalization before this step and get around 3G of left and right file. I have run trinity assembly twice. The first time, five command fails and the second time still three fail. The three clusters looks larger than others. Please let me know if you have any idea about it.

Thanks, 

Hanyan 

Brian Haas <bh...@broadinstitute.org> 于2019年8月21日周三 下午2:35写道：