A few butterfly commands fail repeatedly

[Mauricio Losilla]

Hello,

I hope you are doing well. I am having difficulties finishing a Trinity assembly. A few butterfly commands fail over and over. Here is an example:

cat:  <long path>/read_partitions/Fb_2/CBin_2886/c290194.trinity.reads.fa.out/inchworm.DS.fa.SR.18: No such file or directory

Trinity run failed. Must investigate error above.

warning, cmd: /usr/local/bin/trinityrnaseq/util/support_scripts/../../Trinity --single "<long path>/read_partitions/Fb_2/CBin_2886/c290194.trinity.reads.fa" --output "<long path>/read_partitions/Fb_2/CBin_2886/c290194.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanup --min_kmer_cov 2 --no_distributed_trinity_exec   failed with ret: 6400, going to retry.

When I go into the folder c290194.trinity.reads.fa.out there is no file inchworm.DS.fa.SR.18, but there are (among many others) inchworm.DS.fa.SR.18.tmp & iworm_renamed.18.SR.ok (both have file size 0).

In posts with similar errors, it has been suggested to delete a folder and run Trinity again. I have not tried that because I am not sure what I should delete. This assembly has taken a very long time and I don't want to start over if possible.

I am running Trinity v2.11.0 from the singularity container. 

Thanks!

Mau

[Brian Haas]

Hi Mau,

It sounds like the may have been a hiccup with the filer.  For the commands that failed, just remove the corresponding directories named like: ".trinity.reads.fa.out/"  and then restart your Trinity run. It'll then recompute those subjobs from scratch.  This should solve the problem.

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/350ef4d2-fa04-40e7-a990-08529d276014n%40googlegroups.com.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[Mauricio Losilla]

Hi Brian,

thank you for your reply, I will try your suggestion. 

FWIW, I found this error in a handful of commands, and in every case the missing file is inchworm.DS.fa.SR.18

I just found a slightly different error:

Inchworm failed:

Error, cannot open file <long path>/read_partitions/Fb_2/CBin_2906/c292205.trinity.reads.fa.out/single.fa

For this error, I noticed that the file c292205.trinity.reads.fa.out.Trinity.fasta exists. The command has not been added to recursive_trinity.cmds.completed. 

Should I delete both this fasta file and the folder *trinity.reads.fa.out ?

thanks

Mau

[Brian Haas]

Hi Mau,

I'd only do this for the jobs that weren't able to complete.  Because these flaky filesystem issues can happen from time to time, Trinity will retry individual jobs ~ 10 times before fully giving up on them.   Once Trinity finishes, there should be a list of the jobs that were complete failures, and I'd just focus on those.  The filename will have 'failed' in the name (ignore case).

hth

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/92a2dee8-505d-429e-8645-6aa5dd43568fn%40googlegroups.com.

[Mauricio Losilla]

Thanks Brian. 

I can't find the file with the failed commands. Should it be in the output folder?

I just noticed something else too. Other than the commands that fail, others just hang for hours and hours. I deleted the .trinity.reads.fa.out/ folder from one of these, and monitored the progress when it ran again. It made fat progress for a few minutes, but then hanged again. The last file it wrote is: c11.trinity.reads.fa.out/chrysalis/c0.graph.allProbPaths.fasta, and also chrysalis/tmp.29118.1608404919.stderr  The contents of this file are:

Graph is empty. Quitting.

Graph is empty. Quitting.

Graph is empty. Quitting.

Also, the file butterfly_commands.completed has 47 lines, whereas these 3 files have 49: butterfly_commands, quantifyGraph_commands, quantifyGraph_commands.completed

I suspect this is happening with several commands.

Thanks
Mau

[Brian Haas]

Hi Mau

Are you using the latest version of Trinity?   If there are certain 'tough' clusters, that version should be better at handling them.  However, I was thinking this was more of the 'flaky file system' issue rather than a difficult-data problem.

For the job:  c11.trinity.reads.fa.out

Did you delete that whole directory  c11.trinity.reads.fa.out/ before rerunning the original Trinity job?

Any problem clusters will need to have this done, just targeting the fa.out/ directories.

The file containing the failed Trinity subjobs should show up in the main trinity_out_dir/, I believe.  That will only show up if the entire Trinity job finishes unsuccessfully. 

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/76ddb32d-eaa9-49db-b8dd-cab7e2b4f94fn%40googlegroups.com.

[Mauricio Losilla]

Hi Brian,

I am running Trinity v2.11.0 from the singularity container.

Yeah I every time I deleted the whole *trinity.reads.fa.out/ directory. 

I have not seen the failed commands file, but I guess technically Trinity hasn't finished running. It hangs for days in the aforesaid commands until my walltime expires. Then I run it again and again. I am only ~40 commands away now, so I will keep trying. 

There were two main problems I had with this assembly:

1) After running normalization with default parameters (normalization per sample and later all together) I ended up with 11 billion k-mers. Inchoworm wouldn't finish, so I added the flag --min_kmer_cov 2. K-mer count went down to 2.7 billion. 

2) During phase 2 I reached my file number quota, Trinity couldn't proceed for a while, until I got an extension from my HPC managers. 

If I were to rerun this assembly again, are there any settings I should choose, or do you have any suggestions?

Thanks
Mau

[Brian Haas]

How many reads are you ending up with post-normalization?  

The latest trinity is best at reducing long runtimes for troublesome components, but you might still have encountered some that are just going to require a lot of time to get through.  You can look at the size of those troublesome components to see if they're large.  If they're not, then it could be a data complexity issue (complex graphs).

If you can identify those troublesome commands, then you could target those corresponding Trinity reads fasta files separately with altered parameters.  You could wrap up the current assembly by running Trinity with --FORCE, and then try assembling the remaining read sets separately - play with --min_kmer_cov 3 to see if you can get those to build.

In the end, just make sure that the final Trinity accessions are unique (you don't want the small read set assemblies to collide with the major assembly).

hth

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/da04700a-eac2-4d5e-9ed8-bb6d84fbca8cn%40googlegroups.com.

[Mauricio Losilla]

Hi Brian,

I have 320 million reads post-normalization (both.fa.read_count). 

I have realized that the flag -min_kmer_cov applies to normalization (I thought it was only important during inchworm, so my assembly normalized with -min_kmer_cov 1 and ran inchworm with -min_kmer_cov 2). I don't know if this may explain why some graphs are so complex, but I started a new assembly in parallel that I think will soon catch up with the problematic assembly.

In addition to setting -min_kmer_cov 2 since the beginning, I also set --normalize_max_read_cov 50 (the website says the default is 50, but it seems to me that the default is 200).

The number of reads post-normalization is now 133.5 million (40% of the total of the original assembly). Does this sound OK?

Thanks
Mau

[Brian Haas]

I think the original settings should have been fine, but it'll be interesting to see what happens with the latest normalization results.

The documentation is out of date... it used to be max 50, but now is 200.  I'll update this.

I'd suggest just continuing to delete the .out/ directories corresponding to the failed subjobs, then just rerun your original command. It should pick up where it left off.

best,

~b

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/618f982c-a781-453f-9a1c-e8a5d33cd005n%40googlegroups.com.

[Mauricio Losilla]

Hi Brian,

I am still having this issue. 

Recap: Trinity v2.11.0. A few butterfly commands fail over and over. Trinity hangs on them for days but never completes them. In these cases, there is a file in the partition.trinity.reads.fa.out

folder called tmp*stderr whose contents are: "Graph is empty. Quitting.", but it never really quits. Also, inside the chrysalis folder for the problematic partitions, all the "quantifyGraph_commands" complete, but a few "butterfly_commands" never complete. 

I have deleted the partition.trinity.reads.fa.out folders and restarted the run many times, but the outcome is always the same. 

The source of my reads is RNA from the same tissue from 20 different diploid individuals. I wonder if high polymorphism in some loci is the cause of this? something similar has been discussed here: https://groups.google.com/g/trinityrnaseq-users/c/b2riNdP8Uyc/m/irypIS4cDwAJ

Thanks

Mau

[Brian Haas]

Hi Mau,

If you can privately send me the individual reads.fa files for the small jobs that aren't finishing, I'll give them a look.

best,

~brian

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/a0317127-784d-43fd-83cb-8d20c9f2ca5fn%40googlegroups.com.

[Brian Haas]

Hi Mau,

With a little parameter shifting I was able to get those few jobs to complete.  If you want, I could make a test singularity image for you to use and see if you can get the job to complete with it.

[Mauricio Losilla]

Hi Brian, 

Yeah that sounds wonderful! I would give that a try and report back here 

Thank you very much!

Mau

[Brian Haas]

ok, sounds good.  I'll have it soon.  Will be in touch.

[Brian Haas]

Here's your image, Mau:

https://data.broadinstitute.org/Trinity/TRINITY_SINGULARITY/TMP/trinityrnaseq.v2.12.0-devmau.simg

Let's see how it goes.   You just need to resume your earlier job and it'll pick up the handful that didn't complete earlier.  It might still take a while (like an hour), but should hopefully complete ok.

best,

~brian