Trinity failed at Trinity Phase 2: Assembling Clusters of Reads

[Yunjia Liu]

Hi Brian,

I am running Trinity for a dataset included 63 libraries. I used the following setting, which worked for my previous similar dataset (same amount of libraries but different species). 

Trinity --seqType fq \
    --max_memory 50G \
    --samples_file $wk_DIR/Drosera.samples.txt \
    --CPU 20 \
    --min_contig_length 150 \
    --normalize_by_read_set \
    --min_kmer_cov 2 \
    --monitoring

But the Trinity process is stuck at Trinity Phase 2: Assembling Clusters of Reads. The following error message was attached. The same error messages are repeated several times, which seems Trinity is trying to pick it up. I wonder if I can have some help with this.

cat: /trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/inchworm.DS.fa.SR.18: No such file or directory
cat: /trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/inchworm.DS.fa.SR.20: No such file or directory
cat: /trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/inchworm.DS.fa.SR.22: No such file or directory
cat: /trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/inchworm.DS.fa.SR.24: No such file or directory
Trinity run failed. Must investigate error above.

Thank you!

Best,

Yunjia

[Brian Haas]

Hi,

Once the Trinity job completes, and assuming this was just a hiccup of
some form, you can

delete that failed directory
rm -rf /trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/

and then rerun your original Trinity command. It should pick up where
it left off and retry that job.


If it's a larger problem and all jobs are now failing, it could be
that you ran out of disk space.

hope this helps,

~b

> --
> You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/cfc9bb44-e8cd-4f89-817a-5e1620f16921n%40googlegroups.com.



--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

[Yunjia Liu]

Hi Brian,

Thank you for replying!

It seems the job was failing from the very beginning. "succeeded(0), failed(1) 5.55556% completed." I assumed it was not the problem I
had met before, as several reads failed. I also got the following error messages. I found there was a similar issue in the GitHub community. https://github.com/trinityrnaseq/trinityrnaseq/issues/1152
I don't think my data are metatranscriptomes, just have more conditions and samples. And I checked we should have > 5TB of free space. I wonder if there are some updates for this issue now? Thank you so much!

"/WORK/trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa" --output "/WORK/trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out" --CPU 1 --max_memory 1G --run_as_paired --seqType fa --trinity_complete --full_cleanup --min_contig_length 150 --min_kmer_cov 2 failed with ret: 6400, going to retry. 

best,

Yunjia

[Brian Haas]

Hi Yunjia,

It looks like there might be ~20 subjobs (presumably out of over a
hundred thousand) that may have this problem.

There should be a 'failed commands' file (or similar name) in the
trinity output directory. For each of these, you'd need to remove the
corresponding output directory (for example:
/trinity_Drosera/trinity_out_dir/read_partitions/Fb_1/CBin_1932/c193419.trinity.reads.fa.out/
)
but be sure to only be removing that fa.out/ directory and nothing
above it in the directory hierarchy.

Once those have been removed, you can rerun your original trinity
command and it should rerun those jobs. Assuming it all works fine,
it'll then pull together your final assembly fasta file.

best,

~b

> To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/85a679a3-6ffb-4844-8efb-c6541d58b33an%40googlegroups.com.

[Yunjia Liu]

Hi Brian, 

Thank you for replying. I figured it was the memory issue, so I reran the job on an unlimited space cluster. It went well this time. However, it got an error in the butterfly. 

We are sorry, commands in file: [failed_butterfly_commands.821092.txt] failed.  :-(

Error encountered::  <!----
CMD: /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/trinity-plugins/BIN/ParaFly -c /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/butterfly_commands -shuffle -CPU 1 -failed_cmds failed_butterfly_commands.821092.txt  2>tmp.821092.1692389806.stderr

Errmsg:
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.
warning, cmd: java -Xmx10G -Xms1G -Xss1G   -XX:ParallelGCThreads=2  -jar /storage/home/yml5557/.conda/envs/trinity/opt/trinity-2.9.1/Butterfly/Butterfly.jar -N 100000 -L 150 -F 500 -C /scratch/yml5557/trinity_Drosera/trinity_out_dir/read_partitions/Fb_0/CBin_0/c0.trinity.reads.fa.out/chrysalis/Component_bins/Cbin0/c1.graph  --path_reinforcement_distance=25  --NO_EM_REDUCE  failed with ret: 9, going to retry.

Could you help me out of this? Thank you!!

Best,

Yunjia

[Brian Haas]

Hi Yunjia,

My earlier advice holds for this - deleting the corresponding directory for the sub-job.  Please let me know if that doesn't solve it.

best,

~b

To view this discussion on the web visit https://groups.google.com/d/msgid/trinityrnaseq-users/41c1a024-405d-437b-a6cf-b2a20dd64517n%40googlegroups.com.

[Yunjia Liu]

Hi Brian,

Yes, I noticed that after I sent the email. Thank you. So I deleted the directory below and two others.

/Fb_0/CBin_0/c0.trinity.reads.fa.out

However, it seems the trinity has stuck. The log showed that it froze at succeeded(26) 99.9943% completed, without any further error message. Should I maybe add the -FORCE option? 

Best,

Yunjia

[Brian Haas]

I'd wait it out a bit.  If it continues to fail or if you need to kill it, then yes, you could rerun with the --FORCE option. 

But, be sure to examine the clusters it couldn't assemble. They're often things like microbial genomes / symbionts, or plastids (not host transcripts, usually).