trinity not all specified records have been retrieved

[wildli...@gmail.com]

 

Dear  Brian and Trinity users/developers,

Thank you for maintaining and updating the group.

I am struggling with this famous error for few days. I can not figure out what could be wrong in the fast files :

I have attached a screen shot of my R1/R2 to the email.

Cheers

----------------------------------------------------------------------------------

-------------- Trinity Phase 1: Clustering of RNA-Seq Reads  ---------------------

----------------------------------------------------------------------------------

---------------------------------------------------------------

------ Quality Trimming Via Trimmomatic  ---------------------

<< ILLUMINACLIP:/apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 >>

---------------------------------------------------------------

## Running Trimmomatic on read files: /home/pteske/lustre/sarRNA/R1.fastq, /home/pteske/lustre/sarRNA/R2.fastq

Monday, December 18, 2017: 15:41:06     CMD: java -jar /apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/trimmomatic.jar PE -threads 50 -phred33  /home/pteske/lustre/sarRNA/R1.fastq /home/pteske/lustre/sarRNA/R2.fastq  R1.fastq.P.qtrim R1.fastq.U.qtrim  R2.fastq.P.qtrim R2.fastq.U.qtrim  ILLUMINACLIP:/apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 

TrimmomaticPE: Started with arguments:

 -threads 50 -phred33 /home/pteske/lustre/sarRNA/R1.fastq /home/pteske/lustre/sarRNA/R2.fastq R1.fastq.P.qtrim R1.fastq.U.qtrim R2.fastq.P.qtrim R2.fastq.U.qtrim ILLUMINACLIP:/apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25

Using PrefixPair: 'TACACTCTTTCCCTACACGACGCTCTTCCGATCT' and 'GTGACTGGAGTTCAGACGTGTGCTCTTCCGATCT'

ILLUMINACLIP: Using 1 prefix pairs, 0 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences

Exception in thread "Thread-0" java.lang.RuntimeException: Missing comment line from record: NB501140:56:HJ2VLAFXX:1:11307:20409:12526 1:N:0:CTTGTA

        at org.usadellab.trimmomatic.fastq.FastqParser.parseOne(FastqParser.java:77)

        at org.usadellab.trimmomatic.fastq.FastqParser.next(FastqParser.java:179)

        at org.usadellab.trimmomatic.threading.ParserWorker.run(ParserWorker.java:42)

        at java.lang.Thread.run(Thread.java:745)

Input Read Pairs: 1433000 Both Surviving: 1092386 (76.23%) Forward Only Surviving: 340539 (23.76%) Reverse Only Surviving: 0 (0.00%) Dropped: 75 (0.01%)

TrimmomaticPE: Completed successfully

Monday, December 18, 2017: 15:41:11     CMD: cp R1.fastq.P.qtrim R1.fastq.PwU.qtrim.fq

Monday, December 18, 2017: 15:41:12     CMD: cp R2.fastq.P.qtrim R2.fastq.PwU.qtrim.fq

Monday, December 18, 2017: 15:41:13     CMD: touch trimmomatic.ok

Monday, December 18, 2017: 15:41:13     CMD: gzip R1.fastq.P.qtrim R1.fastq.U.qtrim R2.fastq.P.qtrim R2.fastq.U.qtrim &

---------------------------------------------------------------

------------ In silico Read Normalization ---------------------

-- (Removing Excess Reads Beyond 50 Coverage --

---------------------------------------------------------------

# running normalization on reads: $VAR1 = [

          [

            'R1.fastq.PwU.qtrim.fq'

          ],

          [

            'R2.fastq.PwU.qtrim.fq'

          ]

        ];

Monday, December 18, 2017: 15:41:13     CMD: /apps/chpc/bio/trinity/2.4.0/util/insilico_read_normalization.pl --seqType fq --JM 100G  --max_cov 50 --CPU 50 --output /mnt/lustre/users/pteske/sarRNA/trinity_sardine/insilico_read_normalization   --max_pct_stdev 10000  --left R1.fastq.PwU.qtrim.fq --right R2.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS  

Converting input files. (both directions in parallel)left file exists, nothing to doright file exists, nothing to do-------------------------------------------

----------- Jellyfish  --------------------

-- (building a k-mer catalog from reads) --

-------------------------------------------

-sorting each stats file by read name.

Done converting input files.Thread 4 terminated abnormally: Error, not all specified records have been retrieved (missing 34370) from /mnt/lustre/users/pteske/sarRNA/trinity_sardine/R2.fastq.PwU.qtrim.fq, see file: /mnt/lustre/users/pteske/sarRNA/trinity_sardine/insilico_read_normalization/R2.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /apps/chpc/bio/trinity/2.4.0/util/insilico_read_normalization.pl line 548.

Thread 3 terminated abnormally: Error, not all specified records have been retrieved (missing 34370) from /mnt/lustre/users/pteske/sarRNA/trinity_sardine/R1.fastq.PwU.qtrim.fq, see file: /mnt/lustre/users/pteske/sarRNA/trinity_sardine/insilico_read_normalization/R1.fastq.PwU.qtrim.fq.normalized_K25_C50_pctSD10000.fq.missing_accs for list of missing entries at /apps/chpc/bio/trinity/2.4.0/util/insilico_read_normalization.pl line 548.

Error encountered with thread.

Error encountered with thread.

Error, at least one thread died at /apps/chpc/bio/trinity/2.4.0/util/insilico_read_normalization.pl line 419.

Error, cmd: /apps/chpc/bio/trinity/2.4.0/util/insilico_read_normalization.pl --seqType fq --JM 100G  --max_cov 50 --CPU 50 --output /mnt/lustre/users/pteske/sarRNA/trinity_sardine/insilico_read_normalization   --max_pct_stdev 10000  --left R1.fastq.PwU.qtrim.fq --right R2.fastq.PwU.qtrim.fq --pairs_together --PARALLEL_STATS   died with ret 6400 at /apps/chpc/bio/trinity/2.4.0/Trinity line 2462.

(END)

[Brian Haas]

This error:

 Exception in thread "Thread-0" java.lang.RuntimeException: Missing comment line from record: NB501140:56:HJ2VLAFXX:1:11307:20409:12526 1:N:0:CTTGTA

suggests that there might be a problem with the fastq file, as it may be corrupt or truncated.

You can try running the trimmomatic step directly and see if it fails:

CMD: java -jar /apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/trimmomatic.jar PE -threads 50 -phred33  /home/pteske/lustre/sarRNA/R1.fastq /home/pteske/lustre/sarRNA/R2.fastq  R1.fastq.P.qtrim R1.fastq.U.qtrim  R2.fastq.P.qtrim R2.fastq.U.qtrim  ILLUMINACLIP:/apps/chpc/bio/trinity/2.4.0/trinity-plugins/Trimmomatic/adapters/TruSeq3-PE.fa:2:30:10 SLIDINGWINDOW:4:5 LEADING:5 TRAILING:5 MINLEN:25 

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.
To post to this group, send email to trinityrnaseq-users@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

[wildli...@gmail.com]

Many thanks BRIAN for your kind answer. Before digging more in to the errors  I am wondering if the latest version of the Trinity still requires no spaces in the header and /1 and /2 at the end for the reads or it  is more tolerant of naming ?

It seems that even running Trinity without trimmomatic still complaint about sanity of the fastq files with famous " not all specified records have been retrieved" or "fastq doesn't have four line entries "

Cheers

To post to this group, send email to trinityrn...@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

The latest version should have the same requirements as earlier, in that the fastq files should meet the standard 'old' or 'current' illumina formatting, but the newer version will be better about providing error messages for reads it has difficulty in parsing.

best,

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[wildli...@gmail.com]

Thanks Brian. It is very kind of you. Before start  composing the one liners I guess the attached header is far from being readable : As I see no "/1" and an obvious space 

in the header. I will appreciate  your suggestions .

Many thanks in advance  

cheers

The latest version should have the same requirements as earlier, in that the fastq files should meet the standard 'old' or 'current' illumina formatting, but the newer 

best,

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

That formatting looks like the latest illumina formatting.  It should be fine. Let's see how it goes.  When you rerun it, do it in a new workspace so it won't try to reuse any earlier outputs.

best,

~b

cheers

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[wildli...@gmail.com]

Thanks.sorry for silly question that means running it at it is without header modification: no space removal and no /1 /2 adding ?
I am uploading the data again and update you about progress.
Cheers

[Brian Haas]

Right, should be fine as is.  If it's not, then there's some other issue than the formatting, and we'll hopefully have more clues on the next try.

best,

~b

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.
To post to this group, send email to trinityrnaseq-users@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[wildli...@gmail.com]

 Good day Brian ,

I guess the issue was_as you recommended the corrupted fq files_.

While I run it successfully without bowtie 2 , while using bowtie2 I got embedded error. It is a bit strange since libtbb.so.2 library is installed and in LD_Library _PATH as screen shot show. I will appreciate your recommendations :

Cheers

--------------------------------------------------------

-------------------- Chrysalis -------------------------

-- (Contig Clustering & de Bruijn Graph Construction) --

--------------------------------------------------------

inchworm_target: /mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa

bowite_reads_fa: /mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa

chrysalis_reads_fa: /mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa

* Running CMD: /apps/chpc/bio/trinity/2.4.0/util/support_scripts/filter_iworm_by_min_length_or_cov.pl /mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/inchworm.K25.L25.DS.fa 100 10 > /home/pteske/lustre/Arsalan/trinity/trinity_out/chrysalis/inchworm.K25.L25.DS.fa.min100

* Running CMD: bowtie2-build -o 3 /home/pteske/lustre/Arsalan/trinity/trinity_out/chrysalis/inchworm.K25.L25.DS.fa.min100 /home/pteske/lustre/Arsalan/trinity/trinity_out/chrysalis/inchworm.K25.L25.DS.fa.min100 1>/dev/null

bowtie2-build: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory

Error, cmd: bowtie2-build -o 3 /home/pteske/lustre/Arsalan/trinity/trinity_out/chrysalis/inchworm.K25.L25.DS.fa.min100 /home/pteske/lustre/Arsalan/trinity/trinity_out/chrysalis/inchworm.K25.L25.DS.fa.min100 1>/dev/null 2>tmp.93446.stderr died with ret 32512 at /apps/chpc/bio/trinity/2.4.0/PerlLib/Pipeliner.pm line 166.

        Pipeliner::run('Pipeliner=HASH(0xa61d58)') called at /apps/chpc/bio/trinity/2.4.0/Trinity line 1750

        main::run_chrysalis('/mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/inchworm...', '/mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa', 200, 500, undef, '/mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa', '/mnt/lustre/users/pteske/Arsalan/trinity/trinity_out/both.fa') called at /apps/chpc/bio/trinity/2.4.0/Trinity line 1595

        main::run_Trinity() called at /apps/chpc/bio/trinity/2.4.0/Trinity line 1262

        eval {...} called at /apps/chpc/bio/trinity/2.4.0/Trinity line 1261

Trinity run failed. Must investigate error above.

On Tuesday, December 19, 2017 at 9:28:58 PM UTC+2, Brian Haas wrote:

Right, should be fine as is.  If it's not, then there's some other issue than the formatting, and we'll hopefully have more clues on the next try.

best,

~b

On Tue, Dec 19, 2017 at 11:14 AM, <wildli...@gmail.com> wrote:

Thanks.sorry for silly question that means running it at it is without header modification: no space removal and no /1 /2 adding ?
I am uploading the data again and update you about progress.
Cheers

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

Hi,

This is a different problem:

 bowtie2-build: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory

indicating an issue with the bowtie2 installation.  

You might need to install this:

https://www.threadingbuildingblocks.org/

to get bowtie2 to work.

It's not an absolutely essential part of Trinity, though.

best,

~b

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[wildli...@gmail.com]

Thanks ,

I have already asked the CHPC team to install  tbblib_dev . The error is despite that!

Do you have any clues ?

Many thanks

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

[Brian Haas]

you can try building bowtie2 from source.     If it's not finding the shared library, you might need to set env var LD_LIBRARY_PATH to include where the TBB .so is installed.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsubscribe...@googlegroups.com.

To post to this group, send email to trinityrn...@googlegroups.com.
Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.

--

--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

--
You received this message because you are subscribed to the Google Groups "trinityrnaseq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trinityrnaseq-users+unsub...@googlegroups.com.

To post to this group, send email to trinityrnaseq-users@googlegroups.com.

Visit this group at https://groups.google.com/group/trinityrnaseq-users.
For more options, visit https://groups.google.com/d/optout.