strange stdout

19 views
Skip to first unread message

yaximik

unread,
Dec 23, 2013, 11:51:34 AM12/23/13
to biop...@googlegroups.com
To avoid occasional aborting on error due to unconventional quality values in Illumina reads, I included the following line in processing script:

perl -pne 'if (($. % 4) == 0) {tr/[_-~]/_/}' | #change all scores in the range "_"(95) to "~"(126) to "_"(95)
read_fastq -e base_33 -i - |         # Read in raw FASTQ data from stdin.
......

Recently I got the following to stdout:

[yaximik@G5NNJN1 ILLAgrAdClip]$ ./parallel_ILLAgrAdClip
Processing CH3AskE1MILLib3smpl8PfrHfrRun1R1
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `<<'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `puts'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:126:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `<<'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `puts'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:126:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
Processing CH3AskE1MILLib3smpl8PfrHfrRun1R2
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `<<'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `puts'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:126:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `write': Broken pipe - <STDOUT> (Errno::EPIPE)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `<<'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:118:in `puts'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:126:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `tr!': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/seq.rb:627:in `qual_coerce!'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:120:in `block (4 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:108:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:108:in `block (3 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/filesys.rb:80:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:107:in `block (2 levels) in <main>'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `each'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:106:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/read_fastq:51:in `<main>'
/home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `===': invalid byte sequence in UTF-8 (ArgumentError)
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:142:in `block in get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `each_line'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:140:in `get_entry'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:128:in `each_record'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:41:in `block in <main>'
        from /home/yaximik/Bioinformatics/biopieces/code_ruby/lib/maasha/biopieces.rb:89:in `open'
        from /home/yaximik/Bioinformatics/biopieces/bp_bin/mask_seq:40:in `<main>'
[yaximik@G5NNJN1 ILLAgrAdClip]$

As both cleaned file outputs were created, it looks like the offending lines were just skipped and processing was completed. But what could be reason for this error messages?

Vladimir


 

Martin Asser Hansen

unread,
Dec 27, 2013, 2:27:49 PM12/27/13
to biop...@googlegroups.com
I have seen that error message when the wrong encoding is used. Can you isolate some of the offending entries and send me?


Cheers,


Martin


--
You received this message because you are subscribed to the Google Groups "biopieces" group.
To unsubscribe from this group and stop receiving emails from it, send an email to biopieces+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Vladimir Yamshchikov

unread,
Dec 27, 2013, 8:04:34 PM12/27/13
to biop...@googlegroups.com
I found the reason, rather trivial. For some reason I had to transfer this particular fastq file through an intermediate upload to Macintosh and then to Linux. When the same dataset was transferred to Linux directly from MiSeq, processing was completed without complains.
 
Vladimir
Reply all
Reply to author
Forward
0 new messages