Error raised with split_libraries_fastq.py

29 views
Skip to first unread message

Lauren Czaplicki

unread,
Jun 19, 2017, 12:06:43 PM6/19/17
to Qiime 1 Forum
Hi, 

I ran split libraries on my fastq file with 

split_libraries_fastq.py -i 'Read2.fastq' -o 'Read2' -b 'barcodes.fastq' --store_qual_scores --barcode_type 8 --store_demultiplexed_fastq -q 19 -m '/home/lauren/Desktop/61717_MiSeq/Task3Mappingrevc8bc.txt' 

and got the following error

Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/split_libraries_fastq.py", line 360, in <module>
    main()
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/split_libraries_fastq.py", line 339, in main
    for fasta_header, sequence, quality, seq_id in seq_generator:
  File "/home/lauren/qiime_software/qiime-1.8.0-release/lib/qiime/split_libraries_fastq.py", line 287, in process_fastq_single_end_read_file
    MinimalFastqParser(fastq_read_f,strict=False)):
  File "/home/lauren/qiime_software/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/parse/fastq.py", line 40, in MinimalFastqParser
    yield record[0][1:], record[1], record[3]
IndexError: list index out of range

I thought that perhaps this was an error caused by mismatched labels somewhere, so I ran compare_fastq_labels.py (https://gist.github.com/walterst/5397234 ) but found nothing. 

I attached my mapping file to this post.

The first ten lines of Read2 and barcodes.fastq are as follows

lauren@Carmen:~/Desktop/61717_MiSeq$  qiime > head -n 10 Read2.fastq
@M00830:156:000000000-B9C2B:1:2107:13909:1352 4:N:0:1
GCACTATGCACCGTGTTTCAAGACGGGCTTAAGTTCAGCGGGTTCCTATACAAGCTGTCTCTTATACACATCTCTGATGGCGCGAGGGAGGCGTGTAGATCTCGGTGGTCGCCGGATCCTTATAAAAAAAAAAGAAAAAACGAGACAAGAAAGAAAGAAAGAAACATAAATAAAAACAACGTGCGTAAGATACAAACAGTAACGAGCACAAAACAAAGACAAACTAAGCAATGTAGAGGAAAGTAACGTCGAGTCACGCACACACCCAGCACTACACACAACCGACACAAACTACCACAAA
+
9CCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG@?AEFF,95<F+?7+4+4=4++48A,EE,@,3AAF@FGG>DF?,5+3*:**4:>8,,,,1,,,,,,,481,,,1,=,,,,,,4,,***1*))/**+++++*****+++1)1))..()()()(4(,(0()0**)))).)))-)))(((.()))),2,(-()()-((,((.(((-(((.(,).)(,(4((---1(,((,()))()).((
@M00830:156:000000000-B9C2B:1:2107:19830:1385 4:N:0:1
GTGACTATTCACCGTGTTTCAAGACGGGCTTAAGTTCAGCGGGTTCAACAAGTAAGACTGTCTCTTATACACATCTCTGATGGCGCGAGGGAGGCGTGTAGATCTCGGTGGTAGCCGTAGCAGTAAAAAAAAAAAAAAAAGAAAAAAAAAAATAAAGACCAGTATGAAGACAAACCGAATCAAAAGCCGAAGGAACTACAAGTGACAAAGCAGACTACACATCAGAGTAAACACATACAGCAATACACAGACGAACACAAGTAATAGACAAGCAGCACACAACAAACAGACATGACACACA
+
CCCCCGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGEF+4,9,,:5,A@FE7,CA,++++49,CA,AE<F+@>+8:+55**3,,3,*****4,,3+1+***=+++5+++5*+5****2/9*+*+*)/**)1)))**0**0+*20:***)()(0(0))()(*)///1**1))))(,))))))(,))))).(,(4(((((((()).)))-))((((.((((((-(,(((((((())-))).((
@M00830:156:000000000-B9C2B:1:2107:10354:1399 4:N:0:1
TTTTTTATGGTTCGTTCTTATTACCCTTCTGAATGTCACGCTGATTATTTTGACTTTGAGCGTATCGAGGCTCTTAAACCTGCTATTGAGGCTTGTGGCATTTCTACTCTTTCTCAATCCCCAATGCTTGGCTTCCATAAGCAGATGGATAACCGCATCAAGCTCTTGGAAGAGATTCTGTCTTTTCGTATGCAGGGCGTTGAGTTCGATAATGGTGATATGTATGTTGACGGCCATAAGGCTGCTTCTGAAGTTCGTGATGAGTTTGTATCTGTTACTGAGAAGTTAATGGATGAATTGG

lauren@Carmen:~/Desktop/61717_MiSeq$  qiime > head -n 10 barcodes.fastq
@M00830:156:000000000-B9C2B:1:2107:13909:1352 2:N:0:1
TCCAGTTC
+
CCCCCGGG
@M00830:156:000000000-B9C2B:1:2107:19830:1385 2:N:0:1
AATCCGGA
+
CCCCCGGG
@M00830:156:000000000-B9C2B:1:2107:10354:1399 2:N:0:1
TAGCTTTA


the qiime setup I'm using is below:

System information
==================
         Platform: linux2
   Python version: 2.7.3 (default, Jan 31 2014, 15:02:06)  [GCC 4.6.3]
Python executable: /home/lauren/qiime_software/python-2.7.3-release/bin/python

Dependency versions
===================
                     PyCogent version: 1.5.3
                        NumPy version: 1.7.1
                   matplotlib version: 1.3.1
                  biom-format version: 1.3.1
                         qcli version: 0.1.0
                QIIME library version: 1.8.0
                 QIIME script version: 1.8.0
        PyNAST version (if installed): 1.2.2
                      Emperor version: 0.9.3
RDP Classifier version (if installed): rdp_classifier-2.2.jar
          Java version (if installed): 1.7.0_91

QIIME config values
===================
                     blastmat_dir: /home/lauren/qiime_software/blast-2.2.22-release/data
                         sc_queue: all.q
      topiaryexplorer_project_dir: None
     pynast_template_alignment_fp: /home/lauren/qiime_software/core_set_aligned.fasta.imputed
                  cluster_jobs_fp: /home/lauren/qiime_software/qiime-1.8.0-release/bin/start_parallel_jobs.py
pynast_template_alignment_blastdb: None
assign_taxonomy_reference_seqs_fp: /home/lauren/qiime_software/gg_otus-13_8-release/rep_set/97_otus.fasta
                     torque_queue: friendlyq
   template_alignment_lanemask_fp: /home/lauren/qiime_software/lanemask_in_1s_and_0s
                    jobs_to_start: 1
                cloud_environment: False
                qiime_scripts_dir: /home/lauren/qiime_software/qiime-1.8.0-release/bin
            denoiser_min_per_core: 50
                      working_dir: /tmp/
                    python_exe_fp: /home/lauren/qiime_software/python-2.7.3-release/bin/python
                         temp_dir: /tmp/
                      blastall_fp: /home/lauren/qiime_software/blast-2.2.22-release/bin/blastall
                 seconds_to_sleep: 60
assign_taxonomy_id_to_taxonomy_fp: /home/lauren/qiime_software/gg_otus-13_8-release/taxonomy/97_otu_taxonomy.txt
FF...F...................F.F.....F.
======================================================================
FAIL: test_FastTree_supported_version (__main__.QIIMEDependencyFull)
FastTree is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 365, in test_FastTree_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported FastTree version. 2.1.3 is required, but running 2.1.7.

======================================================================
FAIL: test_INFERNAL_supported_version (__main__.QIIMEDependencyFull)
INFERNAL is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 520, in test_INFERNAL_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported INFERNAL version. 1.0.2 is required, but running # INFERNAL 1.1rc4 (June 2013)
.

======================================================================
FAIL: test_blast_supported_version (__main__.QIIMEDependencyFull)
blast is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 477, in test_blast_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported blast version. 2.2.22 is required, but running 2.2.26.

======================================================================
FAIL: test_python_supported_version (__main__.QIIMEDependencyFull)
python is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 283, in test_python_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported python version. 2.7.3 is required, but running 2.7.6.

======================================================================
FAIL: test_raxmlHPC_supported_version (__main__.QIIMEDependencyFull)
raxmlHPC is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 600, in test_raxmlHPC_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported raxmlHPC version. (7, 3, 0).(7, 3, 0) is required, but running 7.7.2.

======================================================================
FAIL: test_usearch_supported_version (__main__.QIIMEDependencyFull)
usearch is in path and version is supported
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lauren/qiime_software/qiime-1.8.0-release/bin/print_qiime_config.py", line 715, in test_usearch_supported_version
    % ('.'.join(map(str,acceptable_version)), version_string))
AssertionError: Unsupported usearch version. (5, 2, 236).(5, 2, 236) is required, but running 
USEARCH is not freely redistributable and is thus not included in the
default QIIME package.
You may obtain a personal copy of the 32-bit program at no charge.

To use this feature, please go to:

Download USEARCH v5.2.236, then:
    sudo mv usearch* /usr/local/bin/usearch
    sudo chmod a+x /usr/local/bin/usearch

You probably also want to install USEARCH 6.1 as /usr/local/bin/usearch61

This data was generated by an Illumina MiSeq.  I appreciate any help with this troubleshooting adventure!

Best,
Lauren
Task3Mappingrevc8bc.txt

Lauren Czaplicki

unread,
Jun 19, 2017, 3:51:13 PM6/19/17
to Qiime 1 Forum
Found the culprit! The last lines in the file had some sequence information without matching quality information.  I used tail -n -# to create versions of the fastq files without the last lines belonging to the problem read at the end of each file, then tried split_libraries_fastq.py again and it worked!

justink

unread,
Jun 20, 2017, 12:54:54 PM6/20/17
to Qiime 1 Forum
woo hoo. Thanks for posting the solution too.
Reply all
Reply to author
Forward
0 new messages