Silva db: pick_open_reference error: IndexError: cannot do a non-empty take from an empty axes.

[Joany]

Dear Qiime1 Gurus:

I was running the following command:

pick_open_reference_otus.py -i seqs.fna -p /SILVA_128_QIIME_release/SILVA_123_db.parameters.txt -r /SILVA_128_QIIME_release/rep_set/rep_set_all/97/97_otus.fasta -o Silva123_out

where my SILVA_123_db.parameters.txt is:
align_seqs.py:template_fp /SILVA_128_QIIME_release/core_alignment/core_alignment_SILVA128.fna
filter_alignment:allowed_gap_frac 0.80
filter_alignment:entropy_threshold 0.10
filter_alignment:suppress_lane_mask_filter True
assign_taxonomy:reference_seqs_fp /SILVA_128_QIIME_release/rep_set/rep_set_all/97/97_otus.fasta
assign_taxonomy:id_to_taxonomy_fp /SILVA_128_QIIME_release/taxonomy/taxonomy_all/97/taxonomy_7_levels.txt

where: my seqs.fna file is the result of the split_libraries_fastq.py command
$ head seqs.fna
>M40A.Metaxa_0 NS500162:172:HG5CJBGXX:1:11101:25495:1682 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
CCCGTCAATTCCTTTGAGTTTTAACCTTGCGGCCGTACTCCCCAGGCGGAATGCTTAATGCGTTAACTGCGTCACCGAACAGCATGCTGACCGACGACTGGCATTCATCGTTTACGGCGTGGACTACCAGGGTATCTAATCCTGTTTGCTC
>M40A.Metaxa_1 NS500162:172:HG5CJBGXX:1:11101:23150:2968 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0
TGCGATAAATCTTTCTCCCAAAGGACGTATACGGTATTAGCACGAGTTTCCCCATGTTGTTCCGTACTCCAGGACAGTTTCCCACGCGTTACTCACCCGTCTGCCACTCTCTCCGAAGAGATCGTTCGACTTGCATGTGTTAGGCCTGCCG
>M40A.Metaxa_2 NS500162:172:HG5CJBGXX:1:11101:11722:7001 orig_bc=AAAAAAAAAAAA new_bc=AAAAAAAAAAAA bc_diffs=0


When I get the following error about an hour into the run .....

Traceback (most recent call last):
  File "/macqiime/anaconda/bin/pick_open_reference_otus.py", line 453, in <module>
    main()
  File "/macqiime/anaconda/bin/pick_open_reference_otus.py", line 432, in main
    minimum_failure_threshold=minimum_failure_threshold)
  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/workflow/pick_open_reference_otus.py", line 1071, in pick_subsampled_open_reference_otus
    status_update_callback=status_update_callback)
  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/workflow/pick_open_reference_otus.py", line 327, in align_and_tree
    close_logger_on_success=close_logger_on_success)
  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/workflow/util.py", line 122, in call_commands_serially
    raise WorkflowError(msg)
qiime.workflow.util.WorkflowError:

*** ERROR RAISED DURING STEP: Filter alignment
Command run was:
 filter_alignment.py -o Silva123_out/pynast_aligned_seqs -i Silva123_out/pynast_aligned_seqs/rep_set_aligned.fasta --allowed_gap_frac 0.80 --entropy_threshold 0.10 --suppress_lane_mask_filter
Command returned exit status: 1
Stdout:

Stderr
Traceback (most recent call last):
  File "/macqiime/anaconda/bin/filter_alignment.py", line 155, in <module>
    main()
  File "/macqiime/anaconda/bin/filter_alignment.py", line 148, in main
    entropy_threshold=opts.entropy_threshold):
  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/filter_alignment.py", line 78, in apply_lane_mask_and_gap_filter
    entropy_mask = generate_lane_mask(aln, entropy_threshold)
  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/filter_alignment.py", line 146, in generate_lane_mask
    interpolation='nearest')
  File "/macqiime/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3054, in percentile
    interpolation=interpolation)
  File "/macqiime/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2803, in _ureduce
    r = func(a, **kwargs)
  File "/macqiime/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3128, in _percentile
    r = take(ap, indices, axis=axis, out=out)
  File "/macqiime/anaconda/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 121, in take
    return take(indices, axis, out, mode)
IndexError: cannot do a non-empty take from an empty axes.


Just in case, here is the output of print_qiime_config.py
System information
==================
         Platform:    darwin
   Python version:    2.7.10 |Anaconda 2.2.0 (x86_64)| (default, May 28 2015, 17:04:42)  [GCC 4.2.1 (Apple Inc. build 5577)]
Python executable:    /macqiime/anaconda/bin/python

QIIME default reference information
===================================
For details on what files are used as QIIME's default references, see here:
 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions
===================
          QIIME library version:    1.9.1
           QIIME script version:    1.9.1
qiime-default-reference version:    0.1.2
                  NumPy version:    1.9.2
                  SciPy version:    0.15.1
                 pandas version:    0.16.1
             matplotlib version:    1.4.3
            biom-format version:    2.1.4
                   h5py version:    2.4.0 (HDF5 version: 1.8.14)
                   qcli version:    0.1.1
                   pyqi version:    0.3.2
             scikit-bio version:    0.2.3
                 PyNAST version:    1.2.2
                Emperor version:    0.9.51
                burrito version:    0.9.1
       burrito-fillings version:    0.1.1
              sortmerna version:    SortMeRNA version 2.0, 29/11/2014
              sumaclust version:    SUMACLUST Version 1.0.00
                  swarm version:    Swarm 1.2.19 [Jun  2 2015 14:40:16]
                          gdata:    Installed.

QIIME config values
===================
For definitions of these settings and to learn how to configure QIIME, see here:
 http://qiime.org/install/qiime_config.html
 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir:    None
      pick_otus_reference_seqs_fp:    /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                         sc_queue:    all.q
      topiaryexplorer_project_dir:    None
     pynast_template_alignment_fp:    /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta
                  cluster_jobs_fp:    start_parallel_jobs.py
pynast_template_alignment_blastdb:    None
assign_taxonomy_reference_seqs_fp:    /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta
                     torque_queue:    friendlyq
                    jobs_to_start:    1
                       slurm_time:    None
            denoiser_min_per_core:    50
assign_taxonomy_id_to_taxonomy_fp:    /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt
                         temp_dir:    /tmp/
                     slurm_memory:    None
                      slurm_queue:    None
                      blastall_fp:    blastall
                 seconds_to_sleep:    60







Any suggestions on what might be throwing the error up?
Many Thanks!!

[Greg Caporaso]

Hello,

I'm not sure what's going on here offhand. Have you checked whether there are any sequences in the Silva123_out/pynast_aligned_seqs/rep_set_aligned.fasta file that is being passed as input to filter_alignment.py? I'm wondering if all of the sequences may have failed to align with PyNAST, and this filter is failing as a result. 

If the Silva123_out/pynast_aligned_seqs/rep_set_aligned.fasta file isn't empty, could you share that with me? 

Best,

Greg

[Joany]

I finally got my job to run successfully as it produced a biom file called:  otu_table_mc2_w_tax_no_pynast_failures.biom (along w/ a couple of others). What I ended up doing was removing the following parameters from my parameter file, while keeping the rest of the parameter file the same.

filter_alignment:allowed_gap_frac 0.80

filter_alignment:entropy_threshold 0.10

filter_alignment:suppress_lane_mask_filter True

I ran it with the same commands as before, and it worked like a charm. 

[Greg Caporaso]

Glad to hear that you have it working now, thanks for following up with your solution!