Error running usearch61 in pick_open_reference_otus.py

[Brett Younginger]

Hello, 

In an attempt to perform OTU clustering with the pick_open_reference_otus.py script, I have been getting the error message:

Error running usearch61. Possible causes are unsupported version (current supported version is usearch v6.1.544) is installed or improperly formatted input file was provided

It seems like this is a frequent issue for QIIME users as there are many posts to this forum regarding the same problem, yet none of the solutions that I have read seem to apply to my scenario. Here are the details:

• I get the same type of error regardless of whether I'm trying to cluster with uclust or usearch61
• uclust, usearch, and usearch 61 are all executable and in my path:
• which uclust
• ~/my_env/bin/uclust
• uclust --version
• uclust v1.2.22q
• which usearch61
• ~/my_env/bin/usearch61
• usearch61 --version
• usearch_i86linux32 v6.1.544
• same results with usearch...
• my input fasta file seems fine (~2M reads, but passed through cutadapt, trimmomatic, and mothur to remove short seqs and ambiguous base calls)
• my print_qiime_config file returns no errors, although I see no lines referring to usearch or uclust (output attached as a text file)
• I have had no problems running this script in the past locally, but recently installed QIIME on a compute server at my university in order to speed up my workflow. Other QIIME scripts work fine (e.g. split_libraries_fastq.py), however. 

Any help is much appreciated. Thanks in advance for your time. 

-Brett 

[Colin Brislawn]

Hello Brett,

Thanks for getting in touch with us and posting more about your qiime set up. 

How large, in MB or GB is the file size of your seqs.fna file? usearch61 is 32-bit and can't handle very large seqs.fna files, and that could be happening here. 

Colin 

[Brett]

Hey Colin, 

The file is 526 MB. Does uclust have similar limits on file sizes? I get the same error when trying to use uclust instead of usearch61.

Thanks for your prompt reply.

[Colin Brislawn]

Ah OK. That file size should not be a problem. 

Perhaps there is a formatting problem. Try running this script to validate the formatting.

http://qiime.org/scripts/validate_demultiplexed_fasta.html 

Colin 

[Brett]

Colin, 

The log from the validate_demultiplexed_fasta script is attached. The only potential problem I see is that 0.006 percent of the reads don't have valid QIIME fasta labels. Is this enough to cause usearch61 to fail? If not, other thoughts? 

Thanks, 

Brett

[Brett]

Colin, 

I updated my parameters file and may have had some errors in there. Specifically, I had arguments that were for the pick_otus.py script and not pick_open_reference_otus.py. It seems to be working now, but when I look at the "top" screen I see that uclust is the program consuming the most CPU. I specified usearch61 in the params file with the line:

pick_open_reference_otus:otu_picking_method usearch61

Not sure what to make of this, but I will let you know when the script is finished if it somehow reverted to usearch61. 

Cheers, 

Brett

[Brett]

Hey Colin, 

Well, the last command did not fully work. There was an error along the lines of: 

---Fatal error---

Input is not FASTA file

uclust --input open_reference_otus2//rep_set.fna ...

I attached the log file for reference. Maybe the error arose because it is picking reference OTUs from the default greengenes fasta file, not the one I specified in my params file (see line 38 of log file). Not sure why that happened. 

Also, if I try and override the default arguments in the substeps of pick_open_reference_otus.py (like pick_otus:otu_picking_method usearch61), I get an error that this is not possible and the argument must be supplied in the command line. I tried to re-run the script with only pick_open_reference_otus arguments in my params file and I got the earlier message again that initiated this post (Error running usearch61. Possible causes are unsupported version (current supported version is usearch v6.1.544) is installed or improperly formatted input file was provided). 

Thank you for any advice you can offer. I appreciate your time. 

Best regards,

Brett 

[Colin Brislawn]

Hello Brett,

Thanks for validating your fasta file for me and continuing to troubleshoot.

Maybe the error arose because it is picking reference OTUs from the default greengenes fasta file, not the one I specified in my params file (see line 38 of log file).

Good find! That does appear to be the problem and I have no idea why the qiime script is not properly using the database you passed. Perhaps a qiime developer can comment more.

Colin 

[TonyWalters]

Judging from your log file, you have pick_open_reference_otus in your parameters file, rather than pick_otus for specifying which reference files to use. Can you replace any lines that say pick_open_reference_otus with pick_otus in your parameters file, and post the full command you used for your pick_open_reference_otus.py? There should be reference files specified here as well.

[Colin Brislawn]

Good catch Tony! I overlooked that in the parameter file. 

[Brett]

Tony and Colin, 

Thank you for continuing to help me troubleshoot. I appreciate your time. 

I changed my params file to include "pick_otus" instead of "pick_open_reference_otus" and received error messages pertaining to the fact that certain arguments must be in the command line only and others belong only to "pick_open_reference_otus." I have attached a troubleshooting text file with the command, error messages, and the steps I took to rectify. After the fourth attempt, I received the "Error running usearch61" message which initiated this post. The log file for the last attempt is also attached.

Cheers, 

Brett  

 

[TonyWalters]

I'd suggest just doing the command that's raising the error, rather than the workflow script to simplify troubleshooting:

pick_otus.py -i filtered_R1.fasta -o open_reference_otus4//step1_otus_test -r /home/obrett/UNITE/sh_qiime_release_s_31.01.2016/sh_refs_qiime_ver7_97_s_31.01.2016.fasta -m usearch61_ref --enable_rev_strand_match --suppress_new_clusters

Note that I've set a slightly different output folder: 

open_reference_otus4//step1_otus_test 

You'll want to look at the .uc file(s) that is being created-we need to look at the beginning of the file for the exact usearch61 command and at the end of the file to see if there is a specific error message.

[Brett]

Tony, 

The command: 

pick_otus.py -i filtered_R1.fasta -o open_reference_otus4//step1_otus_test -r /home/obrett/UNITE/sh_qiime_release_s_31.01.2016/sh_refs_qiime_ver7_97_s_31.01.2016.fasta -m usearch61_ref --enable_rev_strand_match --suppress_new_clusters 

abundance_sorted.uc:

(head) 

S 0 204 * * * * * BY.1.1.6_3 *

H 0 204 100.0 * 0 0 * BY.1.1.6_4 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_5 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_6 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_7 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_8 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_9 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_10 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_11 BY.1.1.6_3

H 0 204 100.0 * 0 0 * BY.1.1.6_12 BY.1.1.6_3

(tail)

C 341555 1 * * * * * BY.14.4.6_848000 *

C 341556 1 * * * * * BY.14.4.6_847989 *

C 341557 1 * * * * * BY.1.4.6_62403 *

C 341558 1 * * * * * BY.14.4.6_847986 *

C 341559 1 * * * * * BY.14.4.6_847987 *

C 341560 1 * * * * * BY.14.4.6_847992 *

C 341561 1 * * * * * BY.14.4.6_847996 *

C 341562 1 * * * * * BY.14.4.6_847997 *

C 341563 1 * * * * * BY.14.4.6_847993 *

C 341564 1 * * * * * BY.14.4.6_847994 *

ref_clustered.uc:

empty

the error:

File "/home/obrett/my_env/lib/python2.7/site-packages/bfillings/usearch.py", line 1844, in usearch61_ref_cluster

    raise ApplicationError('Error running usearch61. Possible causes are '

burrito.util.ApplicationError: Error running usearch61. Possible causes are unsupported version (current supported version is usearch v6.1.544) is installed or improperly formatted input file was provided

-Brett

[TonyWalters]

Okay, let's try uclust and see if we get something more informative in the .uc files:

pick_otus.py -i filtered_R1.fasta -o open_reference_otus4//step1_otus_test2 -r /home/obrett/UNITE/sh_qiime_release_s_31.01.2016/sh_refs_qiime_ver7_97_s_31.01.2016.fasta -m uclust_ref --enable_rev_strand_match --suppress_new_clusters 

[Brett]

Hi Tony, 

The command: 

pick_otus.py -i filtered_R1.fasta -o open_reference_otus4//step1_otus_test2 -r /home/obrett/UNITE/sh_qiime_release_s_31.01.2016/sh_refs_qiime_ver7_97_s_31.01.2016.fasta -m uclust_ref --enable_rev_strand_match --suppress_new_clusters

Filtered_R1_clusters.uc:

# uclust --input /tmp/UclustExactMatchFilteryq6urR.fasta --id 0.97 --tmpdir /tmp --rev --w 8 --stepwords 8 --usersort --maxaccepts 1 --libonly --stable_sort --maxrejects 8 --lib /home/obrett/UNITE/sh_qiime_release_s_31.01.2016/sh_refs_qiime_ver7_97_s_31.01.2016.fasta --uc open_reference_otus4//step1_otus_test2/filtered_R1_clusters.uc

# version=1.2.22

# Tab-separated fields:

# 1=Type, 2=ClusterNr, 3=SeqLength or ClusterSize, 4=PctId, 5=Strand, 6=QueryStart, 7=SeedStart, 8=Alignment, 9=QueryLabel, 10=TargetLabel

# Record types (field 1): L=LibSeed, S=NewSeed, H=Hit, R=Reject, D=LibCluster, C=NewCluster, N=NoHit

# For C and D types, PctId is average id with seed.

# QueryStart and SeedStart are zero-based relative to start of sequence.

# If minus strand, SeedStart is relative to reverse-complemented seed.

The error:

File "/home/obrett/my_env/lib/python2.7/site-packages/bfillings/uclust.py", line 585, in get_clusters_from_fasta_filepath

    raise ApplicationError('Error running uclust. Possible causes are '

burrito.util.ApplicationError: Error running uclust. Possible causes are unsupported version (current supported version is v1.2.22) is installed or improperly formatted input file was provided

Thanks, 

Brett

[TonyWalters]

Is there more to that .uc file? Can you tail it, or attach the file?

[Brett]

That was all. I attached it anyways. 

[TonyWalters]

Okay, so, if it's balking at that point, there could be something amiss with the input and/or reference reads, as far as uclust is concerned.

Let's try something to isolate the problem. Do:
head -n 100 filtered_R1.fasta > subset_reads.fna

then

pick_otus.py -i filtered_R1.fasta -o test_subset_otu_picking -r subset_reads.fna -m uclust_ref --enable_rev_strand_match --suppress_new_clusters

[Brett]

Hey Tony, 

This seemed to work. Here's part of the output for the .uc file:

filtered_R1_clusters.uc

(head) 

# uclust --input /tmp/UclustExactMatchFilterxE2Xs4.fasta --id 0.97 --tmpdir /tmp --rev --w 8 --stepwords 8 --usersort --maxaccepts 1 --libonly --stable_sort --maxrejects 8 --lib /workspace/scratch/obrett/processing/subset_reads.fna --uc test_subset_otu_picking/filtered_R1_clusters.uc

# version=1.2.22

# Tab-separated fields:

# 1=Type, 2=ClusterNr, 3=SeqLength or ClusterSize, 4=PctId, 5=Strand, 6=QueryStart, 7=SeedStart, 8=Alignment, 9=QueryLabel, 10=TargetLabel

# Record types (field 1): L=LibSeed, S=NewSeed, H=Hit, R=Reject, D=LibCluster, C=NewCluster, N=NoHit

# For C and D types, PctId is average id with seed.

# QueryStart and SeedStart are zero-based relative to start of sequence.

# If minus strand, SeedStart is relative to reverse-complemented seed.

L 2 204 * * * * * BY.1.1.6_3 *

H 2 204 100.0 + 0 0 204M QiimeExactMatch.BY.1.1.6_3 BY.1.1.6_3

(tail)

D 1 7941 * * * * 97.8 BY.1.1.6_2 *

D 2 76530 * * * * 98.3 BY.1.1.6_3 *

D 17 15 * * * * 97.8 BY.1.1.6_18 *

D 18 13 * * * * 98.6 BY.1.1.6_19 *

D 24 115 * * * * 97.5 BY.1.1.6_25 *

D 29 5 * * * * 97.5 BY.1.1.6_30 *

D 34 109 * * * * 98.5 BY.1.1.6_35 *

D 37 70 * * * * 98.3 BY.1.1.6_38 *

D 42 158 * * * * 98.1 BY.1.1.6_43 *

D 47 312 * * * * 98.1 BY.1.1.6_48 *

The directory also contains a log, failures, and the otu map. Let me know if you'd like to see any of them. Think I should download a new reference fasta file? 

Thanks, 

Brett

[TonyWalters]

Downloading a new UNITE reference file might fix it. There could also be non-ascii characters in the file, which may be behind the problem. There is a custom script here:

https://gist.github.com/walterst/0a4d36dbb20c54eeb952
That can clean files. You would want to run this on both the representative fasta file and the taxonomy mapping file.

Once that is done, can you then try running the pick_otus.py command with the cleaned up reference fasta file and see if that helped?

[Brett]

Tony, 

I'm having some issues cleaning up the .fasta and .txt files with your script. The fasta is returned as a binary file and the .txt is returned as blank. 

Any additional tips? 

Thanks, 

Brett

[TonyWalters]

How are you running the script? Can you try running it on a different fasta file, like your input filtered_R1.fasta file and see if it generates a fasta file that you can read?

[Brett]

The script works on other fasta files. I checked my UNITE directory and most of the files were empty, so am not sure what happened. I downloaded new reference fasta and taxonomy files from UNITE and parsed those with your script successfully. After running the standard pick_otus.py with the parsed fasta file, no errors were raised. However, only 10 otus were recovered from ~2M reads. Is this the result of the --suppress_new_clusters argument? 

The command: (mislabelled directory name) 

pick_otus.py -i filtered_R1.fasta -o open_reference_otus5 -r /home/obrett/UNITE2/parsed_97.fasta -m uclust_ref --enable_rev_strand_match --suppress_new_clusters

filtered_R1_clusters.uc:

(head)

# uclust --input /tmp/UclustExactMatchFilter8VaWtf.fasta --id 0.97 --tmpdir /tmp --rev --w 8 --stepwords 8 --usersort --maxaccepts 1 --libonly --stable_sort --maxrejects 8 --lib /home/obrett/UNITE2/parsed_97.fasta --uc open_reference_otus5/filtered_R1_clusters.uc

# version=1.2.22

# Tab-separated fields:

# 1=Type, 2=ClusterNr, 3=SeqLength or ClusterSize, 4=PctId, 5=Strand, 6=QueryStart, 7=SeedStart, 8=Alignment, 9=QueryLabel, 10=TargetLabel

# Record types (field 1): L=LibSeed, S=NewSeed, H=Hit, R=Reject, D=LibCluster, C=NewCluster, N=NoHit

# For C and D types, PctId is average id with seed.

# QueryStart and SeedStart are zero-based relative to start of sequence.

# If minus strand, SeedStart is relative to reverse-complemented seed.

N       *       204     *       *       *       *       *       QiimeExactMatch.BY.1.1.6_3      *

N       *       204     *       *       *       *       *       QiimeExactMatch.BY.1.1.6_68     *

(tail)

D       7849    307     *       *       *       *       97.7    SH032625.07FU_KF757018_reps_singleton   *

D       13146   58      *       *       *       *       97.4    SH032623.07FU_KF757009_reps_singleton   *

D       13899   7       *       *       *       *       97.2    SH011608.07FU_KF150318_reps     *

D       14048   2       *       *       *       *       98.8    SH005938.07FU_EF530941_reps     *

D       14393   3       *       *       *       *       97.6    SH001842.07FU_UDB015619_refs    *

D       15515   2       *       *       *       *       97.9    SH467507.07FU_FJ152489_reps_singleton   *

D       17888   44      *       *       *       *       97.5    SH028821.07FU_EF679363_refs     *

D       25193   2       *       *       *       *       97.4    SH014640.07FU_UDB017896_refs    *

D       29378   3       *       *       *       *       97.2    SH018304.07FU_UDB015265_refs    *

D       36805   3       *       *       *       *       97.3    SH012620.07FU_UDB011614_refs    *

log file attached

-Brett

[TonyWalters]

Okay, so it seems we're getting somewhere now.

The limited number of OTUs is indeed due to closed-reference picking. But, there is another possibility-you should have a /developer/ folder in your UNITE database, and I'd point to those files (the dynamic ones are supposed to be better for taxonomic resolution) and see if that increases the number of OTUs picked. If that increases your OTU count, I'd also set it as your reference file for taxonomic assignments.

You'll want to run the cleaning script on the fasta and taxonomic files before using them.

[Brett]

Tony, 

The dynamic database in the developer folder increased the number of otus, but a large number of failures still occurred. For these data, 273 otus does seem ecologically reasonable. Still, should I be getting this many failures? 

I cleaned this fasta file before running the pick_otus.py. script as you suggested. The log file is attached. 

Thanks, 

Brett

[TonyWalters]

I think there still are quite a few gaps for the ITS data, so I'd still expect quite a few failures, but the /developer/ database looks like the way to go for now. You can do the original open-reference OTU picking workflow to capture the rest of the data that doesn't match the reference database at 97% identity. 

[Brett]

Tony, 

The open reference script proceeded much further than before, but am getting an error on the assign taxonomy steps. I didn't pull the entire log file off the server since I am at home, but I have attached the first few lines and the error message. I guess I can take the resulting fna file and pass that through assign_taxonomy.py separately, but it would be great if I could get the entire script to complete without an error. 

You have been very helpful and I apologize for going through all of those troubleshooting steps earlier, only to find that my reference and taxonomy files were messed up. Not sure why that was the case. Regardless, thanks again for taking the time. 

Best, 

Brett

[TonyWalters]

Well, that's a new error message number for me. If I had to guess, there is something going on with the blastall/formatdb version or installation.

What happens if you run:

formatdb -l "parsed_97_dynamic.fasta.log" -o T -n "parsed_97_dynamic.fasta" -i "/home/obrett/UNITE2/parsed_97_dynamic.fasta" -p F

directly?

[Brett]

Something is not right:

-bash: /home/obrett/blast-2.2.22/bin/formatdb: cannot execute binary file

[TonyWalters]

Okay, so I would suggest reinstalling the blast software (includes formbatdb and blastall). Here is the link to the Linux version:

http://mirrors.vbi.vt.edu/mirrors/ftp.ncbi.nih.gov/blast/executables/release/2.2.22/blast-2.2.22-x64-linux.tar.gz

You should be able to run:
formatdb

and

blastall

at the command line and get the software options returned.

[Brett]

Tony, 

The new blast is installed and I continue to progress further. However, it looks like I now need to point BLASTMAT to the NCBI database or create an .ncbirc file. Any suggestions on which avenue I should pursue? 

Also, thanks for providing the link for blast-2.2.22. I had a tough time previously finding a link for legacy blast.

Cheers, 

Brett

[TonyWalters]

I would create a .ncbirc file in your home directory. I've copied some instructions from the macqiime installation page (http://www.wernerlab.org/software/macqiime/macqiime-installation/installing-blast-in-os-x)

Make a ~/.ncbirc file that points to the BLAST data directory:

nano ~/.ncbirc

And put this text into the file:

[NCBI]
Data=/opt/blast-2.2.22/data/

You would modify the path to point to the /data/ folder from wherever you extracted the blast software.

[Brett]

Getting further each time; however, my .biom file with taxonomy is no good and I received an error that an empty fasta file was passed to the filter_alignment script. Either the align_seqs command failed or my filtering requirements were too stringent. 

-Brett 

[TonyWalters]

Generally you can't build alignments from ITS data-the region is too variable. You might be able to build trees if you had very specific, related taxa targeted with primers, and then built a de novo alignment (pynast with the greengenes core alignment won't work). I would suppress alignment and tree building, and use non-phylogenetic metrics for your data, e.g. bray-curtis for beta diversity distance.

[Brett]

Got it. I did provide the --suppress_align_and_tree argument in my params file. Why no functional biom file with tax, though? 

Thanks, 

Brett

[TonyWalters]

--suppress_align_and_tree is a parameter you pass directly when calling pick_open_reference_otus.py rather than in the parameters file. The full workflow script needs to complete for it to add the taxonomy metadata to the biom file.

[Brett]

Okay, is there a way to tell which parameters are able to be included in a separate file and which must be called at the command line? I've looked on the QIIME website and also using the -h option to see if they are differentiated somehow, but don't see anything. 

Also, in order to add taxonomy to a biom file in the pick_open_reference_otus script, should this argument not be passed (i.e. allow align, even though phylo trees aren't appropriate for ITS)? I believe I have gotten a biom table with tax successfully before using this script, but after using the argument in the command line.

[TonyWalters]

For the workflow script, -h after calling that script will show you what options get directly passed when calling pick_open_reference_otus.py. For each script called within the workflow script, e.g. pick_otus.py, assign_taxonomy.py, etc. you could call -h for each those to see the options. Those parameters would be passed in the parameters file.

taxonomic assignment will be done unless you pass --suppress_taxonomy_assignment, which is different than --suppress_align_and_tree

[Brett]

Hey Tony, 

After installing the linux blast 2.2.22 and making an .ncbirc file that points to the directory, pick_open_reference_otus.py is running smoothly. I had a few issues when trying to pass the core_diversity_analyses.py script, as I had the newer version of matplotlib installed (1.5.0) and had to change it to the older version (1.4.3). Also, our server had no graphic support, so I had to create a config file for matplotlib to get it to work. Things seem good for now. 

Thank you for all of your persistent help and advice over the past several days. You are doing a great job of supporting microbial ecologists! 

-Brett 

[Colin Brislawn]

Hello Brett,

Thanks for sticking with us through this process. The errors you had with matplotlib are being addressed over here, but the fix is not in the main branch yet and lots of folks are having problems: 

https://github.com/biocore/qiime/issues/2117 

Glad you got it working! Let us know if you have any other questions,

Colin 

[Brett]

Thank you, Colin!