unable to use usearch61 in pick_closed_reference_otus.py

58 views
Skip to first unread message

Angela Zou

unread,
Jul 18, 2017, 12:04:09 PM7/18/17
to Qiime 1 Forum
Hello,

I submitted the following command 

pick_closed_reference_otus.py -i seqs.fna -o study_closed_otus -p $location/closed_otus_params.txt

with the options

pick_otus:otu_picking_method usearch61
pick_otus:enable_rev_strand_match True 

in closed_otus_params.txt.

When I run the command, I get the error

Traceback (most recent call last):
  File "/hpf/tools/centos6/python/2.7.9/bin/pick_closed_reference_otus.py", line 233, in <module>
    main()
  File "/hpf/tools/centos6/python/2.7.9/bin/pick_closed_reference_otus.py", line 224, in main
    status_update_callback=status_update_callback)
  File "/hpf/tools/centos6/python/2.7.9/lib/python2.7/site-packages/qiime/workflow/upstream.py", line 348, in run_pick_closed_reference_otus
    % (otu_picking_method, ' '.join(reference_otu_picking_methods))
AssertionError: Invalid OTU picking method supplied: usearch61. Valid choices are: blast uclust_ref usearch61_ref usearch_ref sortmerna


This is confusing because in upstream.py usearch61 is indeed not an option, but it is accepted in pick_otus.py. My goal is to actually run vsearch according to the instructions here: https://groups.google.com/d/msg/qiime-forum/oPqd7dzxQO0/8tPc3yA9FgAJ
I think I implemented it correctly because when I type in usearch61 --version, I get 
vsearch v2.4.1_linux_x86_64, 126.1GB RAM, 32 cores


So I am not sure why usearch61 is an invalid option. Thanks. 

Colin Brislawn

unread,
Jul 18, 2017, 7:30:08 PM7/18/17
to Qiime 1 Forum
Good afternoon,

Here is what I think is happening: pick_closed_reference_otus only does ref-based OTU picking, so this script wants your paramater to say "usearch61_ref" instead of "usearch61." I know this is a little silly, but I think that might allow the script to finish.

Let me know what you find,
Colin

Angela Zou

unread,
Jul 19, 2017, 12:20:09 PM7/19/17
to Qiime 1 Forum
Hi Colin,

The script does run now, but it is taking a much longer time (or maybe it is hanging?) than uclust, I left the script running yesterday afternoon and it is still not done currently. For my current dataset, I can just run uclust but in the future I would like to switch over to vsearch if possible. 

Thanks,

Angela 

Colin Brislawn

unread,
Jul 19, 2017, 1:18:54 PM7/19/17
to qiime...@googlegroups.com
Hello Angela,

It's good to head that it's running.

OTU picking does take a long time, as it has to process your full dataset down to a single .biom table. This process can be sped up by using multiple threads so that it runs some parts of this process in parallel. Are you using -a -O to run this script in parallel?
You can lean more about these settings here: http://qiime.org/scripts/pick_closed_reference_otus.html 

Edit: I think running vsearch is a great idea! Qiime 1 does not support vsearch, but Qiime 2 will. https://docs.qiime2.org 
You could also make a .biom table with vsearch by running vsearch directly, but a little outside the scope of these forums.


Colin

Angela Zou

unread,
Jul 19, 2017, 1:53:53 PM7/19/17
to Qiime 1 Forum
Hi Colin,

What I meant was that I don't think the script is actually running properly because using default uclust it took less than an hour, while vsearch is taking more than a day. I will definitely have to take a look at Qiime 2, thanks. 


Angela 

Colin Brislawn

unread,
Jul 19, 2017, 2:23:41 PM7/19/17
to qiime...@googlegroups.com
Hello Angela,

Oh, sorry for my mistake. I should have read your post more carefully and noticed that you are using the vsearch / usearch workaround. This method is not supported in qiime 1 because it works on some scripts, but not in others. I'm not sure what's causing your particular error. 

Closed-ref clustering is a pretty robust method, so you should see similar results by using uclust, usearch61, or vsearch. The database has a much larger effect than the clustering algorithm. Would you like to try running one of these methods, or would you like to try closed-ref clustering using vsearch directly?

Colin


Angela Zou

unread,
Jul 19, 2017, 2:52:28 PM7/19/17
to Qiime 1 Forum
Hi Colin,

Sticking with uclust is fine for now, I am mainly trying vsearch because I heard it is must faster than uclust, which will be advantageous for down the road when I get a lot more data. I think in the future I will just try closed-ref with vsearch directly. Thanks. 

Angela 

Colin Brislawn

unread,
Jul 19, 2017, 6:07:25 PM7/19/17
to Qiime 1 Forum
Hello Angela,

It's definitely good to try out different methods and software! While some parts of vsearch are faster than uclust / usearch, the main functions of search and clustering of vsearch are slightly slower because the implementation of the algorithm uses exact alignments. I still like vsearch, partly because it uses these exact alignments, and also because it's open source and can scale to very large databases. Usearch requires a paid license if you want to use the 64-bit version that can handle larger files.

I'm glad uclust is working well for you. Let me know if you have any other questions,
Colin

Reply all
Reply to author
Forward
0 new messages