unable to use usearch61 in pick_closed_reference_otus.py

[Angela Zou]

Hello,

I submitted the following command 

pick_closed_reference_otus.py -i seqs.fna -o study_closed_otus -p $location/closed_otus_params.txt

with the options

pick_otus:otu_picking_method usearch61

pick_otus:enable_rev_strand_match True 

in closed_otus_params.txt.

When I run the command, I get the error

Traceback (most recent call last):

  File "/hpf/tools/centos6/python/2.7.9/bin/pick_closed_reference_otus.py", line 233, in <module>

    main()

  File "/hpf/tools/centos6/python/2.7.9/bin/pick_closed_reference_otus.py", line 224, in main

    status_update_callback=status_update_callback)

  File "/hpf/tools/centos6/python/2.7.9/lib/python2.7/site-packages/qiime/workflow/upstream.py", line 348, in run_pick_closed_reference_otus

    % (otu_picking_method, ' '.join(reference_otu_picking_methods))

AssertionError: Invalid OTU picking method supplied: usearch61. Valid choices are: blast uclust_ref usearch61_ref usearch_ref sortmerna

This is confusing because in upstream.py usearch61 is indeed not an option, but it is accepted in pick_otus.py. My goal is to actually run vsearch according to the instructions here: https://groups.google.com/d/msg/qiime-forum/oPqd7dzxQO0/8tPc3yA9FgAJ

I think I implemented it correctly because when I type in usearch61 --version, I get 

vsearch v2.4.1_linux_x86_64, 126.1GB RAM, 32 cores

https://github.com/torognes/vsearch

So I am not sure why usearch61 is an invalid option. Thanks. 

[Colin Brislawn]

Good afternoon,

Here is what I think is happening: pick_closed_reference_otus only does ref-based OTU picking, so this script wants your paramater to say "usearch61_ref" instead of "usearch61." I know this is a little silly, but I think that might allow the script to finish.

Let me know what you find,

Colin

[Angela Zou]

Hi Colin,

The script does run now, but it is taking a much longer time (or maybe it is hanging?) than uclust, I left the script running yesterday afternoon and it is still not done currently. For my current dataset, I can just run uclust but in the future I would like to switch over to vsearch if possible. 

Thanks,

Angela 

[Colin Brislawn]

Hello Angela,

It's good to head that it's running.

OTU picking does take a long time, as it has to process your full dataset down to a single .biom table. This process can be sped up by using multiple threads so that it runs some parts of this process in parallel. Are you using -a -O to run this script in parallel?

You can lean more about these settings here: http://qiime.org/scripts/pick_closed_reference_otus.html 

Edit: I think running vsearch is a great idea! Qiime 1 does not support vsearch, but Qiime 2 will. https://docs.qiime2.org 

You could also make a .biom table with vsearch by running vsearch directly, but a little outside the scope of these forums.

Colin

[Angela Zou]

Hi Colin,

What I meant was that I don't think the script is actually running properly because using default uclust it took less than an hour, while vsearch is taking more than a day. I will definitely have to take a look at Qiime 2, thanks. 

Angela 

[Colin Brislawn]

Hello Angela,

Oh, sorry for my mistake. I should have read your post more carefully and noticed that you are using the vsearch / usearch workaround. This method is not supported in qiime 1 because it works on some scripts, but not in others. I'm not sure what's causing your particular error. 

Closed-ref clustering is a pretty robust method, so you should see similar results by using uclust, usearch61, or vsearch. The database has a much larger effect than the clustering algorithm. Would you like to try running one of these methods, or would you like to try closed-ref clustering using vsearch directly?

Colin

[Angela Zou]

Hi Colin,

Sticking with uclust is fine for now, I am mainly trying vsearch because I heard it is must faster than uclust, which will be advantageous for down the road when I get a lot more data. I think in the future I will just try closed-ref with vsearch directly. Thanks. 

Angela 

[Colin Brislawn]

Hello Angela,

It's definitely good to try out different methods and software! While some parts of vsearch are faster than uclust / usearch, the main functions of search and clustering of vsearch are slightly slower because the implementation of the algorithm uses exact alignments. I still like vsearch, partly because it uses these exact alignments, and also because it's open source and can scale to very large databases. Usearch requires a paid license if you want to use the 64-bit version that can handle larger files.

I'm glad uclust is working well for you. Let me know if you have any other questions,

Colin