The difference between unclassified, unidentified, others?

[Ahmed Abdelfattah]

Hello everyone!

I know this was discussed in a way before but I would like to get a clear idea about the meaning of the unclassified, unidentified, others that I get in my do assign taxonomy and plot taxa summary? 

in addition I would like to ask about constructing trees and the commands that require trees, I'm working with ITS sequences and it seems that making trees forfungal sequnces with qiime is not yet well established since many posts have spoken about this issue previously

Thanks a lot for your support 

Have a productive day everyone 

Ahmed

[Yoshiki Vázquez Baeza]

Hello Ahmed,

For your question about the differences, please see these two threads (thanks W. Jai for the links):

https://groups.google.com/d/msg/qiime-forum/uJ89GGM6C8E/d3d9XAyxS_kJ

https://groups.google.com/forum/#!msg/qiime-forum/bMM5i67S2Rk/7XRqCt0kfIkJ

Thanks!

Yoshiki.

--
 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ahmed Abdel Fattah]

Dear Yoshiki,

Thanks for your reply, I didn't want to contact you in the holidays not to disturb. As I understood from the threads you sent me "others" means the sequence is ambiguous - is it only for quality or read length reasons? - also I still didn't know what 1- Unclassified;Other 

2- k__Fungi;Other 

3- k__Fungi;p__unidentified? 

Thanks in advance, and Happy new year! 

Ahmed

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "Qiime Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/qiime-forum/JSaRE9xCN4k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to qiime-forum...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Ahmed Abdel Fattah

[Antonio González Peña]

Out of curiosity and to give you a better answer, which reference and
classifier are you using?

Antonio

[Ahmed Abdel Fattah]

Hi Antonio,

I'm using UNITE database and usearch61, is this what you are asking about?

Thanks for offering to help 

[Antonio González Peña]

Thanks for the info. UNITE makes sense and my guess is that you are
assigning taxonomies using RDP and not usearch61.

Anyway, there are different things going on here but basically (copied
from one of the links previously posted): The "Other" assignments are
due to ambiguity when the RDP classifier tries to assign below the
order level in this case (can't decide between distinct taxa).
Additionally, UNITE has some species names that have no lower levels
identified so you will get p__unidentified or o__unidentified and a
know species.

Does this make sense?


On Wed, Jan 1, 2014 at 10:17 AM, Ahmed Abdel Fattah

[Ahmed Abdel Fattah]

thanks for the reply

I attached the txt and log files -the output of  pick_open_reference_otus.py command. 

I made a mistake it's not usearch61 it's uclust, here's the command:

pick_open_reference_otus.py 

-i myfile.fna 

-o output director 

-r '/home/qiime/Desktop/Shared_Folder/UNITE/unite_taxonomy_21nov2011/unite_ref_seqs_21nov2011.fasta' 

-m uclust 

-aO 8

if it's true that I used RDP is normal to have results as shown in the txt file. I mean it assigned taxonomy although it's a database for bacteria?

if this is an issue what do I need to do? Please notice that I tried to use -p params to indicate that  I want to use this specific classifier and database but I didn't succeed 

[Antonio González Peña]

I guess that now you are clear on your original questions, right?

A couple of things:
- There is a new release of UNITE, which can be downloaded from here:
http://unite.ut.ee/repository.php. Suggest using this one.
- Looking at the log and your command, it seems that you picked by
reference (first step of the open reference pipeline) against UNITE
but assigned taxonomy using Greengenes. Suggest redoing using the new
release and configuring the assign_taxonomy.py step using the
parameters file, here is the tutorial
http://qiime.org/documentation/qiime_parameters_files.html.

On Wed, Jan 1, 2014 at 11:33 AM, Ahmed Abdel Fattah

[Ahmed Abdel Fattah]

Antonio,

I will try it and let you know how it went. In the mean while please confirm if I'm right about my previous doubts, 1- the "others" was due to RDP and once I assign taxonomy with uclust or usearch61 it should be fine, is that right?

2- I don't need to do pick_open_reference_otus.py instead assign_taxonomy.py is enough, and I can replace the output with what I had?

It might seem obvious for some but for beginners like me it needs a bit of patience.

I sincerely thank you for your help

[Antonio González Peña]

1. Yes but you should use uclust not usearch61
2. Yes but you should clarify that in your methods; picked with an
older version of UNITE and assigned taxonomy with a newer one. Note
that this will change your OTUs a little bit, not sure how much, but
that should not change your overall results.

On Wed, Jan 1, 2014 at 2:56 PM, Ahmed Abdel Fattah

[Ahmed Abdel Fattah]

Thanks for your reply,

I tried to do assign taxonomy but it didn't work and there was this error,

qiime@qiime-VirtualBox:~$ assign_taxonomy.py -i '/home/qiime/Desktop/Shared_Folder/New_Analysis/Mainrequiredfiles/seqs_rev_primer_truncated_modifiedmp.fna' -r '/home/qiime/Desktop/Shared_Folder/UNITE/sh_qiime_release_19.12.2013/sh_qiime_release_19.12.2013/sh_refs_qiime_ver6_97_19.12.2013.fasta' -t '/home/qiime/Desktop/Shared_Folder/UNITE/sh_qiime_release_19.12.2013/sh_qiime_release_19.12.2013/sh_taxonomy_qiime_ver6_97_19.12.2013.txt' -o '/home/qiime/Desktop/Shared_Folder/New_Analysis/2014_assign_taxa/Assign_taxaonomy_output' 

Traceback (most recent call last):

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/assign_taxonomy.py", line 307, in <module>

    main()

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/assign_taxonomy.py", line 285, in main

    result_path=temp_result_path,log_path=log_path)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/assign_taxonomy.py", line 425, in __call__

    max_memory=max_memory, tmp_dir=tmp_dir)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 509, in train_rdp_classifier_and_assign_taxonomy

    tmp_dir=tmp_dir)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 479, in train_rdp_classifier

    return app(training_seqs_file)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/pycogent_backports/rdp_classifier.py", line 321, in __call__

    result = super(RdpClassifier, self).__call__(data=data, remove_tmp=remove_tmp)

  File "/home/qiime/qiime_software/pycogent-1.5.3-release/lib/python2.7/site-packages/cogent/app/util.py", line 251, in __call__

    open(errfile).read())

cogent.app.util.ApplicationError: Unacceptable application exit status: 1

Command:

cd "/home/qiime/"; java -Xmx1500M -cp "/home/qiime/qiime_software/rdpclassifier-2.2-release/rdp_classifier-2.2.jar" edu.msu.cme.rdp.classifier.train.ClassifierTraineeMaker "/tmp/RdpTaxonomy_IhY5dO.txt" "/tmp/tmpfdMvsOceuiqdOfbNzbRb.txt" 1 version1 cogent "/tmp/RdpTrainer_0gvFlr" > "/tmp/tmpqEpCwfkOGB01xxtBOgML.txt" 2> "/tmp/tmpUmAWKXbM2xGQutt3TAZl.txt"

StdOut:

StdErr:

Copyright 2006 Michigan State University Board of Trustees.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

Authors's mailng address:

Center for Microbial Ecology

2225A Biomedical Physical Science

Michigan State University

East Lansing, Michigan USA 48824-4320

E-mail: James R. Cole at co...@msu.edu

Qiong Wang at wang...@msu.edu

James M. Tiedje at tie...@msu.edu

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

at edu.msu.cme.rdp.classifier.train.RawHierarchyTree.initWordOccurrence(RawHierarchyTree.java:108)

at edu.msu.cme.rdp.classifier.train.TreeFactory.addSequencewithLineage(TreeFactory.java:211)

at edu.msu.cme.rdp.classifier.train.TreeFactory.addSequence(TreeFactory.java:139)

at edu.msu.cme.rdp.classifier.train.ClassifierTraineeMaker.<init>(ClassifierTraineeMaker.java:47)

at edu.msu.cme.rdp.classifier.train.ClassifierTraineeMaker.main(ClassifierTraineeMaker.java:133)

Then I tried to do pick_open_reference_otus.py and I got this error 

qiime@qiime-VirtualBox:~$ pick_open_reference_otus.py -i '/home/qiime/Desktop/Shared_Folder/New_Analysis/Mainrequiredfiles/seqs_rev_primer_truncated.fna' -o '/home/qiime/Desktop/Shared_Folder/New_Analysis/2014_assign_taxa/2014taxa' -r '/home/qiime/Desktop/Shared_Folder/UNITE/sh_qiime_release_s_19.12.2013/sh_qiime_release_s_19.12.2013/sh_refs_qiime_ver6_97_s_19.12.2013.fasta' 

Traceback (most recent call last):

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/pick_open_reference_otus.py", line 249, in <module>

    main()

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/pick_open_reference_otus.py", line 222, in main

    status_update_callback=status_update_callback)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/workflow/pick_open_reference_otus.py", line 861, in pick_subsampled_open_reference_otus

    status_update_callback=status_update_callback)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/workflow/pick_open_reference_otus.py", line 312, in align_and_tree

    close_logger_on_success=close_logger_on_success)

  File "/home/qiime/qiime_software/qiime-1.7.0-release/lib/qiime/workflow/util.py", line 117, in call_commands_serially

    raise WorkflowError, msg

qiime.workflow.util.WorkflowError: 

*** ERROR RAISED DURING STEP: Filter alignment

Command run was:

 filter_alignment.py -o /home/qiime/Desktop/Shared_Folder/New_Analysis/2014_assign_taxa/2014taxa/pynast_aligned_seqs -i /home/qiime/Desktop/Shared_Folder/New_Analysis/2014_assign_taxa/2014taxa/pynast_aligned_seqs/rep_set_aligned.fasta 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/filter_alignment.py", line 146, in <module>

    main()

  File "/home/qiime/qiime_software/qiime-1.7.0-release/bin/filter_alignment.py", line 94, in main

    raise ValueError, ("An empty fasta file was provided. "

ValueError: An empty fasta file was provided. Did the alignment complete sucessfully? Did PyNAST discard all sequences due to too-stringent minimum length or minimum percent ID settings?

and the OTU table has bacterial classification although Indicated in the parameter file the UNITE files. please notice that I downloaded the UNITE files form the link you sent me, where it says qiime releases and downloaded this one:

6

2013-12-19

2 452

43 492

Current

Download

As above, but also comprises all global and 97% singletons.

Please find  attached below the parameters file and the  out_table,

Thanks a lot for your advice

[Antonio González Peña]

Error with assign_taxonomy.py: I think you are using an older version
of QIIME due to the default amount of RAM that this process is
requesting from Java (-Xmx1500M). Anyway, you should be able to use
--rdp_max_memory 4000 to fix it. If you get the same error use 8000.

Error with pick_open_reference_otus.py:By default this script will try
to align your sequences and build a tree, which is not possible for
ITS. Anyway, have you checked the output folder? My guess is that you
have an OTU table or at least you should have everything to build one.
BTW in the newest version of this script you can pass
--suppress_align_and_tree to avoid these steps.

> 62013-12-19 2 45243 492 CurrentDownload As above, but also comprises all

[Devin Leopold]

I have also recently tried the -m uclust option with the assign_taxonomy.py using the UNITE database as a reference. At first I was very disappointed with the results because I was not able to assign anything to most of my sequences, even at low sequence similarity. Because UCLSUT is a semi-global alignment, I did not think it should matter that the UNITE reference sequences are mostly longer than my amplicons (my amplicons average ~225 and unite amplicon average ~550). But, I tried trimming the reference database using my primers and using the trimmed reference seqs in assign taxonomy. I found that this led to a much higher proportion of sequences being assigned! -Devin

[Ahmed Abdel Fattah]

Hi Again,

So that was right I updated qiime to 1.8 and I didn't need to specify the amount of memory while executing the command assign_taxonomy.py. I did it and I got 2 (log and txt) files as the output, but I don't know how to go on for further analysis without an OTU table nor a tree.

here is the command in case I made something wrong 

assign_taxonomy.py -i '/home/qiime/Desktop/Shared_Folder/New_Analysis/2014_assign_taxa/seqs_rev_primer_truncated_modifiedmp.fna' -r '/home/qiime/Desktop/Shared_Folder/UNITE/sh_qiime_release_19.12.2013/sh_qiime_release_19.12.2013/sh_refs_qiime_ver6_97_19.12.2013.fasta' -t '/home/qiime/Desktop/Shared_Folder/UNITE/sh_qiime_release_19.12.2013/sh_qiime_release_19.12.2013/sh_taxonomy_qiime_ver6_97_19.12.2013.txt' 

2- I tried the command  pick_open_reference_otus.py but I guess I have a problem understanding the concept of reference OTU, so I downloaded several files from UNITE and I dunno which one I should include in the parameter file and which I should include in the command as -r. ? and is there difference between the UNITE OTUs (ITS) in "Data Files and Other Resources

", and the ones available in 

http://unite.ut.ee/repository.php

is the first one old the second one is an update or I'm totally wrong?

Thanks a a lot as usual, I'm grateful to your help

Ahmed

[Antonio González Peña]

1. Use non phylogenetic metrics for alpha (do alpha_diversity.py -s to
see the possible options) and beta diversity (do beta_diversity.py -s
to see the possible options)

2. Even if you pick by reference there is no tree so you might want to
stay with the analysis you already have. Anyway, you should use the
first one, top, of the list. More info:
http://qiime.org/tutorials/otu_picking.html#closed-reference-otu-picking

On Tue, Jan 7, 2014 at 10:57 AM, Ahmed Abdel Fattah