Beta_diversity_through_plots.py error: No valid samples.environments found

[Rachel C.]

Hi all

I am trying to do a beta diversity analysis with the beta_diversity_through_plots.py script (I have also tried with the beta_diversity.py script) and keep getting the error "No valid samples/environments found. Check whether tree tips match otus/taxa present in samples/environments"

Input:

beta_diversity_through_plots.py -i otu_table_mc2_w_tax_F_L6.biom -o bdiv_through_plots/ -t rep_phylo.tre -m RC_summary_unix_corrected.txt

*The rep_phylo.tre file is the output of the make_phylogeny.py command I ran on a .fna file of my forward read sequences. The mapping file has been validated and worked fine with other scripts.

Here is the output/error:

Traceback (most recent call last):

  File "/Users/rcapouya/.conda/envs/qiime/bin/beta_diversity_through_plots.py", line 153, in <module>

    main()

  File "/Users/rcapouya/.conda/envs/qiime/bin/beta_diversity_through_plots.py", line 150, in main

    status_update_callback=status_update_callback)

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/qiime/workflow/downstream.py", line 183, in run_beta_diversity_through_plots

    close_logger_on_success=close_logger_on_success)

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/qiime/workflow/util.py", line 122, in call_commands_serially

    raise WorkflowError(msg)

qiime.workflow.util.WorkflowError: 

*** ERROR RAISED DURING STEP: Beta Diversity (weighted_unifrac)

Command run was:

 beta_diversity.py -i otu_table_mc2_w_tax_F_L6.biom -o bdiv_through_plots/ --metrics weighted_unifrac  -t rep_phylo.tre 

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/Users/rcapouya/.conda/envs/qiime/bin/beta_diversity.py", line 152, in <module>

    main()

  File "/Users/rcapouya/.conda/envs/qiime/bin/beta_diversity.py", line 145, in main

    opts.output_dir, opts.rows, full_tree=opts.full_tree)

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/qiime/beta_diversity.py", line 180, in single_file_beta

    make_subtree=(not full_tree))

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/qiime/beta_metrics.py", line 44, in result

    is_symmetric=is_symmetric, modes=["distance_matrix"], **kwargs)

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/cogent/maths/unifrac/fast_unifrac.py", line 466, in fast_unifrac

    envs, count_array, unique_envs, env_to_index, node_to_index, env_names, branch_lengths, nodes, t = _fast_unifrac_setup(t, envs, make_subtree)

  File "/Users/rcapouya/.conda/envs/qiime/lib/python2.7/site-packages/cogent/maths/unifrac/fast_unifrac.py", line 194, in _fast_unifrac_setup

    raise ValueError, "No valid samples/environments found. Check whether tree tips match otus/taxa present in samples/environments"

ValueError: No valid samples/environments found. Check whether tree tips match otus/taxa present in samples/environments

In other similar questions, posters were using a tree from the greengenes fileset, however I used UNITE for these fungal OTUS and there is no .tre file in that set of files. I am not sure if I am using the wrong file or what. I apologize if this question has been asked before, I didn't find any solutions in previously answered questions that solved my issue.

Thank you,

Rachel

[Colin Brislawn]

Hello Rachel,

Let's make sure you are using the right tree file. I'll walk you through the process of building a proper tree. 

Like you mentioned, some qiime users have used the greengenes tre with this script. This only works is they performed closed-ref OTU picking because the OTUs will be the same as those in that greengenes tree. For all other qiime users (who don't perform closed-ref OTU picking) you will have to build a tree from your OTU centroids.

*The rep_phylo.tre file is the output of the make_phylogeny.py command I ran on a .fna file of my forward read sequences. 

Try building a tree from your OTU centroids, the output file with a name like rep_set.fna, like this:

1. align your sequences (maybe using muscle if there is no aligned referance)
2. filter those alignments
3. built a tree from them

Here are the three scripts:

http://qiime.org/scripts/align_seqs.html

http://qiime.org/scripts/filter_alignment.html

http://qiime.org/scripts/make_phylogeny.html

Once you built a tree from your OTU centroids, "tree tips [will] match otus/taxa present in samples/environments".

Good luck!

Colin Brislawn 

[Rachel C.]

Thanks Colin, I will try to go through it that way. 

I do have a question about this though:

"This only works is they performed closed-ref OTU picking because the OTUs will be the same as those in that greengenes tree. For all other qiime users (who don't perform closed-ref OTU picking) you will have to build a tree from your OTU centroids."

Isn't using UNITE also closed-ref, just a different set of OTUS being examined than greengenes (16S vs ITS)? 

[Colin Brislawn]

Unite and greengenes are both databases for different marker genes. Any database can used with any OTU picking method, whether that's a closed-ref method or open-ref method. While databases and OTU picking methods are often used together, they are independant pieces which can be swapped out. 

What kind of OTU picking did you use? Open-ref, closed-ref, or de novo? What algorithm did you use? The uclust algorithm is the default in qiime, but others like cd-hit and swarm are also available. 

Colin Brislawn 

[Rachel C.]

I did use pick_open_reference_otus.py now that I look back at my script and just used the default UCLUST algorithm. It was a little while ago so I forgot I did open reference. Sorry, I'm new-ish to this facet of research and I'm still getting things mixed up.

-Rachel

[Colin Brislawn]

No worries Rachel! I'm glad we have the opportunity to loop back and take a look at what the indevidual scripts are doing. Let me know if you have any other questions while running those scripts.

Colin

[Rachel C.]

Thanks Colin

I am actually running into an error saying that the sequences must be the same length to run the align_seqs.py script

I found a post in the forum of someone having the same issue (I am not sure why I didn't have this issue previously seeing as I have gotten all the way to OTU tables before without this error), so hopefully someone is able to find a solution

https://groups.google.com/forum/#!topic/qiime-forum/8PMVthpvSpY

[Colin Brislawn]

Oh, if you are the Rachel from that other thread, the issue be with associated with ITS and may not be easily 'solvable;' ITS reads are diverse which makes alignment hard. I can help you run down this alignment problem, or you can try a non-phylogenetic approach as a suggest in that other thread. 

Keep in touch!

Colin

[Rachel C.]

Yes, that's me in the other thread. I tried the Bray-Curtis method and it worked, though my principal_coordinates.py has a lot of negative values. I'll keep working on that and I'll keep trying to work with you to build the phylogenetic tree, even if it just for the experience of working with QIIME errors

Thanks for all your help!