HELP, results produced by old version qiime can not be reproduced by the new version

[yan he]

Hi Greg, Tony and Will,

I started a project about 1.5 year ago which compares datasets of same sampels produced by two different methods. At that time, Qiime does not use biome file instead of otu table (Sorry I don't remember the exact version, I think it is 1.3). Below is the results:
unweighted unifrac:                      

                                                                                                                                                        weighted unifrac:

tre was produced by pynast alignment (pynast_template_alignment_fp    core_set_aligned.fasta.imputed -e 50)
There are 8 colors in the figures with 4 paris (red-purple, blue-yellow, green-pink, oringe-cyan). It is very clear in the unweighted unifrac that samples are seperated by different pairs. In the weighted unifrac, though samples in each pair are seperated, they are still more similar to each other than from different pairs.

But now we've already updated to Qiime to 1.6, so when I redo the analysis (with the same clustering results and any commands used), what I got is this:
unwieghted unifrac:                                                                                                             weighted unifrac

This time, for the uniweghted unifrac, the old conclusion can not be drawn because the seperation of samples are different. For the weighted one, the pattern is somewhat similar but serves worse to my previous conclusion.

I further changed the alignment method to produce the tre file using muscle, this is what I had:
unweighted unifrac:                                                                                                                                                  weighted unifrac:

The unweighted unifrac is quite similar to the unweighted unifrac using the old Qiime.

But the weighted unifrac is a little bit messed up....

And I also tried bray_curtis distance to generate PCA, this is what I got:

Nice and clear.

Among the 4 methods, 3 unweighted ones (bray_curtis distance is generated in a unweighted way) shows the same pattern except the unweighted unifrac using pynast alignment in the Qiime 1.6.
For the 3 weighted methods, 3 patterns are revealed, but according to the composition of the 8 groups (see below), The first one (pynast, old Qiime) is the most consistent.

A,B,C and D are 4 pairs and two groups in each pair. In a weighted view, Groups in each pair are more similar to each other than from different pairs.
Sorry  that I have troubles to get the old version tested again, since we've already abundoned it. And I think there may be something wrong with the pynast alignment in the new version Qiime (at least in mine which I don't know how to determine). And I really appreciate if you could help me with this confusing situation!

Thanks,
Yan

[Will Van Treuren]

Hi Yan,

Please post your question in only one thread -- multiple threads are confusing and duplicate effort. We will respond as soon as we have an answer.

Thanks,

Will 

--
 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Will Van Treuren]

Hi Yan, 

This is a complex question to answer and I need a little bit more information. To make sure I understand your question I will rephrase in my own words and you can correct me if I am wrong. 

""

You completed beta diversity analysis in QIIME 1.3 (or 1.4) and used a tree created with the PyNAST algorithm to perform weighted and unweighted unifrac. You see clustering in the unewighted unifrac plots (along PC2 and PC3 most clearly) between the samples pairs you expected. The sample pairs are (V4F-V6R, V6F-V6R). 

You completed beta diversity analysis with QIIME 1.6 and used a tree created with the PyNAST algorithm to perform weighted and unweighted unifrac, and you also performed Bray Curtis. Although the unweighted unifrac PC1 explains significantly more of the variance of the data, you get clusters that aren't really as tight. The Bray Curtis plot, which does not use the tree, shows very tight clustering on expected lines. 

You are concerned that something has changed with the PyNAST algorithm and that would explain the difference in your results. 

""

If my understanding above is correct, I need to know what the exact steps (including commands) that you used in QIIME 1.3, and the exact steps, (including commands) that you used in QIIME 1.6. 

Thanks,

Will 

On Fri, Jun 7, 2013 at 12:01 PM, yan he <bioy...@gmail.com> wrote:

--

[yan he]

Thanks, Will.

 I edited in your rephrase as I thought this may be helpful to a better understanding. See below in red words:

在 2013年6月8日星期六UTC+8上午2时40分04秒，Will Van Treuren写道：

Hi Yan, 

This is a complex question to answer and I need a little bit more information. To make sure I understand your question I will rephrase in my own words and you can correct me if I am wrong. 

""

You completed beta diversity analysis in QIIME 1.3 (or 1.4) and used a tree created with the PyNAST algorithm to perform weighted and unweighted unifrac. You see clustering in the unewighted unifrac plots (along PC2 and PC3 most clearly I think it's PC1 and PC2) between the samples pairs you expected. The sample pairs are (V4F-V6R, V6F-V6R). In the meantime, weighted unifrac was also to my expect as they were consistent with the composition figure.

You completed beta diversity analysis with QIIME 1.6 and used a tree created with the PyNAST algorithm to perform weighted and unweighted unifrac, and you also performed Bray Curtis. Although the unweighted unifrac PC1 explains significantly more of the variance of the data, you get clusters that aren't really as tight. And the clustering pattern changed, (in the new pynast unweighted unifrac, samples are mainly seperated by two pairs, you can see red, green, blue and orange in the up-left while the other four in the right-bottom) thus I could not draw a conclution that samples are more similar to each other in one pair than to different pairs. The Bray Curtis plot, which does not use the tree, shows very tight clustering on expected lines. I also tried using muscle to generate the tre file for unifrac, the unweighted is quite similar to my old one but the weighted is not.

You are concerned that something has changed with the PyNAST algorithm (or the default reference dataset? which hard for me to found out since I don't know where to find the old reference) and that would explain the difference in your results. 

""

If my understanding above is correct, I need to know what the exact steps (including commands) that you used in QIIME 1.3, and the exact steps, (including commands) that you used in QIIME 1.6.

 
Steps in Qiime 1.3:

align_seqs.py -i rep.fa -o pynast -e 50
filter_alignment.py -i pynast/rep_aligned.fasta -o filter
make_phylogeny.py -i filter/rep_aligned_pfiltered.fasta -o tre
beta_diversity_through_plots.py -i otu_table -o bdiv/ -t tre -m mapping_file

Steps in Qiime 1.6:
align_seqs.py -i rep.fa -o pynast -e 50
filter_alignment.py -i pynast/rep_aligned.fasta -o filter
make_phylogeny.py -i filter/rep_aligned_pfiltered.fasta -o tre
convert_biom.py -i otu_table -o otu_biom --biom_table_type "otu table"
beta_diversity_through_plots.py -i otu_biom -o bdiv/ -t tre -m mapping_file

Because I wrote a homemade pipeline to do it, so I can assure you that they are almost identical except the convert steps

[Will Van Treuren]

Hi Yan, 

After some internal discussion, we don't think that the PyNAST template filepath has changed. It should always have been available from here under core_set_aligned.fasta.imputed.  Just to make sure, you did not repick otus, or repick your rep set between the time you used qiime 1.3 and qiime 1.6? 

Can you do diffs on similarly named files from your align_seqs.py 1.3 and 1.6 output and post them if there are any differences?

Thanks,

Will 

[yan he]

Hi Will,

Inspired by your idea, I looked further to check the whole process of generating the tre file (pynast, filter_alignment, make_phylogeny) and I think I found the problem which is in the filter_alignment.py.

I tested the PCoA analysis without filter_alignment (pynast alignment, the results with filter_alignment was already posted on the first floor), this is what I got:


unweighted (above) and weighted (below):
Pics(see below)

The unweighted unifrac pattern was almost reproduced and for the weighted one, by switching the angel, the result was also very similar to my old one. And based these results, I think my conclusion still works.
Nevertheless, as far as I am concerned, filter_alignment should not have so much impact on the clustering results (since I thought they were used mainly to reduce computational burdon). What do you think about this?

Thanks and sorry that I have troubles arranging the position of these figures. Hope you don't mind:)
Yan

[yan he]

Hi Will,

One more important finding, my original sequences are 60bp, but after filter_alignment.py, only about 15 bp were left and the rest were filtered.

Thanks,
Yan

[Will Van Treuren]

Hi Yan,

Filtering the alignment is mainly useful to get rid of the hypervariable bases in your alignment which send a confusing or misleading phylogenetic signal. I am concerned about how short your sequences are -- 60 bp is short to start, and 15 bp after lane masking is going to be very hard to use to get a meaningful tree. Can you tell me how your sequences were generated and what filtering/otu picking steps you applied?

Thanks,
Yan

--

 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.

[yan he]

Hi Will,

My sequences were from 16S rRNA V6 region, they are only around 65 bp and trimmed to 60 bp. I used TSC clustering method to do the otu picking (but I don't think otu pikcing method have an influence in the filtering step, since I tried uclust and the same situation happened). About filtering step, I used filter_alignment.py with default settings (only -i and -o were used). BTW, such problem didn't happen in the 1.3 version.

Please let me know if you need any other details.

Thanks,
Yan

在 2013年6月11日星期二UTC+8上午2时46分46秒，Will Van Treuren写道：

[Will Van Treuren]

Hi Yan, 

Can you post the exact commands you used for the filtering steps? We are looking for which lanemask file you used in 1.3 and now in 1.6. 

The fact that Bray-Curtis appears to perform really well on your data makes sense given the length of the sequences that are used to build the tree (too short to get good resolution). 

Thanks,

Will 

[yan he]

Hi Will,

I didn't specify -m while using filter_alignment.py, so my command was very simple: filter_alignment.py -i align.fa -o output_dir

I don't know what is lanemask file and what does it use for. And below is my config file, hope it helps.

# qiime_config
# WARNING: DO NOT EDIT OR DELETE Qiime/qiime_config
# To overwrite defaults, copy this file to $HOME/.qiime_config or a full path
# specified by $QIIME_CONFIG_FP and edit that copy of the file.

cluster_jobs_fp
python_exe_fp   python
working_dir
blastmat_dir
blastall_fp     blastall
pynast_template_alignment_fp    /usr/local/data/core_set_aligned.fasta.imputed
pynast_template_alignment_blastdb
template_alignment_lanemask_fp  /usr/local/data/lanemask_in_1s_and_0s.txt
jobs_to_start   1
seconds_to_sleep        2
qiime_scripts_dir       /usr/local/bin
temp_dir        /tmp/
denoiser_min_per_core   50
cloud_environment       False
topiaryexplorer_project_dir
torque_queue    friendlyq
assign_taxonomy_reference_seqs_fp
assign_taxonomy_id_to_taxonomy_fp
qiime_test_data_dir
sc_queue        all.q
 

在 2013年6月11日星期二UTC+8下午10时54分21秒，Will Van Treuren写道：

[Will Van Treuren]

Is there any way you can post the qiime config from your 1.3 run of this data?

Will 

[yan he]

Hi Will,

Sorry that we've already given up the old version and I don't know how to find the qiime config file back. I can only say that the default setting was used in the Qiime 1.3 for the qiime config.

Thanks,
Yan

在 2013年6月13日星期四UTC+8上午3时22分21秒，Will Van Treuren写道：

[Will Van Treuren]

Hi Yan,

Without access to complete information its going to be very difficult for me to completely reconstruct what happened and why you are getting the results you see. I think your best bet is to use the Bray-Curtis distance (it looked the best of any of the plots whether they were the 1.3 or 1.6 generated ones). Another possible idea is to do filtering based on base entropy rather than using the lane mask. This may retain more of the sequence information and give you better results. Look at the filter_alignment.py script for the options available in the entropy filtering regard. A final idea is to try and verify that the distance matrix you got from the QIIME 1.3 run is not actually similar to the distance matrix you got from the QIIME 1.6 run (just scaled and rotated or something) You could do this verification with a procrustes plot. Look here for more instructions.

Best, 

Will 

[yan he]

Hi Will,

After some trials, we decided to use jaccard distance to generate the PCA figure. I noticed that threre are abund and binary jaccard, do they respectively represent weighted and unweighted?

Thanks,
Yan

在 2013年6月14日星期五UTC+8下午2时35分20秒，Will Van Treuren写道：

[Will Van Treuren]

Hi Yan, 

Yes, binary Jaccard (and all the metrics in QIIME labeled binary) just take presence/absence data into account. The abundance Jaccard (and all the metrics in QIIME labeled abund) take into account the actual abundance in the calculations.

Best, 

Will 

[yan he]

Ok, Thanks!

在 2013年6月16日星期日UTC+8上午12时52分12秒，Will Van Treuren写道：