Beta diversity with unifrac metrics (tree file problems)

[Andrea Vidal Durà]

Hello,

I am working with data from Illumina MiSeq Platform. I have used UPARSE pipeline (in Usearch) to do all the data analysis but now for downstream analysis I am using different softwares including QIIME. 

I managed to get my OTU table in biom format and the tree file as well by different methods (-maketree command in muscle and also -make_phylogeny.py). In muscle the tree file is .phy format but I've been told is equivalent to .tre format so I just changed the extension (which I am not very confident with) before applying tge -beta_diversity command in qiime. 

At the end,  -beta_diversity.py with the weighted_unifrac metrics option DID NOT WORK  in any of my tests... 

Any help?

Thanks!

examples that did not work:

muscle -maketree -in seqs.afa -out seqs.phy

#I changed the extension .phy by .tre

beta_diversity.py -i otu_table_2.json -m weighted_unifrac -o beta_div_weighted/ -t seqs.tre

muscle -in otus_500000.fasta -out seqs.afa

# .afa is a fasta file with the sequences aligned. So it's a .fasta file: seqsaligned.fasta

make_phylogeny.py -i seqsaligned.fasta -t muscle

beta_diversity.py -i otu_table_2.json -m weighted_unifrac -o beta_diversityweighted/ -t seqsaligned.tre

[Embriette]

Hi Andrea,

Would you please send us your error outputs (copy paste is fine)?

Thanks!

Embriette

[Andrea Vidal Durà]

beta_diversity.py -i otu_table_2.json -m weighted_unifrac -o beta_diversityweighted/ -t seqsaligned.tre

Traceback (most recent call last):

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/bin/beta_diversity.py", line 152, in <module>

    main()

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/bin/beta_diversity.py", line 145, in main

    opts.output_dir, opts.rows, full_tree=opts.full_tree)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/qiime/beta_diversity.py", line 180, in single_file_beta

    make_subtree=(not full_tree))

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/qiime/beta_metrics.py", line 44, in result

    is_symmetric=is_symmetric, modes=["distance_matrix"], **kwargs)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/cogent/maths/unifrac/fast_unifrac.py", line 466, in fast_unifrac

    envs, count_array, unique_envs, env_to_index, node_to_index, env_names, branch_lengths, nodes, t = _fast_unifrac_setup(t, envs, make_subtree)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/cogent/maths/unifrac/fast_unifrac.py", line 194, in _fast_unifrac_setup

    raise ValueError, "No valid samples/environments found. Check whether tree tips match otus/taxa present in samples/environments"

ValueError: No valid samples/environments found. Check whether tree tips match otus/taxa present in samples/environments

{x-philes.leeds.ac.uk:ee11avd:881}%

[Embriette]

Hi Andrea,

It looks like the OTU IDs and the taxon IDs on your tree tips do not match, and this is why the command is failing. Can you manually check your OTU IDs in your .biom table and the taxonomy contained in the tree to see if this is the case? If they look equal, can you check if there is an underscore in the IDs? If so, you'll need to rename your ORU IDs and taxon IDs on the tree to eliminate this underscore (i.e., replace it with a "."). This is because in the Newick format, the underscore indicates a space: http://evolution.genetics.washington.edu/phylip/newicktree.html

Let me know how this goes!

Embriette

[Andrea Vidal Durà]

Hi Embriette,

I did have underscores, so as you suggested, I replaced it with a "." but I got the same problem running beta_diversity.py. Thanks for your help. I don't know what else to do. But thanks a lot for your suggestions!!

This is now the output in the server:

Traceback (most recent call last):

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/bin/beta_diversity.py", line 152, in <module>

    main()

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/bin/beta_diversity.py", line 145, in main

    opts.output_dir, opts.rows, full_tree=opts.full_tree)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/qiime/beta_diversity.py", line 145, in single_file_beta

    otu_table = load_table(input_path)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/biom/parse.py", line 551, in load_table

    table = parse_biom_table(fp)

  File "/apps/developers/libraries/qiime/1.9.1/1/bit-64-canopy-1.6.2/lib/biom/parse.py", line 309, in parse_biom_table

    t = Table.from_json(json.load(fp), input_is_dense=input_is_dense)

  File "/apps/developers/compilers/canopy/1.6.2/1/bit-64/appdata/canopy-1.6.2.3262.rh5-x86_64/lib/python2.7/json/__init__.py", line 290, in load

    **kw)

  File "/apps/developers/compilers/canopy/1.6.2/1/bit-64/appdata/canopy-1.6.2.3262.rh5-x86_64/lib/python2.7/json/__init__.py", line 338, in loads

    return _default_decoder.decode(s)

  File "/apps/developers/compilers/canopy/1.6.2/1/bit-64/appdata/canopy-1.6.2.3262.rh5-x86_64/lib/python2.7/json/decoder.py", line 366, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "/apps/developers/compilers/canopy/1.6.2/1/bit-64/appdata/canopy-1.6.2.3262.rh5-x86_64/lib/python2.7/json/decoder.py", line 382, in raw_decode

    obj, end = self.scan_once(s, idx)

ValueError: Expecting : delimiter: line 2 column 7 (char 10)

Thanks!!

[Embriette]

Hi Andrea,

Would you please send me your input files as well as the script you ran so I can take a look?

Thanks!

Embriette

[Embriette]

Hi Andrea,

When trying to convert your OTU table to txt format using biom convert, the script fails-both with biom 1.0 and with 2.1.3 (current version). Therefore, it seems that your OTU table is not formatted correctly. What I would recommend is that you recreate it, and avoid opening it in anything except for a plain text editor. At this point, run the biom summarize-table command on  your table to ensure it is in the correct format-if it isn't, that script will not work. Then, once you have an OTU table that will work, you can work on comparing your OTU table/mapping file/tree files to make sure the SampleIDs match between the mapping file and OTU table, and that the tree tips and OTU IDs in the OTU table match up (using a custom script here: https://gist.github.com/walterst/3144b1401fbda7bab464). Although you replaced the underscores with periods, it still doesn't look like your OTU IDs and tree tip IDs match up. For example, in seqs_new.tre, the tips look like this: OTU.10331;size=2; where the OTU ID in the OTU table is OTU.10331. The mismatch is even more apparent in seqsaligned.tre, where the tips look like this: seq_107. It's really important that the aligned fasta file you use for tree building has labels that match your OTU IDs, and that the output tree has tips that match these same OTU IDs, otherwise QIIME won't be able to connect the tree data to the OTU table.

Hope this helps!

Embriette