biom convert causes metadata to be lost

[Taruna]

Hello! I'm trying to add metadata from the mapping file to a biom table which was generated using Qiime 1.9.0. When I convert the biom table into a text file to check the table, none of the metadata columns show up. I understand that new version of Qiime uses a different format for biom tables. Anyone also experienced this issue? Below are link to my mapping file and the original biom table as well as the commands I'm running. Anyone has a suggestion for how not to lose metadata when convert biom table to tsv format please?

Files:

Mapping file is here.

Biom table is here.

Commands:

biom add-metadata -i bittermelon.biom -o bittermelon_w_metadata.biom --sample-metadata-fp 948-SoilProject-mapping.txt

biom add-metadata -i bittermelon.biom -o bittermelon_w_metadata_2.biom --sample-metadata-fp ../948-SoilProject-mapping.txt --sample-header Treatment,Host,Plant

biom convert -i bittermelon_w_metadata.biom -i bittermelon_w_metadata.txt --to-tsv

biom convert -i bittermelon_w_metadata_2.biom -i bittermelon_w_metadata_2.txt --to-tsv

Thanks for the help!

[Colin Brislawn]

Hello Taruna,

This page has lots of examples of using the 'biom convert' commands:

http://biom-format.org/documentation/biom_conversion.html 

I think adding the --header-key flag might be a step in the right direction. Note that a .txt biom files only support one metadata column.

I'm not sure if the classic .txt file format really supports metadata, compared to the JSON or HDF5 formats which support metadata very well for both samples and OTUs. May I ask about your next steps and continued goals?

Colin

[Taruna]

Hi Colin,

Thank you for your response! My ultimate goal is to get Phyloseq to read the OTU tables and taxonomy file so we can generated some pretty graphs. Getting the data into Phyloseq has been a pain so I wrote a Python script that basically converts the taxonomy file to a format that Phyloseq can read. May be I'm doing something wrong but I never got Phyloseq to read the taxonomy file (I always get some error) but after running my Python script that "fixes" the taxonomy file, Phyloseq is able to read the taxonomy file. 

The database we use for our 18S amplicon seqs is SILVA so perhaps the formatting of the taxonomic strings causes Phyloseq to choke. 

Anyway, I'm currently trying to get Phyloseq to just read the mapping file instead of forcing the metadata to show up in a text version of the biom table. Our lab is hoping to have a Phyloseq tutorial that covers everything from importing data to visualizing diversity stats. 

If you have any suggestions, I'm all ears! Thanks again for your help!

Taruna

[Jai Ram Rideout]

Hi Taruna,

You might check with the PhyloSeq developers (it's a separate project from QIIME). I know they have some great tutorials that might be useful.

Best,

Jai

[Colin Brislawn]

Hello Taruna,

Oh, I use Phyloseq too. Once I get my taxonomy strings into the JSON .biom table, I import it into phyloseq like this:

no_meta <- import_biom(file.path('otus_vsearch/otu_table_w_tax.biom'),
                       file.path('otus_vsearch/rep_set.tre'))
no_meta

meta <- import_qiime_sample_data(file.path('meta/meta.txt'))

full <- merge_phyloseq(meta, no_meta)
full

This import_biom() function works with greengenes taxonomy. As long as you are using a qiime compatible databases, this should work too. Are you using the version of silva over here: https://www.arb-silva.de/download/archive/qiime/ 

Also, I'm still importing the qiime metadata separately, then merging it. Let me know how well this methods works for you. 

Colin

[Taruna]

Hey Colin! Yes, I'm using the 128 release...So I tried your method and everything looks good up until I actually try to create a stacked plot with respect to different taxonomic levels like class, order, etc. Can you please share some of the downstream commands you use in R for visualizing your data? Thanks, Colin!

[Taruna]

Well...I'll be damned! I have no idea what I did differently but I was able to make the stacked bar plot based on taxonomy. It looks a weird... there are a bunch of black stacks because I have a filtered biom table that has no OTUs for all samples but I'm sure filtering the OTU table should be that difficult...Anyway! Still would love to see your analysis commands please :) Thank you!

[Colin Brislawn]

Hello Taruna,

I would be happy to share! First, you have probably discovered the phyloseq tutorials, which show off (most) of what phyloseq can do. 

https://joey711.github.io/phyloseq/ 

Next, I really like this DIY tutorial, in which Joey, the creator of phyloseq, fully replicates another paper:

https://joey711.github.io/phyloseq-demo/Restroom-Biogeography.html

Here is one of my papers. This code is not very well written, so let me know if you have questions. 

https://github.com/pnnl/bernstein-2016-productivity-and-diversity/tree/master/analysis/phyloseq 

And finally, I've attached an .rmd file of a current project. This is how I currently build papers, because it makes replication easy. 

This may be more than you asked for, but I hope it's helpful.

Let me know if you have more questions,

Colin

 

[Taruna]

Awesome! I have looked over the first Phyloseq tutorial but haven't come across the second one you sent. I'll definitely check that one out! 

Thank you for sharing your code...I'll post any questions if and when I have them on Github :-) Thanks so much, Colin! 

Taruna 

[Colin Brislawn]

Glad to help!

I've had a great experience using phyloseq as a 'gateway drug' to the world of R software. I hope you have a great experience too.

Colin

[Taruna]

Hey Colin! Sorry to bug with this question but I have googled it to death and any solution is not working for me.

Since you use Phyloseq, can you please tell me know you normalize your raw otu table in phyloseq? Thank you so much!

[Colin Brislawn]

Sure!

Phyloseq supports single rarefaction / rarefying / subsampling to an even depth (even though the lead dev does not recommend this). 

phyloseq.object.rare <- rarefy_even_depth(phyloseq.object.raw)

Phyloseq also supports scaling to 1 / relative abundance (ra). 

phyloseq.object.ra <- transform_sample_counts(phyloseq.object.raw, function(x) x / sum(x))

The transform sample counts() function is super flexible. You could easily make it scale to 100 if you wanted.

phyloseq.object.ra <- transform_sample_counts(phyloseq.object.raw, function(x) x*100 / sum(x))

Colin

PS. The qiime forums are for qiime things, but I ventured this suggestion and wanted to follow up.

[Taruna]

Thanks! Yes I know...Sorry! Desperate times were calling for desperate measures :-( Okay so I transformed the OTU table correctly. So glad to have your confirmation!

[Colin Brislawn]

Great. I'm glad you got this working.

Colin