Problem from PICRUSt to STAMP

[Eustacia Lee]

Hello,

I would like to analyse metagenome data from PICRUSt using STAMP but failed to load the files into STAMP. 

I have few questions as below:

1) Is it possible to use output (predict metagenome and categorize by function) from the online version of PICRUSt if I want to analyse using STAMP?

2) If yes, then how can I convert from picrustc to spf?  Which type of output format should I choose? biom or Legacy Qiime format (tab-delimited)?

   I tried to change picrustc to txt and then use the following command to convert to spf, but still unable to import into STAMP.

  sed '1d' predicted_metagenomes.txt | rev | cut -f 2- | rev > predicted_metagenome.spf

3) If I discard the header and last column manually, how can I convert to spf?

Appreciate if anyone could advice me.

Thanks.

Regards,

Eustacia

[Morgan Langille]

Hi Eustacia, 

STAMP actually has built in support now for BIOM files, so you can use STAMP to do the conversion for you. I have updated the PICRUSt documentation to reflect this change. See here:

http://picrust.github.io/picrust/tutorials/stamp_tutorial.html#stamp-tutorial

Thanks,

Morgan Langille

Assistant Professor & Canada Research Chair Candidate

Department of Pharmacology

Dalhousie University
http://morganlangille.com

--
You received this message because you are subscribed to the Google Groups "picrust-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to picrust-user...@googlegroups.com.
To post to this group, send email to picrus...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eustacia Lee]

Hi Dr Morgan,

Thank you for your prompt reply. I will try again.

Regards,

Eustacia Lee Soo Ching

Department of Parasitology

Faculty of Medicine

University of Malaya

Email: eusta...@gmail.com

--
You received this message because you are subscribed to a topic in the Google Groups "picrust-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/picrust-users/fVxZpYEiSBM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to picrust-user...@googlegroups.com.

[H Zhang]

Eustacia,
Before you use the sed command, you want to make sure that the file
predicted_metagenomes.txt
is in classic otu table format, not the biom format. That did it for me.
Husen Zhang

[Richa Bharti]

Hello,
I have some similar issues. I used Picrust galaxy. For STAMP I am using Categorize_by_function.biom file and according to the link you mentioned. I Create STAMP profile from...->BIOM profile and Metadata field to KEGG_Pathways. here I am getting an error
#### An error has occured:AttributeError: 'Table' object has no attribute 'observation_metadata'It occured at line 89 of file createProfileBiomDlg.py.####
I am not able to understand what could be reason for this error.
Thank you

[Morgan Langille]

I believe this is an error due to having different BIOM versions. If you are running STAMP under Windows, then it packages the appropriate BIOM version. If you are running it on Mac or Linux you will need the BIOM version >2.0 (NOTE: PICRUSt will not work yet with this BIOM version...so this is a bit of a pain). 

--------------------------------------------

Dr. Morgan Langille

Assistant Professor & Canada Research Chair Candidate

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[RB]

So cool I was working with linux and it was really giving me trouble. Big thanks!!!
but i have another question I am giving a metadata file to stamp
Samples    BF    Gender    Mutant    Day    breeding cage Nr    Cage Nr
01_d00    F    Female    WT    d00    24    11
01_d21    F    Female    WT    d21    24    11
01_d56    F    Female    WT    d56    24    11
02_d00    B    Female    WT    d00    25    14
02_d21    B    Female    WT    d21    25    14
 And there is some formatting issues i guess
So I would like to know few things
1) should there be same sample number and in same order in profile file and meta file
2) Only one column(factor) can be added and not multiple for comparison in metafile
If feel the file is ok but somehow its not giving me enough option to explore other factors
Thank you again

[Morgan Langille]

The first column must contain the sample ids that exactly match your sample ids in the picrust file. The sample ids do not need to be the same order. Also, make sure that you are using tabs to separate the columns and not spaces. 

For debugging purposes you can simplify the map file to only a single group and add on columns if previous versions load. I find the "cut" command useful for this:

e.g.

cut -f 1,2 map.tab > simple_map.tab

cut -f 1,3 map.tab > another_simple_map.tab

etc. 

--------------------------------------------

Dr. Morgan Langille

Assistant Professor & Canada Research Chair Candidate

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[Chenhao Li]

I was getting the following errors using STAMP on a biom file.

AttributeError: 'Table' object has no attribute 'sample_ids'
It occured at line 104 of file createProfileBiomDlg.py.

PICRUSt was run with biom1.3.1, and STAMP was installed with biom2.1. Would this be a problem? If so, how can I convert the biom from PICRUSt to the compatible version for STAMP?

[Morgan Langille]

The BIOM file can be read with either version. However, if you have STAMP installed on linux or a mac you will need to update your BIOM version to 2.1 to use STAMP correctly.

--------------------------------------------

Dr. Morgan Langille

Assistant Professor & Canada Research Chair Candidate

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[GDP]

Hi,
I have the same issue with STAMP, but I am not able to find out how to change the version of the BIOM file. How would you convert the biom version of the categorized file in order to use it in stamp?
Thank you very much
g

[daniel....@gmail.com]

Hello Dr Morgan,
I'm having a probelm with converting BIOM from PICRUSt into the STAMP file and I believe it is related to older version of the BIOM file. Is there any option to convert BIOM fiel to newer BIOM file before I use the STAPM converter?
Thank you
Daniel Laubitz

[Morgan Langille]

It really shouldn't matter what BIOM version the file is. Maybe you can send the file directly to my email and I can see if it is something obvious.

--------------------------------------------

Dr. Morgan Langille

Assistant Professor

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[daniel....@gmail.com]

That would be wonderful if you could help me. I am new tp Python, PICRUSt and STAMP so probably there is something obvious. I will send it using my gmail address.

Thank you

Daniel Laubitz

[Morgan Langille]

Just for other users of picrust reading this thread. I wrote my own biom_to_stamp.py script that handles PICRUSt output and QIIME output. The script is located in this github repostitory:

https://github.com/mlangill/microbiome_helper

And the command I run for PICRUSt conversion:

biom_to_stamp.py -m KEGG_Pathways metagenome_prediction_L3.biom > metagenome_prediction_L3.spf

--------------------------------------------

Dr. Morgan Langille

Assistant Professor

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[John Chase]

Hello, 

 

I was having a similar issue to the previous posts here, where when I tried to load a biom file into STAMP I would get an error of one sort or another. The script that Morgan provided converted the biom table into a STAMP file with no problems and I was able to load it into STAMP. 

Thank you for providing this script.

John

[Morgan Langille]

You are welcome and thanks for the feedback! Since it seems useful I will try and include this script in a future version of PICRUSt. 

--------------------------------------------

Dr. Morgan Langille

Assistant Professor

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[Daniela Vargas Robles]

Hello Morgan,
Thank you for your help. I´m trying to run.

biom_to_stamp.py -m KEGG_Pathways metagenome_prediction.biom > metagenome_prediction_L3_2.spf

In uninstalled biom-format 1.3.1 and installed biom-format 2.1.3.
I downloaded the "microbiome_helper-master" to get the biom_to_stamp.py command. But I get this error:
-bash: biom_to_stamp.py: command not found

What do you thing is happening? I attach the .biom file

[Mehrbod Estaki]

Hi Dr. Morgan,

I've used your biom_to_stamp.py script perfectly fine in the past but I'm running into a new issue which I think results from the new "normalize_table.py" in Qiime. 

I normalized_by_copy_number my OTU table through PICRUSt then stabilized the new biom table in Qiime 1.9 using normalize_table.py (CSS default). This is done with Biom 2.0 version.

Using  the older biom 1.3 version I then ran "predict_metagenome.py" on the newly normalized/stabilized otu table.

This script returned no errors and the new biom table looks to have worked perfectly fine.

When I tired to convert this new biom table into .spf for STAMP however I get the following error:

=====================================

biom_to_stamp.py -m KEGG_Pathways predic_metag_CSS_norm_cop_num_sing_rem.biom > predic_met_stamp.spf

Traceback (most recent call last):

  File "/home/bestaki/microbiome_helper/biom_to_stamp.py", line 138, in <module>

    main()

  File "/home/bestaki/microbiome_helper/biom_to_stamp.py", line 133, in main

    print "\t".join(row)

TypeError: sequence item 0: expected string, list found

==================================================

 

Similarly when I try to do the conversion on STAMP (running on Redhat) I get this error:

===============

AttributeError: 'Table' object has no attribute 'sample_ids'
It occured at line 104 of file createProfileBiomDlg.py.

==========================

Any suggestions would be greatly appreciated.

-Bod

[Morgan Langille]

There is an issue with the qiime normalize_table.py script that loses or misformats the metadata in the OTU table. I submitted an official issue to qiime and it looks like they are hoping to fix it for the next minor release 1.9.1:

https://github.com/biocore/qiime/issues/1929

 

--------------------------------------------

Dr. Morgan Langille

Assistant Professor

Department of Pharmacology, Dalhousie University

http://morganlangille.com

[Mehrbod Estaki]

Excellent. Thank you very much for the update. The little discussion in the link was also very useful.

[Heidi Smith]

Greetings,
There was an error message posted about a month ago in regards to biom_to_stamp.py and I am currently getting the same error and was hoping that someone could help me to figure out how to move forward.

The error is: -bash: biom_to_stamp.py: command not found

I got the biom_to_stamp.py command from the github site and put it into the scripts folder in picrust and get the same error as above.

Any help is much appreciated
Thank you,
Heidi

[Sami Pietilä]

Hi,

I got also this error: "AttributeError: 'Table' object has no attribute 'observation_metadata'

It occured at line 89 of file createProfileBiomDlg.py."

I am using Qiime 1.9.1 and biom-format 2.1.

The biom_to_stamp.py also fails:

python ../microbiome_helper/biom_to_stamp.py -m KEGG_Pathways predicted_metagenomes.L3.biom > metagenome_prediction_L3.spf

Traceback (most recent call last):

  File "../microbiome_helper/biom_to_stamp.py", line 147, in <module>

    main()

  File "../microbiome_helper/biom_to_stamp.py", line 141, in main

    row.extend(map(str,obs_vals))

AttributeError: 'str' object has no attribute 'extend'

I have used these instructions to make biom files:

https://picrust.github.io/picrust/tutorials/metagenome_prediction.html#metagenome-prediction-tutorial

https://picrust.github.io/picrust/scripts/categorize_by_function.html#categorize-by-function

How to import the BIOM files to STAMP?

Thanks

[Julia Cope]

You need to be sure your biom files are the JSON formatted before the
import feature of STAMP will work with them. Read the biom convert
--help output for full instructions for your version of biom. Know
that Qiime 9.1 uses the HDF5 not the JSON biom format. It's a one
command conversion.

You will still have issues with the longer 3 segmented KEGG picrust
output as there are unclassified higher categories and later columns
are named. You can correct this by running the script Dr. Parks has on
the STAMP website, or by renaming the Unclassified columns to
something a bit more useful. I use the second column to concatenate
onto the first. It goes from: Unclassified;metabolism;arginine
synthesis to Unclassified_metabolism;metabolism;arginine synthesis.

Also, be sure to remove the zero rows that picrust puts into your
file. You shouldn't waste statistical power on empty rows.

Happy hunting!

-Julia

> --
> You received this message because you are subscribed to the Google Groups
> "picrust-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to picrust-user...@googlegroups.com.
> To post to this group, send email to picrus...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--

Julia Cope, Ph.D.
Department of Pathology
Baylor College of Medicine

[Sami Pietilä]

Hi Julia,

I converted the BIOM file to JSON, but still got the same error message. I seems the predicted_metagenomes_json.L3.biom does not contain string "observation_metadata". 

Perhaps I have something wrong with following commands?

With these commands I create predicted_metagenomes_json.L3.biom from otu_table_mc2_w_tax_no_pynast_failures.biom (produced by open reference otu picking):

filter_otus_from_otu_table.py -i ../otus/otu_table_mc2_w_tax_no_pynast_failures.biom -o closed_otus.biom --negate_ids_to_exclude -e /home/apps/qiime-env/1.9.1/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

biom convert -i closed_otus.biom -o closed_otus_json.biom --to-json --table-type "OTU Table"

normalize_by_copy_number.py -i closed_otus_json.biom -o fa_normalized_otu_table.biom

predict_metagenomes.py -i fa_normalized_otu_table.biom -o metagenome_predictions.biom

categorize_by_function.py -f -i metagenome_predictions.biom -c "KEGG_Pathways" -l 3 -o predicted_metagenomes.L3.biom    

biom convert -i predicted_metagenomes.L3.biom -o predicted_metagenomes_json.L3.biom --to-json --table-type "OTU table"

Thanks

[Julia Cope]

Have you verified the OTU table is clean before putting it into picrust? Please convert it to a tab delimited text file and open it in your favorite processor and verify that the OTU are all numeric, none start with New. Or Clean.

I have often encountered issues with qiime's filter script not performing as expected.

[Sami Pietilä]

Hi Julia,

I converted closed_otus.biom and fa_normalized_otu_table.biom to tsv files and opened them with libreoffice. There seems to be OTU IDs, per sample counts (and normalized counts) and taxonomy. For me, the tables looked ok. The files did not contain "New" or "Clean".

Are there additional issues I can still check?

Thanks

[Maria G.]

Hi Heidi,

I am having the same problem and I would like to ask if you managed to sort it out.

Cheers,

Maria