Denoising and chimera removal in ion torrent data

[selva]

Hi,
   i'm sequencing 16s rDNA Amplicones using Ion torrent pgm. From output i got the sequences in fastq format, then to use the 454-tutorial i convert it into fasta format-.fna. now for denoising i need sff.txt file and for chimera checking after pyNAST alignment, i need a reference_set_aligned.fasta, to run the following command,

identify_chimeric_seqs.py -m ChimeraSlayer -i rep_set_aligned.fasta -a reference_set_aligned.fasta -o chimeric_seqs.txt
or,
is there any other way to denoise and chimera removal of iontorrent produced data? please help in this issue. thankyou.

[Tony Walters]

I don't think you'll be able to do formal denoising with IonTorrent data (but someone who has tried this successfully feel free to contradict me). Although there are parallels in the chemistry between IonTorrent and 454, the error models have been specific for 454 data.

You can do chimera checking though. The reference aligned fasta (-a) for identify_chimeric_seqs.py can be downloaded here: http://greengenes.lbl.gov/Download/Sequence_Data/Fasta_data_files/core_set_aligned.fasta.imputed (you may already have this file somewhere on your system)

After filtering chimeras, you might do a quick-and-dirty approach of filtering low abundance OTUs (e.g. singletons, or maybe OTUs with less than 3 or 4 sequences per OTU, depending upon your sequencing depth) as a way of removing noise, using the filter_otus_from_otu_table.py script. You may use an arbitrary threshold for the percent abundance, so --min_count_fraction 0.00005 to filter 0.005 % and lower (see http://qiime.org/scripts/filter_otus_from_otu_table.html)

-Tony

--
 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[selva]

hi tony
 thank you verymuch. I will try that. I also tried to combine all my demultiplexed fastq file by converting them into fasta files. i used the following script,

add_qiime_labels.py -i ~/bioaerosol_test3 -m bioaerosol_test3/fasting_map_combine.txt -c InputFileName -n 10000000 - combined_fasta, during this i encountered the following error,

Usage: add_qiime_labels.py [options] {-m/--mapping_fp MAPPING_FP -i/--fasta_dir FASTA_DIR -c/--filename_column FILENAME_COLUMN}

[] indicates optional input (order unimportant)
{} indicates required input (order unimportant)

A metadata mapping file with SampleIDs
and fasta file names (just the file name itself, not the full or relative
filepath) is used to generate a combined fasta file with valid
QIIME labels based upon the SampleIDs specified in the mapping file.

See: http://qiime.org/documentation/file_formats.html#metadata-mapping-files
for details about the metadata file format.

Example mapping file:
#SampleID    BarcodeSequence    LinkerPrimerSequence    InputFileName    Description
Sample.1    AAAACCCCGGGG    CTACATAATCGGRATT    seqs1.fna    sample.1
Sample.2    TTTTGGGGAAAA    CTACATAATCGGRATT    seqs2.fna    sample.2

This script is to handle situations where fasta data comes already
demultiplexed into a one fasta file per sample basis.  Only alters
the fasta label to add a QIIME compatible label at the beginning.

Example:
With the metadata mapping file above, and an specified directory containing the
files seqs1.fna and seqs2.fna, the first line from the seqs1.fna file might
look like this:
>FLP3FBN01ELBSX length=250 xy=1766_0111 region=1 run=R_2008_12_09_13_51_01_
AACAGATTAGACCAGATTAAGCCGAGATTTACCCGA

and in the output combined fasta file would be written like this
>Sample.1_0 FLP3FBN01ELBSX length=250 xy=1766_0111 region=1 run=R_2008_12_09_13_51_01_
AACAGATTAGACCAGATTAAGCCGAGATTTACCCGA

No changes are made to the sequences.


Example usage:
Print help message and exit
 add_qiime_labels.py -h

Example: Specify fasta_dir as the input directory of fasta files, use the metadata mapping file example_mapping.txt, with the metadata fasta file name column specified as InputFileName, start enumerating with 1000000, and output the data to the directory combined_fasta
 add_qiime_labels.py -i fasta_dir -m example_mapping.txt -c InputFileName -n 1000000 -o combined_fasta

add_qiime_labels.py: error: option -m: file does not exist: 'fasting_map_combine.txt'

i also followed your previous suggestions with another qiime user, to locate the file, just by dragging it over the terminal. I don't know whether i'm doing it right or not, when drag my folder over terminal it just open new terminal window. can you please suggest what could be the correction and error i have done? thank you.

[Tony Walters]

Selva,

I'm not really sure what's going on with your terminal opening new terminals. You might just have to manually type out the full filepaths to your file (or make sure that you're in the right directory and use the exact file name or relative filepath).

You might take a look at this as a guide to learning how to navigate: http://cli.learncodethehardway.org/book/

You can also use the 

find `pwd` -name X

where X is the filename you wish to find, and this will show the absolute filepath to that file.

--

[selva]

hi Tony
after the command pwd, i found my filepath, but now its showing a different error, my previous error was mapping file does not exist, now it is add_qiime_labels.py: error: Positional argument detected: -
 Be sure all parameters are identified by their option name.
 (e.g.: include the '-i' in '-i INPUT_DIR')
selvasankar@selvasankar-VPCEB34EN[selvasankar]  qiime > pwd           [ 8:37PM]
/home/selvasankar
selvasankar@selvasankar-VPCEB34EN[selvasankar]  qiime > add_qiime_labels.py -i /home/selvasankar/bioaerosol_test3 -m bioaerosol_test3/fasting_map_combine.txt -c InputFileName -n 10000000 - combined_fasta

add_qiime_labels.py: error: Positional argument detected: -
 Be sure all parameters are identified by their option name.
 (e.g.: include the '-i' in '-i INPUT_DIR')
selvasankar@selvasankar-VPCEB34EN[selvasankar]  qiime >
please help in this issue, thanks

On Thursday, February 20, 2014 5:57:35 PM UTC-6, selva wrote:

[Tony Walters]

It looks like you intended to have a -o combined_fasta at the end of your command, but you left off the -o, so it's not picking up that parameter correctly. If you see a "positional argument" error, check the parameters carefully for the correct names, and be careful about spacing (e.g. - o combined_fasta would also error).

--

[Gregg Iceton]

Agreed - there is no peer reviewed method for Ion Torrent denoising at present.  Chris Quince is looking at building an error model for Ion Torrent, but the chemistry and signal processing change so often it is challenging.  However, there is a beta version of Acacia which claims to be able to do so, and FlowClus can also do so with the addition of a small script to adjust the sff produced by Torrent Suite.  To get the sff, you need to go to Torrent Suite and create it using the FileExporter plugin.

[selva]

thanks a lot buddies.

[selva]

Hi Tony,
for chimera removal i used the following commands.

identify_chimeric_seqs.py -m ChimeraSlayer -i otus/pynast_aligned_seqs/combined_seqs_rep_set_aligned.fasta -a otus/core_set_aligned.fasta.imputed -o chimeric_seqs.txt

filter_fasta.py -f otus/pynast_aligned_seqs/combined_seqs_rep_set_aligned.fasta -o non_chimeric_rep_set_aligned.fasta -s chimeric_seqs.txt -n
then to exclude chimeric sequences i used,

make_otu_table.py -i otus/otu_table_summary.txt -o otu_table.biom_sin_chimera -e chimeric_seqs.txt -t otus/uclust_assigned_taxonomy/combined_seqs_rep_set_tax_assignments.txt

but i encountered following problems,
Usage: make_otu_table.py [options] {-i/--otu_map_fp OTU_MAP_FP -o/--output_biom_fp OUTPUT_BIOM_FP}

[] indicates optional input (order unimportant)
{} indicates required input (order unimportant)

The script make_otu_table.py tabulates the number of times an OTU is found in each sample, and adds the taxonomic predictions for each OTU in the last column if a taxonomy file is supplied.

Example usage: 
Print help message and exit

 make_otu_table.py -h

Make OTU table: Make an OTU table from an OTU map (i.e., result from pick_otus.py) and a taxonomy assignment file (i.e., result from assign_taxonomy.py). Write the output file to otu_table.biom.
 make_otu_table.py -i otu_map.txt -t tax_assignments.txt -o otu_table.biom

Make OTU table, excluding OTU ids listed in a fasta file: Make an OTU table, excluding the sequences listed in pynast_failures.fna. Note that the file pass as -e must end with either '.fasta' or '.fna'.
 make_otu_table.py -i otu_map.txt -o otu_table_no_pynast_failures.biom -e pynast_failures.fna

Make OTU table, excluding a list of OTU ids: Make an OTU table, excluding the sequences listed in chimeric_seqs.txt
 make_otu_table.py -i otu_map.txt -o otu_table_non_chimeric.biom -e chimeric_seqs.txt

make_otu_table.py: error: option -i: file does not exist: 'otus/otu_map.txt'
selvasankar@selvasankar-VPCEB34EN[obesity_2]  qiime > make_otu_table.py -i otus/ otu_table_summary.txt -o otu_table.biom_sin_chimera -e chimeric_seqs.txt -t otus/wf_taxa_summary/otu_table_L6.txt
Usage: make_otu_table.py [options] {-i/--otu_map_fp OTU_MAP_FP -o/--output_biom_fp OUTPUT_BIOM_FP}

[] indicates optional input (order unimportant)
{} indicates required input (order unimportant)

The script make_otu_table.py tabulates the number of times an OTU is found in each sample, and adds the taxonomic predictions for each OTU in the last column if a taxonomy file is supplied.

Example usage: 
Print help message and exit

 make_otu_table.py -h

Make OTU table: Make an OTU table from an OTU map (i.e., result from pick_otus.py) and a taxonomy assignment file (i.e., result from assign_taxonomy.py). Write the output file to otu_table.biom.
 make_otu_table.py -i otu_map.txt -t tax_assignments.txt -o otu_table.biom

Make OTU table, excluding OTU ids listed in a fasta file: Make an OTU table, excluding the sequences listed in pynast_failures.fna. Note that the file pass as -e must end with either '.fasta' or '.fna'.
 make_otu_table.py -i otu_map.txt -o otu_table_no_pynast_failures.biom -e pynast_failures.fna

Make OTU table, excluding a list of OTU ids: Make an OTU table, excluding the sequences listed in chimeric_seqs.txt
 make_otu_table.py -i otu_map.txt -o otu_table_non_chimeric.biom -e chimeric_seqs.txt

make_otu_table.py: error: option -i: not a regular file (can't be a directory!): 'otus/'
selvasankar@selvasankar-VPCEB34EN[obesity_2]  qiime > make_otu_table.py -i otus/otu_table_summary.txt -o otu_table.biom_sin_chimera -e chimeric_seqs.txt -t otus/wf_taxa_summary/otu_table_L6.txt
Traceback (most recent call last):
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/bin/make_otu_table.py", line 82, in <module>
    main()
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/bin/make_otu_table.py", line 77, in main
    ids_to_exclude)
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/lib/qiime/make_otu_table.py", line 70, in make_otu_table
    " Original error message: %s" % (str(e)))
ValueError: Couldn't create OTU table. Is your OTU map empty? Original error message: max() arg is an empty sequence
selvasankar@selvasankar-VPCEB34EN[obesity_2]  qiime >    

can you suggest  in this issue? where i made error? what kindo of corrections i have to do.   thank you.                                                                        
 

[Tony Walters]

Selva, the input (-i) parameter to make_otu_table.py is the OTU mapping file. This file is in the output directory of the OTU picking, e.g. uclust_picked_otus/, and usually ends with the name _seqs_otus.txt. It looks like you passed the summary file for the OTU table.

-Tony 

[selva]

hi Tony,
 thanks for that suggestion, after that i could create the otu_table without trouble. But now i cameup with another issue. Once i removed chimeric_sequence.txt from my otu table, when i tried to create its wf_taxa_summary, in that when i see the taxa summary chart, still i could see many of chimeras like archaea and some other bacterias those who not even present in any of the samples, it still appears in the barchart and area chart of tax summary. How can i eliminate those list from the taxa summary set. i hereby attached its screenshot for an example. please guide me in this issue. thank you.   

/home/selvasankar/Pictures/Screenshot from 2014-02-24 15:09:30.png

[Tony Walters]

Selva,

It looks like there are zero counts for some of those taxa, like the Archaea. Can you try running filter_otus_from_otu_table.py (http://qiime.org/scripts/filter_otus_from_otu_table.html) with the setting -n 1 parameter to remove zero count OTUs, and do taxa summaries on the resulting filtered OTU table?

[selva]

Hi TONY
after your suggestion i followed the following scirpt,

filter_otus_from_otu_table.py -i otu_table.biom -o otu_table_non_chimeric.biom -e chimeric_seqs.txt -n 1 
to generate new otu_table free from those bacteria with 0 count, but it looks the same, here i attached its example. please help me to solve it out. thank you. 


	
	
	
	

[Tony Walters]

Selva, I think that either you 1. did not filter the original OTU table correctly, or 2. are using an incorrect (unfiltered) OTU table as input for your taxonomy plots.

Post every exact command you used from making the OTU table to generating the taxonomy plots.

[selva]

hi Tony
these are those commands i used, Please check it,
to identify chimeric sequences:

 identify_chimeric_seqs.py -m ChimeraSlayer -i otus/pynast_aligned_seqs/combined_seqs_rep_set_aligned.fasta -a otus/core_set_aligned.fasta.imputed -o chimeric_seqs.txt

to filter:

filter_fasta.py -f otus/pynast_aligned_seqs/combined_seqs_rep_set_aligned.fasta -o non_chimeric_rep_set_aligned.fasta -s chimeric_seqs.txt -n

to make otu table:

make_otu_table.py -i otus/uclust_picked_otus/combined_seqs_otus.txt -o otu_table.biom -e chimeric_seqs.txt -t otus/uclust_assigned_taxonomy/combined_seqs_rep_set_tax_assignments.txt

to filter otus from otu table:

filter_otus_from_otu_table.py -i otu_table.biom -o otu_table_non_chimeric.biom -e chimeric_seqs.txt -n 1

these are those commands i followed, please check it out. thankyou.

[Tony Walters]

What command did you use for generating the taxonomy plots?

And can you

1. convert the biom table (the otu_table.biom file from your above make_otu_table.py command) to a tab separated format, following the 6th example on this page to convert the taxonomy: http://biom-format.org/documentation/biom_conversion.html

2. Open your chimeric_seqs.txt file, and search for some of the IDs in the tab-separated OTU table created in step one. If they are present, they should be in the left-hand column (the OTU IDs).

[selva]

hi Tony
for taxanomy plots i used the following command,

summarize_taxa_through_plots.py -i otu_table_non_chimeric.biom -o otus/wf_taxa_summarynew -m fasting_map_combine.txt.

to accordance with your last message, i converted my biomtable to a tab separated format, i compared some of those ids of biomtable tab separated format with chimeric sequence.txt, those ids of chimeric sequences.txt i checked were not in the tab separated biom table. i 've attached those files for your revision. what could be the next step? do you think i did somewhere wrong? i tried to attach my tab separated biom format, but its a bit large size. so i could n't.

[Tony Walters]

This indicates that the chimeric OTU ids have been filtered out.

Do the summarize_taxa_through_plots.py with the otu_table.biom file. Any taxa that show up here are from something *other* than the chimeric sequences. They could be rare taxa-you can use filter_otus_from_otu_table.py to create a filtered table with rare taxa removed.

[Gregg Iceton]

Just to add my two penneth, in my experience the HTML counts always give counts of 0 or so with Ion Torrent data.  Maybe its better just to look at the relative abundances instead?

[selva]

Hi Tony,
 i'm sure that we have filtered out the chimeric sequences, but still i could see those rare taxas which is 0 count. is there anyother way to eliminate those from the taxa summary  plot? because the number of bacterias with 0 count, when i see the bar plot , it looks odd.

[selva]

Hi Gregg
 then what could be way to remove those? relative abundace- did you mean the heatmap? please explain, thank you.

[Tony Walters]

Selva, it's actually summarized to a relative abundance by default. You will need to pass a parameters file (-p) when you call summarize_taxa_through_plots.py that sets the abundance to absolute.

http://qiime.org/scripts/summarize_taxa.html

The parameters file should look like this:

summarize_taxa:absolute_abundance True

[selva]

Hi Tony ,
 thankyou verymuch for your suggestion. But how could we remove those taxas with 0 count in the summary bar plots..?here in this i have created a  table summarized taxa with absolute abundance. but can i show it in a graphical way? 

[Tony Walters]

Did you create the plots with the absolute abundance table? It should show non-zero values in the plots then (instead of rounding very small relative abundance values to zero).

If you want to filter particular taxa, you can use this script: http://qiime.org/scripts/filter_taxa_from_otu_table.html

[selva]

Hi Tony
i followed the following scrit to create the absolute abundance table, but i dont know how to create the plots with it. could you please explain me this with some example?, i have also attached that absolute abundance table txt. thank you

summarize_taxa.py -i otus/otu_table.biom -o tax_mapping/ -m fasting_map_combine.txt -a -L 6

[Tony Walters]

The parameters file I pointed earlier would be one way to change the workflow so it first summarizes taxa with absolute, rather than relative values, and then plots them. The workflow calls summarize_taxa.py and then plot_taxa_summary.py.

You can directly plot the output files (comma separated) of the summarize_taxa.py call you did above by using plot_taxa_summary.py (http://qiime.org/scripts/plot_taxa_summary.html).

[selva]

hi Tony,
 when i tried to workout this command, i encountered the following error, can you please go through it to find out what mistake i did? thank you.

selvasankar@selvasankar-VPCEB34EN[selvasankar]  qiime > cd ~/bioaerosol_1/tax_mapping                                                 [ 5:16PM]
selvasankar@selvasankar-VPCEB34EN[tax_mapping]  qiime > plot_taxa_summary.py -i fasting_map_combine_L6.txt -l Phylum,Class,Genus -c pie,bar,area -o phylum_class_genus_charts/

Traceback (most recent call last):

  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/bin/plot_taxa_summary.py", line 299, in <module>
    main()
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/bin/plot_taxa_summary.py", line 185, in main
    taxonomy_color_prefs_and_map_data_from_options(opts)
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/lib/qiime/colors.py", line 561, in taxonomy_color_prefs_and_map_data_from_options
    parse_taxa_summary_table(counts_f)
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/lib/qiime/parse.py", line 572, in parse_taxa_summary_table
    result = parse_classic_otu_table(lines,count_map_f=float)
  File "/home/selvasankar/qiime_software/qiime-1.8.0-release/lib/qiime/parse.py", line 557, in parse_classic_otu_table
    valid_fields = asarray(fields[1:], dtype=float)
  File "/home/selvasankar/qiime_software/numpy-1.7.1-release/lib/python2.7/site-packages/numpy/core/numeric.py", line 320, in asarray
    return array(a, dtype, copy=False, order=order)
ValueError: could not convert string to float: GCAACACCATCC
selvasankar@selvasankar-VPCEB34EN[tax_mapping]  qiime >

[Tony Walters]

Selva,

In the plot_taxa_summary page I linked before, it lists what the input files should be.

See http://qiime.org/scripts/plot_taxa_summary.html

The input should *not* be your metadata mapping file. It's the summarized OTU tables that are output from summarize_taxa. Don't pass -l values unless you're going to specify the matching summarized taxa tables (e.g. level 2 for phylum, level 3 for class, level 6 for genus).

[selva]

hi Tony,
 i did not give my metadata mapping file as input, i just gave my summarize_taxa output file only. i here attached that file for your reference. thank you

[Tony Walters]

As the file starts with this line:

"#SampleID BarcodeSequence LinkerPrimerSequence InputFileName Description"

which is from the metadata mapping file. You should have other files in your tax_mapping/ directory that are summarized OTU tables (you only specified level 6 in your command above).

I think this will be much, much easier if you use a parameters file and rerun summarize_taxa_through_plots with it. I've attached one for you.

summarize_taxa_through_plots.py -i otu_table_non_chimeric.biom -o taxa_summary_absolute_abundance/  -p qiime_parameters_selva.txt

You will have to make sure those filepaths are correct-I can't tell you what those are.

[selva]

Hi Tony,
 i have followed this command without using my metadata file, now it allows me to create plot_taxa_summary. THANKYOU VERYMUCH FOR YOUR PATIENCE WITH ME.

summarize_taxa.py -i otu_table.biom -o ./tax
plot_taxa_summary.py -i tax/otu_table_L6.txt  -c pie,bar,area -o phylum_class_genus_charts
Thankyou.

[selva]

hi Tony,
thanks for the parameter file, now it has created directly taxa_summary_plot with absolute abundance. thanks again 

[Nastassia Patin]

Hi Gregg,

Sorry to jump in on this months-old post, but I"m interested in knowing more about FlowClus. Everything I've found seems to say it's designed for 454 error rates - do you have any documentation on how it works for Ion Torrent data? 

Thanks,

Nastassia

[Gregg Iceton]

Hi.  No documentation I'm afraid, and the last time I discussed it with the author he had to write a little script to adjust the Ion Torrent data to suit the program.  The program is at https://github.com/jsh58/FlowClus and the author's contact details are also there.  I'm afraid I haven't used the program as yet.