All samples filtered out error on 454 reads

[Ugin Levin]

Hello all!

So, I have reads from 2013 atherosclerosis study (5 first heads attached), performed using 454 Roche sequencing technology. I'm running qiime 1.7 code, which have been wroten for Illumina data, and get this error :

Traceback (most recent call last):
  File "/home/ubuntu/qiime_software/qiime-1.7.0-release/bin/single_rarefaction.py", line 79, in <module>
    main()
  File "/home/ubuntu/qiime_software/qiime-1.7.0-release/bin/single_rarefaction.py", line 76, in main
    empty_otus_removed=(not opts.keep_empty_otus),subsample_f=subsample_f)
  File "/home/ubuntu/qiime_software/qiime-1.7.0-release/lib/qiime/rarefaction.py", line 63, in rarefy_to_file
    subsample_f=subsample_f)
  File "/home/ubuntu/qiime_software/qiime-1.7.0-release/lib/qiime/rarefaction.py", line 177, in get_rare_data
    otu_table = filter_samples_from_otu_table(otu_table, otu_table.SampleIds, seqs_per_sample, inf)
  File "/home/ubuntu/qiime_software/qiime-1.7.0-release/lib/qiime/filter.py", line 504, in filter_samples_from_otu_table
    return otu_table.filterSamples(filter_f)
  File "/home/ubuntu/qiime_software/biom-format-1.1.2-release/lib/python2.7/site-packages/biom/table.py", line 534, in filterSamples
    raise TableException, "All samples filtered out!"
biom.exception.TableException: All samples filtered out!
Converting BIOM...
Traceback (most recent call last):
  File "/home/ubuntu/qiime_software/biom-format-1.1.2-release/bin/convert_biom.py", line 194, in <module>
    main()
  File "/home/ubuntu/qiime_software/biom-format-1.1.2-release/bin/convert_biom.py", line 116, in main
    input_f = open(opts.input_fp,'U')

Also in otu picking I got something like this :
Num OTUs:123
Num new OTUs:0
Num failures:3191

for every read file.

Then in single_rarefaction.py I got error.

First guess was the problem is in quality scores. So I varied parameters -q and -r of split_libraries_fastq.py scrypt. Nothing significantly has changed. Tried change phred_offset, also no result.

Next : I tried to figure out if I should give barcodes or/and primers info (now empty mapping file is being passed). Tried to ran extract_barcodes.py, it didn't work for me.

Any insights or ideas are welcome, please let me know which additional information about my case shoud I share.

Thank you,

Ugin Levin

[Stefan Janssen]

Hi Urgin,
first: can you updated to qiime 1.9.1: http://qiime.org/install/upgrade.html ?
How many samples are in your experiment? Are the reads already split into those samples or are they all in one huge file?
What is your goal in the end? What kind of graph / table / statistics do you want to compute for those reads?

[Ugin Levin]

Hi Stefan,
thank you for the answer.
There are about 100 reads, they all each are in separate file already.
Main goal is to get OTU-abundance tables using grean genes and HITdb bases ( two runs for one dataset )

[Stefan Janssen]

do you mean 100 reads or 100 samples?

[Ugin Levin]

Sorry, about 100 samples and each sample about 5k reads. I can give exact numbers, while server is on, if it is needed.

[Stefan Janssen]

Looks like you don't have to split_libraries at all, because the reads of your samples are already in separate files. I suggest to do the OTU picking now. From your post I guess you want to do closed reference against GreenGenes. I would use the sortmerna algorithm:

pick_otus.py -m sortmerna -i YOUR_FILE f

for each file separately and later merge all 100 results with merge_otu_tables.py into one biom table.
Then, you can do rarefaction and all other downstream analyses.

[Ugin Levin]

Thanks, I'll give it a try and then let you know.

[Ugin Levin]

I am now perfoming these commands with my data and wondering if I should use single_rarefaction.py?

Usually I do it between pick_closed_reference_otus.py and convert_biom.py/summarize_taxa.py like this :

single_rarefaction.py -i otu/otu_table.biom -o otu_rar/otu_table.biom -d 5000

[Stefan Janssen]

Not easy to say. First, you need to think about rarefaction at all. Why do you want to apply it? What is the purpose?
Typically, we want to avoid that low abundant taxa govern statistics. However, rarefaction throws away many sequence reads for samples with high read numbers.
Try http://qiime.org/scripts/make_rarefaction_plots.html and find a suitable n by staring at the plots and spotting a point where the curves saturate and you don't loose too many samples.
But it is really up to your scientific judgement if and how you want to use rarefaction. We have ever ongoing debates about this in our lab.