Core diversity error message

[ciara keating]

Hi everyone,

Complete QIIME beginner here. I have been using Mothur and now want to use MacQIIME for my core diversity etc analysis. However, I am encountering error messages and the only output I get is the log file, rarefied number biom and biom table summary. I've the log below and have attached the mapping file (it appears that the other files are too large to attach). I'd appreciate any help or advice you could give. I've been researching myself to figure it out but I've been unsuccessful! I'm wondering is it maybe to do with my negative control being in the sample which would be one sample on its own different to all the others?

The command I ran was: (9986 being my lowest sample count/selected non phylogenetic diversity as I had issues making the tree [I had the same tree issue in Mothur as my distance file was 249gb])

core_diversity_analyses.py -i /Users/ciarakeating2/codindex.01.0.01.biom -m /Users/ciarakeating2/mapping_fileCK.txt -e 9986 -o /Users/ciarakeating2/core_diversity01/ -c feed,tank,week,health --nonphylogenetic_diversity --suppress_group_significance

The log file is as follows:

Logging started at 16:16:18 on 09 Feb 2016

QIIME version: 1.9.1

qiime_config values:

pick_otus_reference_seqs_fp /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

sc_queue all.q

pynast_template_alignment_fp /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

cluster_jobs_fp start_parallel_jobs.py

assign_taxonomy_reference_seqs_fp /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

torque_queue friendlyq

jobs_to_start 1

denoiser_min_per_core 50

assign_taxonomy_id_to_taxonomy_fp /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

temp_dir /tmp/

blastall_fp blastall

seconds_to_sleep 60

parameter file values:

beta_diversity:metrics bray_curtis

alpha_diversity:metrics observed_otus,chao1

parallel:jobs_to_start 1

Input file md5 sums:

/Users/ciarakeating2/codindex.01.0.01.biom: b03c55151235c24de4e2611394762ce9

/Users/ciarakeating2/mapping_fileCK.txt: b9330b9c0d2f2c31b249770ffd866aec

Executing commands.

# Generate BIOM table summary command 

biom summarize-table -i /Users/ciarakeating2/codindex.01.0.01.biom -o /Users/ciarakeating2/core_diversity01//biom_table_summary.txt 

Stdout:

Stderr:

# Filter low sequence count samples from table (minimum sequence count: 9986) command 

filter_samples_from_otu_table.py -i /Users/ciarakeating2/codindex.01.0.01.biom -o /Users/ciarakeating2/core_diversity01//table_mc9986.biom -n 9986

*** ERROR RAISED DURING STEP: Filter low sequence count samples from table (minimum sequence count: 9986)

Command run was:

 filter_samples_from_otu_table.py -i /Users/ciarakeating2/codindex.01.0.01.biom -o /Users/ciarakeating2/core_diversity01//table_mc9986.biom -n 9986

Command returned exit status: 1

Stdout:

Stderr

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/filter_samples_from_otu_table.py", line 162, in <module>

    main()

  File "/macqiime/anaconda/bin/filter_samples_from_otu_table.py", line 138, in main

    write_biom_table(filtered_otu_table, output_fp)

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/util.py", line 577, in write_biom_table

    biom_table.to_hdf5(biom_file, generated_by, compress)

  File "/macqiime/anaconda/lib/python2.7/site-packages/biom/table.py", line 3535, in to_hdf5

    self.group_metadata(axis='observation'), 'csr', compression)

  File "/macqiime/anaconda/lib/python2.7/site-packages/biom/table.py", line 3507, in axis_dump

    formatter[category](grp, category, md, compression)

  File "/macqiime/anaconda/lib/python2.7/site-packages/biom/table.py", line 243, in general_formatter

    compression=compression)

  File "/macqiime/anaconda/lib/python2.7/site-packages/h5py/_hl/group.py", line 99, in create_dataset

    dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)

  File "/macqiime/anaconda/lib/python2.7/site-packages/h5py/_hl/dataset.py", line 60, in make_new_dset

    raise ValueError("Shape tuple is incompatible with data")

ValueError: Shape tuple is incompatible with data

[Husen Zhang]

Ciara -
Did you generate your biom file in mothur?  If so, this thread may help you out.

Husen Zhang

[ciara keating]

Hi Dr. Zhang,

That's perfect that conversion worked for the command. Could you advise me on whether I should do the core diversity analysis on my other biom files. Ie at 02, 03 and 04 levels from Mothur? Also if there is an issue using the Mothur generated fasta file to use the make.phylogeny command? I tried to do this command in Mothur (Dist.seqs and Clearcut) but it failed at clearcut as the distance file was 249 gb. I'm just wondering is there a way around this in QIIME? 

Thanks for your help,

Ciara

[Colin Brislawn]

Hello Ciara,

I'm not sure how mothur attempts tree building, but I can describe how qiime does it. Qiime aligns OTU centroids to greengenes database using a nast alignment, filters elements of that alignment, then constructs a tree using fasttree. While only using OTU centroids, you will have a much smaller alignment and tree file, so this may scale better for your project. It sounds like your 'Mothur generated fasta file' contains all your reads, so you could try the qiime method on the fasta file only containing OTU centroids. 

Colin Brislawn