QIIME 1.5.0 installation - FAIL: test_denoiser_supported_version

[Martin Asser Hansen]

I am installing QIIME 1.5.0 and is getting one final FAIL:

======================================================================

FAIL: test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/maasha/install/QIIME/bin/print_qiime_config.py", line 536, in test_denoiser_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

Ideas to resolve this are very welcome!

Here is the entire output of print_qiime_config.py:

maasha@orsted:~$ print_qiime_config.py -t

System information

==================

         Platform:      linux3

   Python version:      2.7.1 (r271:86832, May 14 2012, 13:13:00)  [GCC 4.4.5]

Python executable:      /home/maasha/install/QIIME1.5/bin/python

Dependency versions

===================

                     PyCogent version:  1.5.1

                        NumPy version:  1.5.1

                   matplotlib version:  1.1.0

                  biom-format version:  0.9.3

                QIIME library version:  1.5.0

                 QIIME script version:  1.5.0

        PyNAST version (if installed):  1.1

RDP Classifier version (if installed):  rdp_classifier-2.2.jar

QIIME config values

===================

                     blastmat_dir:      None

                         sc_queue:      all.q

      topiaryexplorer_project_dir:      None

     pynast_template_alignment_fp:      /home/maasha/install/QIIME/data/gg_otus_4feb2011/rep_set/gg_97_otus_4feb2011_aligned.fasta

                  cluster_jobs_fp:      /home/maasha/Install/QIIME/bin/start_parallel_jobs.py

pynast_template_alignment_blastdb:      /home/maasha/install/QIIME/data/core_set_aligned.fasta.imputed

assign_taxonomy_reference_seqs_fp:      None

                     torque_queue:      friendlyq

              qiime_test_data_dir:      None

   template_alignment_lanemask_fp:      /home/maasha/install/QIIME/data/lanemask_in_1s_and_0s

                    jobs_to_start:      10

                cloud_environment:      False

                qiime_scripts_dir:      /home/maasha/Install/QIIME/bin

            denoiser_min_per_core:      50

                      working_dir:      None

                    python_exe_fp:      python

                         temp_dir:      /tmp

                      blastall_fp:      blastall

                 seconds_to_sleep:      60

assign_taxonomy_id_to_taxonomy_fp:      None

running checks:

test_FastTree_supported_version (__main__.Qiime_config)

FastTree is in path and version is supported ... ok

test_INFERNAL_supported_version (__main__.Qiime_config)

INFERNAL is in path and version is supported ... ok

test_ParsInsert_supported_version (__main__.Qiime_config)

ParsInsert is in path and version is supported ... ok

test_R_supported_version (__main__.Qiime_config)

R is in path and version is supported ... ok

test_ampliconnoise_install (__main__.Qiime_config)

AmpliconNoise install looks sane. ... ok

test_blast_supported_version (__main__.Qiime_config)

blast is in path and version is supported ... ok

test_blastall_fp (__main__.Qiime_config)

blastall_fp is set to a valid path ... ok

test_blastmat_dir (__main__.Qiime_config)

blastmat_dir is set to a valid path. ... ok

test_cdbtools_supported_version (__main__.Qiime_config)

cdbtools is in path and version is supported ... ok

test_cdhit_supported_version (__main__.Qiime_config)

cd-hit is in path and version is supported ... ok

test_chimeraSlayer_install (__main__.Qiime_config)

no obvious problems with ChimeraSlayer install ... ok

test_clearcut_supported_version (__main__.Qiime_config)

clearcut is in path and version is supported ... ok

test_cluster_jobs_fp (__main__.Qiime_config)

cluster_jobs_fp is set to a valid path and is executable ... ok

test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use ... FAIL

test_for_obsolete_values (__main__.Qiime_config)

local qiime_config has no extra params ... ok

test_matplotlib_suported_version (__main__.Qiime_config)

maptplotlib version is supported ... ok

test_mothur_supported_version (__main__.Qiime_config)

mothur is in path and version is supported ... ok

test_muscle_supported_version (__main__.Qiime_config)

muscle is in path and version is supported ... ok

test_numpy_suported_version (__main__.Qiime_config)

numpy version is supported ... ok

test_pplacer_supported_version (__main__.Qiime_config)

pplacer is in path and version is supported ... ok

test_pynast_suported_version (__main__.Qiime_config)

pynast version is supported ... ok

test_pynast_template_alignment_blastdb_fp (__main__.Qiime_config)

pynast_template_alignment_blastdb, if set, is set to a valid path ... ok

test_pynast_template_alignment_fp (__main__.Qiime_config)

pynast_template_alignment, if set, is set to a valid path ... ok

test_python_exe_fp (__main__.Qiime_config)

python_exe_fp is set to a working python env ... ok

test_python_supported_version (__main__.Qiime_config)

python is in path and version is supported ... ok

test_qiime_scripts_dir (__main__.Qiime_config)

qiime_scripts_dir, if set, is set to a valid path ... ok

test_qiime_test_data_dir (__main__.Qiime_config)

qiime_test_data_dir, if set, is set to a valid path ... ok

test_raxmlHPC_supported_version (__main__.Qiime_config)

raxmlHPC is in path and version is supported ... ok

test_rtax_supported_version (__main__.Qiime_config)

rtax is in path and version is supported ... ok

test_temp_dir (__main__.Qiime_config)

temp_dir, if set, is set to a valid path ... ok

test_template_alignment_lanemask_fp (__main__.Qiime_config)

template_alignment_lanemask, if set, is set to a valid path ... ok

test_uclust_supported_version (__main__.Qiime_config)

uclust is in path and version is supported ... ok

test_usearch_supported_version (__main__.Qiime_config)

usearch is in path and version is supported ... ok

test_working_dir (__main__.Qiime_config)

working_dir, if set, is set to a valid path ... ok

======================================================================

FAIL: test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/maasha/install/QIIME/bin/print_qiime_config.py", line 536, in test_denoiser_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

Ran 34 tests in 0.324s

FAILED (failures=1)

And here is what I have done so far step-by-step:

QIIME-1.5.0 install

===================

  Documentation: http://qiime.org/install/install.html

  home    = /home/maasha/

  install = /home/maasha/install

  prefix  = /home/maasha/install/QIIME/

  source  = /home/maasha/install/QIIME/src/

System

======

Linux-3.2.0-0.bpo.1-amd64 x86_64 GNU/Linux

Debian 6.0.4

Prerequisites

=============

   Installing Python-2.7.1

   -----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz

   3. tar -zxvf Python-2.7.1.tgz

   4. ln -s Python-2.7.1 Python

   5. cd Python/

   6. configure --prefix=/home/maasha/install/QIIME/

   7. make

   8. make install

   Installing Numpy-1.5.1

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/numpy/NumPy/1.5.1/numpy-1.5.1.tar.gz

   3. tar -zxvf numpy-1.5.1.tar.gz

   4. ln -s numpy-1.5.1 numpy

   5. cd numpy/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing PyCogent-1.5.1

   -------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/pycogent/PyCogent/1.5.1/PyCogent-1.5.1.tgz

   3. tar -zxvf PyCogent-1.5.1.tgz

   4. ln -s PyCogent-1.5.1 PyCogent

   5. cd PyCogent/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing setuptools-0.6.c11 (required to build biom)

   -----------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz

   3. tar -zxvf setuptools-0.6c11.tar.gz

   4. ln -s setuptools-0.6c11 setuptools

   5. cd setuptools/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing Sphinx-1.1.3 (required to build html docs for Qiime)

   ---------------------------------------------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://pypi.python.org/packages/source/S/Sphinx/Sphinx-1.1.3.tar.gz

   3. tar -zxvf Sphinx-1.1.3.tar.gz

   4. ln -s Sphinx-1.1.3 Sphinx

   5. cd Sphinx/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing biom-format-0.9.3

   ----------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget https://github.com/downloads/biom-format/biom-format/biom-format-0.9.3.tgz

   3. tar -zxvf biom-format-0.9.3.tgz

   4. ln -s biom-format-0.9.3 biom-format

   5. cd biom-format/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

Installing QIIME

================

   Installing QIIME-1.5.0

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/qiime/releases/Qiime-1.5.0.tar.gz

   3. tar -xvf Qiime-1.5.0.tar.gz

   4. ln -s Qiime-1.5.0 Qiime

   5. cd Qiime/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Configuring QIIME

   -----------------

   1. Users must add the following line to their ~/.bashrc file:

        /home/maasha/install/QIIME/bashrc

      Then source .bashrc or relogin.

   2. Users must copy configuration file to their home directory:

        cp $QIIMEPATH/src/Qiime/qiime/support_files/qiime_config ~/.qiime_config

   3. Test configuration by running

        print_qiime_config.py -t

Installing Extras

=================

   Installing uclust

   -----------------

   1. cd /home/maasha/install/QIIME/bin/

   2. wget http://www.drive5.com/uclust/uclustq1.2.22_i86linux64

   3. ln -s uclustq1.2.22_i86linux64 uclust

   4. chmod 755 uclustq1.2.22_i86linux64

   Installing PyNast-1.1

   ---------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/pynast/PyNAST%20releases/PyNAST-1.1.tgz

   3. tar -zxvf PyNAST-1.1.tgz

   4. ln -s PyNAST-1.1 PyNAST

   5. cd PyNAST/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing matplotlib-1.1.0

   ---------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/matplotlib-1.1.0.tar.gz

   3. tar -zxvf matplotlib-1.1.0.tar.gz

   4. ln -s matplotlib-1.1.0 matplotlib

   5. cd matplotlib/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

   Installing cytoscape-2.8.3

   --------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://chianti.ucsd.edu/Cyto-2_8_3/Cytoscape_2_8_3_unix.sh

   3. sh Cytoscape_2_8_3_unix.sh

   4. install in /home/maasha/install/QIIME/

   5. cd /home/maasha/install/QIIME/bin/

   6. ln -s ../Cytoscape_v2.8.3/cytoscape.sh cytoscape

   Installing R-2.12

   -----------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://mirrors.dotsrc.org/cran/src/base/R-2/R-2.12.0.tar.gz

   3. tar -zxvf R-2.12.0.tar.gz

   4. ln -s R-2.12.0 R

   5. cd R/

   6. ./configure --prefix=/home/maasha/install/QIIME/

   7. make

   8. make install

   9. ../../bin/R

   10. install.packages('randomForest')

   11. q()

   Installing FastTree-2.1.3

   -------------------------

   1. mkdir /home/maasha/install/QIIME/src/FastTree

   2. cd /home/maasha/install/QIIME/src/FastTree

   3. wget http://www.microbesonline.org/fasttree/FastTree-2.1.3.c

   4. gcc -Wall -O3 -finline-functions -funroll-loops -o FastTree -lm FastTree.c

   5. cd /home/maasha/install/QIIME/bin/

   6. ln -s ../src/FastTree/FastTree FastTree

   Installing RDP classifier

   -------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/rdp-classifier/rdp-classifier/rdp_classifier_2.2.zip

   3. unzip rdp_classifier_2.2.zip

   4. add export RDP_JAR_PATH="$QIIMEPATH/src/rdp_classifier_2.2/rdp_classifier-2.2.jar" to bashrc

   Installing usearch-5.2.32

   -------------------------

   1. obtain license: http://www.drive5.com/usearch/nonprofit_form.html

   2. download binary to /home/maasha/install/QIIME/bin/

   3. chmod 755 usearch5.2.32_i86linux32

   4. ln -s usearch5.2.32_i86linux32 usearch

   Installing blast-2.2.22

   -----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget ftp://ftp.ncbi.nlm.nih.gov/blast/executables/release/2.2.22/blast-2.2.22-x64-linux.tar.gz

   3. tar -zxvf blast-2.2.22-ia64-linux.tar.gz

   4. ln -s blast-2.2.22 blast

   5. cd /home/maasha/install/QIIME/bin/

   6. for i in ../src/blast/bin/*; do ln -s $i `basename $i`; done

   Installing CD-HIT-3.1.1

   -----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://www.bioinformatics.org/download/cd-hit/cd-hit-2007-0131.tar.gz

   3. tar -zxvf cd-hit-2007-0131.tar.gz

   4. cd cd-hit/

   5. make

   6. cd /home/maasha/install/QIIME/bin

   7. for i in cd-hit cd-hit cd-hit-est cd-hit-2d cd-hit-est-2d cd-hit-div; do ln -s ../src/cd-hit/$i $i; done

   Installing ChimeraSlayer

   ------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/microbiomeutil/microbiomeutil-r20110519.tgz

   3. tar -zxvf microbiomeutil-r20110519.tgz

   4. add export PATH="$QIIMEPATH/src/microbiomeutil-r20110519/ChimeraSlayer:$PATH" to bashrc

   Installing Mothur-1.25.0

   ------------------------

   1. mkdir /home/maasha/install/QIIME/src/Mothur-1.25.0

   2. cd /home/maasha/install/QIIME/src/

   3. ln -s Mothur-1.25.0 Mothur

   4. cd Mothur/

   5. wget http://www.mothur.org/w/images/6/6d/Mothur.1.25.0.zip

   6. unzip Mothur.1.25.0.zip

   7. edit makefile and comment out line 31: #TARGET_ARCH += -arch x86_64 

   8. make

   9. cd /home/maasha/install/QIIME/bin/

   10. ln -s ../src/Mothur/Mothur.source/mothur mothur

   Installing clearcut-1.0.9

   -------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://www.mothur.org/w/images/9/91/Clearcut.source.zip

   3. unzip Clearcut.source.zip

   4. cd clearcut/

   5. make

   6. cd /home/maasha/install/QIIME/bin/

   7. ln -s ../src/clearcut/clearcut clearcut

   Installing raxml-7.3.0

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget ftp://thebeast.colorado.edu/pub/QIIME-v1.5.0-dependencies/stamatak-standard-RAxML-5_7_2012.tgz

   3. tar -zxvf stamatak-standard-RAxML-5_7_2012.tgz

   4. ln -s stamatak-standard-RAxML-5_7_2012 raxml

   5. cd raxml/

   6. make -f Makefile.SSE3.gcc

   7. cd /home/maasha/install/QIIME/bin/

   8. ln -s ../src/raxml/raxmlHPC-SSE3 raxmlHPC

   Installing infernal-1.0.2

   -------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget ftp://selab.janelia.org/pub/software/infernal/infernal.tar.gz

   3. tar -zxvf infernal.tar.gz

   4. ln -s infernal-1.0.2/ infernal

   5. cd infernal/

   6. configure --prefix=/home/maasha/install/QIIME/

   7. make

   8. make install

   Installing cdbtools

   -------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget ftp://occams.dfci.harvard.edu/pub/bio/tgi/software/cdbfasta/cdbfasta.tar.gz

   3. tar -zxvf cdbfasta.tar.gz

   4. cd cdbfasta/

   5. make

   6. cd /home/maasha/install/QIIME/bin/

   7. ln -s ../src/cdbfasta/cdbfasta cdbfasta

   8. ln -s ../src/cdbfasta/cdbyank cdbyank

   Installing Muscle-3.8.31

   ------------------------

   1. cd /home/maasha/install/QIIME/bin/

   2. wget http://www.drive5.com/muscle/downloads3.8.31/muscle3.8.31_i86linux64.tar.gz

   3. tar -zxvf muscle3.8.31_i86linux64.tar.gz

   4. ln -s muscle3.8.31_i86linux64 muscle

   Installing rtax-0.981

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://dev.davidsoergel.com/trac/rtax/raw-attachment/wiki/Releases/rtax-0.981.tgz

   3. tar -zxvf rtax-0.981.tgz

   4. ln -s rtax-0.981 rtax

   5. cd /home/maasha/install/QIIME/bin/

   6. ln -s ../src/rtax/rtax rtax

   Installing pplacer-1.1

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://matsen.fhcrc.org/pplacer/builds/pplacer-v1.1-Linux.tar.gz

   3. tar -zxvf pplacer-v1.1-Linux.tar.gz

   4. ln -s pplacer-v1.1.alpha12-Linux-2.6.32/ pplacer

   5. cd /home/maasha/install/QIIME/bin/

   7. for i in guppy pplacer rppr; do ln -s ../src/pplacer/$i $i; done

   Installing ParsInsert-1.04

   --------------------------

   1. mkdir /home/maasha/install/QIIME/src/ParsInsert-1.04

   2. cd /home/maasha/install/QIIME/src/

   3. ln -s ParsInsert-1.04 ParsInsert

   4. cd ParsInsert/

   5. wget http://downloads.sourceforge.net/project/parsinsert/ParsInsert.1.04.tgz

   6. make

   7. cd /home/maasha/install/QIIME/bin/

   8. ln -s ../src/ParsInsert/ParsInsert ParsInsert

   Installing AmpliconNoiseV1.25

   -----------------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://ampliconnoise.googlecode.com/files/AmpliconNoiseV1.25.tar.gz

   3. tar -zxvf AmpliconNoiseV1.25.tar.gz

   4. ln -s AmpliconNoiseV1.25 AmpliconNoise

   5. cd AmpliconNoise/

   6. make

   7. make install

   8. cd /home/maasha/install/QIIME/bin/

   9. for i in ../src/AmpliconNoise/bin/*; do ln -s $i `basename $i`; done

Installing tools from Roche

===========================

  1. copy sfffile to /home/maasha/install/QIIME/bin/

  2. copy sffinfo to /home/maasha/install/QIIME/bin/

Installing data

===============

   mkdir /home/maasha/install/QIIME/data

   Installling greengenes core set data file

   -----------------------------------------

   1. cd /home/maasha/install/QIIME/data

   2. wget http://greengenes.lbl.gov/Download/Sequence_Data/Fasta_data_files/core_set_aligned.fasta.imputed

   3. edit ~/.qiime_config and set pynast_template_alignment_blastdb /home/maasha/install/QIIME/data/core_set_aligned.fasta.imputed

   Installing greengenes alignment lanemask file

   ---------------------------------------------

   1. cd /home/maasha/install/QIIME/data

   2. wget http://greengenes.lbl.gov/Download/Sequence_Data/lanemask_in_1s_and_0s

   3. edit ~/.qiime_config and set template_alignment_lanemask_fp /home/maasha/install/QIIME/data/lanemask_in_1s_and_0s

   Installing greengenes taxonomy

   ------------------------------

   1. cd /home/maasha/install/QIIME/data

   2. wget http://greengenes.lbl.gov/Download/Sequence_Data/Fasta_data_files/Caporaso_Reference_OTUs/gg_otus_4feb2011.tgz

   3. tar -zxvf gg_otus_4feb2011.tgz 

   3. edit ~/.qiime_config and set pynast_template_alignment_fp /home/maasha/install/QIIME/data/gg_otus_4feb2011/rep_set/gg_97_otus_4feb2011_aligned.fasta

The bashrc file (/home/maasha/install/QIIME/bashrc)

===================================================

QIIMEPATH="/home/maasha/install/QIIME"

export BLASTMAT="$QIIMEPATH/src/blast/data"

export PATH="$QIIMEPATH/bin:$PATH"

export PATH="$QIIMEPATH/src/microbiomeutil-r20110519/ChimeraSlayer:$PATH"

export PATH="$QIIMEPATH/src/AmpliconNoise/Scripts:$PATH"

export PATH="$QIIMEPATH/src/AmpliconNoise/bin:$PATH"

export PYTHONPATH="$QIIMEPATH/src/PyCogent:$PYTHONPATH"

export PYTHONPATH="$QIIMEPATH/lib:$PYTHONPATH"

export RDP_JAR_PATH="$QIIMEPATH/src/rdp_classifier_2.2/rdp_classifier-2.2.jar"

export PYRO_LOOKUP_FILE="$QIIMEPATH/src/AmpliconNoise/Data/LookUp_E123.dat"

export SEQ_LOOKUP_FILE="$QIIMEPATH/src/AmpliconNoise/Data/Tran.dat"

The .qiime_config file (~/.qiime_config)

========================================

# qiime_config

# WARNING: DO NOT EDIT OR DELETE Qiime/qiime_config

# To overwrite defaults, copy this file to $HOME/.qiime_config or a full path

# specified by $QIIME_CONFIG_FP and edit that copy of the file.

cluster_jobs_fp /home/maasha/Install/QIIME/bin/start_parallel_jobs.py

python_exe_fp   python

working_dir

blastmat_dir

blastall_fp     blastall

pynast_template_alignment_fp     /home/maasha/install/QIIME/data/gg_otus_4feb2011/rep_set/gg_97_otus_4feb2011_aligned.fasta

pynast_template_alignment_blastdb       /home/maasha/install/QIIME/data/core_set_aligned.fasta.imputed

template_alignment_lanemask_fp  /home/maasha/install/QIIME/data/lanemask_in_1s_and_0s

jobs_to_start   10

seconds_to_sleep        60

qiime_scripts_dir       /home/maasha/Install/QIIME/bin

temp_dir        /tmp

denoiser_min_per_core   50

cloud_environment       False

topiaryexplorer_project_dir

torque_queue    friendlyq

Martin's comments

=================

1. I did not install Java. I fear building Java from source, so relying on Debian packages.

Cheers,


Martin

[Jose Carlos Clemente]

Martin,

do you have ghc installed? That will be needed to compile the denoiser binary.

Jose

[maasha]

So this issue has to do with AmpliconNoise? According to print_qiime_config.py -t AmpliconNoise looks ok -> "AmpliconNoise install looks sane. ... ok"

ghc-6.12.1 is installed. It is pretty ancient since it is a Debian Stable package, but I believe it was OK for compiling QIIME 1.3.

What is the exact name of the missing 'Denoiser flowgram aligner'?

Cheers,

Martin

[Jose Carlos Clemente]

Martin,

the issue is not with AmpliconNoise but with Denoiser, which is not
working in your install:

denoiser aligner is ready to use ... FAIL

the binary you are looking for is called FlowgramAli_4frame, which
should be under Qiime/qiime/support_files/denoiser/bin. Since it is
not there, try going to ...denoiser/FlowgramAlignment, type make and
copy the binary to denoiser/bin.

Let me know how it goes,

Jose

[Antonio González Peña]

Hi Martin,

Due to our move to the newest Ubuntu LTS, Denoiser now needs a newer
ghc. We tested with 7.4.1 provided by the repositories without a
problem.

If you want some more install instructions check the end of this thread:
http://groups.google.com/group/qiime-forum/browse_thread/thread/f149e9f80b327df2

Cheers

--
Antonio González Peña
Research Assistant, Knight Lab
University of Colorado at Boulder
https://chem.colorado.edu/knightgroup/

[Martin Asser Hansen]

Hello,

Now I have build ghc-7.4.1 manually, and then rebuild Qiime (setup.py clean/build/install).

Running print_qiime_config.py -t then said:

Ran 34 tests in 0.547s

OK

However, after relogin the FAIL reappears!?!

FlowgramAli_4frame is located in

../qiime/support_files/denoiser/bin/

and is soft linked to the Qiime/bin directory.

Thus typing cd && Flowgramali_4frame yields:

maasha@orsted:~$ FlowgramAli_4frame 

Usage: FlowgramAli_4frame mode error_profile input_file

where mode:

        -align                   Align all flowgrams in input against first flowgram in input

        -score                   Only compute alignment score

        -flow-lengths            print translated nucleotide length if flowgrams in input

        -relscore                Compute lenghth normalized alignment score

        -relscore_pairid         Compute length normalized alignment score and report %pair id of aligned seqs

        -self                    Fast self-alignment score for all flowgrams in input

        -gapless                 Fast gapless alignment of first flowgram in input against rest in input

So Flowgramali_4frame is in place and executable and I don't understand the fail:

======================================================================

FAIL: test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/maasha/install/QIIME/bin/print_qiime_config.py", line 536, in test_denoiser_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

?

Cheers,

Martin

[Jose Carlos Clemente]

Martin,

what if you try to add the directory in which the binary resides to
your path, rather than adding a link to qiime/bin? Does that solve the
problem?

Jose

[Martin Asser Hansen]

I already did try this. Problem stays the same.

Cheers,

Martin

[Jose Carlos Clemente]

Martin,

after checking the code, the binary for the denoiser is expected to be
exactly at this directory:

/path/to/your_qiime/qiime/support_files/denoiser/bin/FlowgramAli_4frame

so even if it's in your path, but not in that directory, it won't
work. Could you check if this fixes it?

Jose

[Martin Asser Hansen]

Hm,

maasha@orsted:~$ cd install/QIIME

maasha@orsted:~/install/QIIME$ find | grep FlowgramAli_4frame

./src/Qiime-1.5.0/qiime/support_files/denoiser/bin/FlowgramAli_4frame

./src/Qiime-1.5.0/qiime/support_files/denoiser/FlowgramAlignment/FlowgramAli_4frame.lhs

./src/Qiime-1.5.0/qiime/support_files/denoiser/FlowgramAlignment/FlowgramAli_4frame.hi

./src/Qiime-1.5.0/qiime/support_files/denoiser/FlowgramAlignment/FlowgramAli_4frame.o

./src/Qiime-1.5.0/qiime/support_files/denoiser/FlowgramAlignment/FlowgramAli_4frame

./qiime/support_files/denoiser/bin/FlowgramAli_4frame

maasha@orsted:~/install/QIIME$ ll ./qiime/support_files/denoiser/bin/FlowgramAli_4frame

-rwxr-xr-x 1 maasha users 1564038 May 18 20:19 ./qiime/support_files/denoiser/bin/FlowgramAli_4frame

And still:

======================================================================

FAIL: test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/maasha/install/QIIME/bin/print_qiime_config.py", line 536, in test_denoiser_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

----------------------------------------------------------------------

Cheers,

Martin

[Jose Carlos Clemente]

Martin,

we are working on figuring out what might be the issue. In the
meantime, what if you try to denoise the tutorial dataset?

http://www.qiime.org/tutorials/denoising_454_data.html

If that works, the problem would be in the test rather than in the
denoiser install.

Jose

[Jens Reeder]

Hi Martin,

From your notes:

Installing QIIME

================

   Installing QIIME-1.5.0

   ----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://downloads.sourceforge.net/project/qiime/releases/Qiime-1.5.0.tar.gz

   3. tar -xvf Qiime-1.5.0.tar.gz

   4. ln -s Qiime-1.5.0 Qiime

   5. cd Qiime/

   6. ../../bin/python setup.py build

   7. ../../bin/python setup.py install

I gather that you installed qiime without specifying any location. As far as I can test on my machine, this doesn't work for the qiime denoiser setup.
You need to make sure that the install-data path points to the same place as the purelib path, e.g:

python setup.py install --install-scripts=~/qiime/bin/ --install-purelib=~/qiime/lib/ --install-data=~/qiime/lib/

The qiime installation page has some more details on that.
Let me know if that works for you.

Jens

[Martin Asser Hansen]

@Jens

Note that I build all QIIME components on top of my own Python compilation that was build and installed with --prefix=/home/maasha/install/QIIME. Thus, to my understanding, all QIIME components build with Python should then be installed to the correct local locations per default with explicit setting install paths. I do this without sudo, and I get no permission errors indicating that the installation is in any way system-wide.

Cheers,

Martin

[Martin Asser Hansen]

@Jose,

OK, I am getting this error message from Denoiser:

maasha@orsted:~/scratch/Course$ nice denoise_wrapper.py -v -i HN42GJ101_max5000.sff.txt -f HN42GJ101_max5000.fasta -o Denoise -m EMTM_2012_Mapping_File.txt -n 8

Traceback (most recent call last):

  File "/home/maasha/install/QIIME/bin/denoise_wrapper.py", line 159, in <module>

    main()

  File "/home/maasha/install/QIIME/bin/denoise_wrapper.py", line 145, in main

    titanium=opts.titanium)

  File "/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/denoise_wrapper.py", line 34, in fast_denoiser

    verbose=verbose, titanium=titanium)

  File "/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/denoiser/flowgram_clustering.py", line 556, in denoise_seqs

    check_flowgram_ali_exe()

  File "/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/denoiser/utils.py", line 55, in check_flowgram_ali_exe

    ("The alignment program is not where it's supposed to be: %s" %  ali_fp)

cogent.app.util.ApplicationNotFoundError: The alignment program is not where it's supposed to be: /home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/support_files/denoiser/bin/FlowgramAli_4frame

And FlowgramAli_4frame is in fact not found at that location, which contains:

maasha@orsted:~/install/QIIME/lib/python2.7/site-packages/qiime/support_files/denoiser$ ll

total 0

drwxr-xr-x 4 maasha users  32 May 14 14:08 .

drwxr-xr-x 8 maasha users  90 May 14 14:08 ..

drwxr-xr-x 2 maasha users  91 May 14 14:08 Data

drwxr-xr-x 2 maasha users 111 May 14 14:08 TestData 

So, I have the executable sitting in the source tree:

/home/maasha/install/QIIME/src/Qiime-1.5.0/qiime/support_files/denoiser/bin/FlowgramAli_4frame

And if I copy the executable to the expected location then print_qiime_config.py -t is A-OK. So something funky happened during the installation. And possible related  ... denoiase_wrapper.py errs:

denoise_wrapper.py -v -i HN42GJ101_max5000.sff.txt -f HN42GJ101_max5000.fasta -o Denoise -m EMTM_2012_Mapping_File.txt -n 8 --force

Traceback (most recent call last):

  File "/home/maasha/Install/QIIME/bin/denoiser_preprocess.py", line 137, in <module>

    main()

  File "/home/maasha/Install/QIIME/bin/denoiser_preprocess.py", line 120, in main

    primer=opts.primer)

  File "/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/denoiser/preprocess.py", line 202, in preprocess

    barcode_mapping = extract_barcodes_from_mapping(labels)

  File "/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/denoiser/flowgram_filter.py", line 253, in extract_barcodes_from_mapping

    flowgram_id = tmatch.group(2)

AttributeError: 'NoneType' object has no attribute 'group'

Martin

[Jose Carlos Clemente]

Martin,

ok so copying the binary did work, that's something at least.

Concerning the error you are seeing, two things: did you check your
mapping file with check_id_map.py? And how did you generate the
flowgrams file (sff.txt)?

Jose

[Martin Asser Hansen]

Mapping file checks out OK:

check_id_map.py -m EMTM_2012_Mapping_File.txt -o mapping_output

No errors or warnings were found in mapping file.

And sff.txt was generated with sffinfo 2.6 (20110517_1502) like this:

sffinfo HN42GJ101_max5000.sff > HN42GJ101_max5000.sff.txt

Cheers

Martin

[Jose Carlos Clemente]

Seems like we've seen this before, can you check the thread below and
see whether that fixes your problem?

https://groups.google.com/group/qiime-forum/browse_thread/thread/b4d906e8b6b34709

Jose

[Martin Asser Hansen]

I read though the thread where it looked like the culprit was bad sequence names. Mine are of the format HN42GJ101C9ECG which should be OK. But if this error is unrelated to the installation of denoiser, then I guess I should open a new thread (after checking existing).

Cheers,

Martin

[Jens Reeder]

Hi Masha,

glad to hear that copying the file worked for you.
Maybe one more note on this in case other have the same problem.
The standards python setup:

$ python setup.py install

will put the binary somewhere outside of the python site_packages. I guess in your case into:

/home/maasha/install/QIIME/qiime/support_files/...

But it is expected in

/home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/support_files

Pointing the install-data to
/home/maasha/install/QIIME/lib/python2.7/site-packages/
should have done the trick as well.

Jens

[Jens Reeder]

Martin,

can you post the first few records of your fasta file?

Jens

 p.s. and yes opening a new thread would be the nicest way, but since we are already way into it...

[Martin Asser Hansen]

Here goes:

maasha@orsted:~/scratch/Course$ head -n 20 HN42GJ101_max5000.fasta

>HN42GJ101B0K59 length=288 xy=0709_2719 region=1 run=R_2012_05_16_05_20_33_

TAGTCGCATACCTACGGGAGGCAGCAGTCGAGAATTTTCACAATGGGGGAAACCCTGATG

GAGCGACGGCGCGTGAGGATGAAGCCTTCGGTTGTAAACTCCTGTCATGCGGGAACAAGC

AAGTGATAGTACCGCAAGAGGAAGAGACGGCTAACTCTGTGCCAGCAGCCGCGGTAATAC

AGAGGTCTCAAGCGTTGTTCGGATTCATTGGGCGTAAAGGGTGCGTAGGTGGTGGCGCAA

GTCTGGTGTGAAATCTCGGAGCTCAACTCCGAAATTGCACCGGATACT

>HN42GJ101BUK3T length=315 xy=0641_1223 region=1 run=R_2012_05_16_05_20_33_

CTACGCTCTACCTATGGGCGAGCAGTCGAGAATTTTCTACAATGGGCGAACCCTGATGGA

GCGACGGCGCGTGAGGATGAAGCTTCGGTTGTAACTCCTGTCATGCGGGAACAAGCAAGT

GATAGTACCGCAAGAGGAAGAGACGGCTAACTCTGTGCCAGCAGCCGCGGTAATACAGAG

GTCTCAAAGCGTTGTTCGGATTCATTGGGCGTAAAGGGTGCGTAGGTGGTGATGCAAGTC

TGGTGTGAAATCTCGGGGCTCAACTCCGAAATTGCACCGGATACTGCGTGAACTCGAGGA

CTGTAGAGGAGATCG

>HN42GJ101AO57R length=346 xy=0169_2341 region=1 run=R_2012_05_16_05_20_33_

ACAGTCGTGCCCTATGGGGCAGCAGTGGGGAAGTATTGGACAATGGGCGCAAGCCTGATC

CAGCCATGCCGCGTGAGTGATGAAGGCCCTAGGGTTGTAAAGCTCTTTCGGCGGGGAAGA

TAATGACGGTACCCGCAGAAGAAGCCCCGGCTAACTTCGTGCCAGCAGCCGCGGTAATCG

AAGGGGGCTAGCGTTGCTCGGAATCACTGGGCGTAAAGCGCAACGTTAGGCGGCTTCTTA

AAGTCAGGGGTGAAATCCTGGAGCTCAAACTCCAGAACTTGCCTTTGATACTGAGAGGCC

TAGAGTCCGGGANAGGTGAGTGGAACCTCCGAGTGTAGAGGTGAAA

Martin

[Jens Reeder]

Hi Martin,

This file looks like a regular fasta file translated from the .sff file.
But, for the step that you are trying to run, you need the output fasta from split_libraries.py.
That file will look as described here.

Jens

[Martin Asser Hansen]

Thanks Jens,

That was of cause the problem. Me bad. It would be useful with a more relevant error message though.

So all-in-all my QIIME1.5 installation is now complete. My install instructions - with the last tweaks - are below just in case anyone can use them.

Cheers,

Martin

QIIME-1.5.0 install

===================

  Documentation: http://qiime.org/install/install.html

  home    = /home/maasha/

  install = /home/maasha/install

  prefix  = /home/maasha/install/QIIME/

  source  = /home/maasha/install/QIIME/src/

System

======

Linux-3.2.0-0.bpo.1-amd64 x86_64 GNU/Linux

Debian 6.0.4

Prerequisites

=============

   Installing Python-2.7.1

   -----------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz

   3. tar -zxvf Python-2.7.1.tgz

   4. ln -s Python-2.7.1 Python

   5. cd Python/

   6. ./configure --prefix=/home/maasha/install/QIIME/

   7. make

   8. make install

   Installing ghc-7.4.1

   --------------------

   1. cd /home/maasha/install/QIIME/src/

   2. wget http://www.haskell.org/ghc/dist/7.4.1/ghc-7.4.1-src.tar.bz2

   3. ln -s ghc-7.4.1 ghc

   4. cd ghc/

2. Fix for denoiser where the FlowgramAli_4frame binary for some reason is mislocated:

   cp /home/maasha/install/QIIME/src/Qiime-1.5.0/qiime/support_files/denoiser/bin/FlowgramAli_4frame

   /home/maasha/install/QIIME/lib/python2.7/site-packages/qiime/support_files/denoiser

[maasha]

OK, Now I am installing Qiime-1.6.0 and am stuck here again:

======================================================================

FAIL: test_denoiser_supported_version (__main__.Qiime_config)

denoiser aligner is ready to use

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/maasha/install/QIIME1.6/bin/print_qiime_config.py", line 538, in test_denoiser_supported_version

    "which components of QIIME you plan to use.")

AssertionError: Denoiser flowgram aligner not found or not executable.This may or may not be a problem depending on which components of QIIME you plan to use.

I did copy FlowgramAli_4frame, which saved the day last time, but that does not do the trick no more :o(

cp /home/maasha/install/QIIME1.6/src/Qiime-1.6.0/qiime/support_files/denoiser/bin/FlowgramAli_4frame

/home/maasha/install/QIIME1.6/lib/python2.7/site-packages/qiime/support_files/denoiser

The detailed version of what I did is here: https://gist.github.com/4656341



Cheers,

Martin

[Jai Ram Rideout]

Hi Martin,

Can you please send the full output of running 'print_qiime_config.py -t'?

Thanks,

Jai

--
 
---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group, send email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Martin Asser Hansen]

Sure:

https://gist.github.com/4657199

Martin

[Jai Ram Rideout]

Hi Martin,

Thanks for providing the output. It looks like there's something strange with your QIIME config because qiime_scripts_dir points to /home/maasha/Install/QIIME/bin, but from your install notes, the QIIME scripts should be under /home/maasha/install/QIIME1.6/.

Can you please run the following commands and send me the output?

echo $PATH

echo $PYTHONPATH

Also, please execute the 'python' command and then run the following commands:

import qiime

qiime.__file__

Thanks,

Jai

[Martin Asser Hansen]

Of cause! Me bad - mixing QIIME1.5 and QIIME1.6 installs. It works now, and I am down to the usearch err - which I chose to ignore.

Thanks

Martin

[Jai Ram Rideout]

Hi Martin,

Glad to hear you got it working! We highly recommend updating to the version of usearch supported by QIIME 1.6.0 because there were a number of bug fixes between the version you have and the recommended version.

You can obtain usearch from http://www.drive5.com/usearch/download.html

-Jai

...

[Message clipped]