MacQIIME----ImportError: cannot import name HAVE_H5PY

[Ming LIao]

Hello, dear all

I find out the latest discussion about this problem was several months ago, but there was still no solution. Are there anyone who have fixed this problem now?  

I am using MacQIIME, there was the problem:

MacQIIME blk071:Yao $ pick_otus.py 

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/pick_otus.py", line 20, in <module>

    from qiime.util import (parse_command_line_parameters, create_dir,

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/util.py", line 41, in <module>

    from biom.util import compute_counts_per_sample_stats, biom_open, HAVE_H5PY

ImportError: cannot import name HAVE_H5PY 

[zhiying Guo]

Hello,

You installed QIIME on MAC or Virtual Box? I think python maybe something wrong on you installed QIIME.

--

---
You received this message because you are subscribed to the Google Groups "Qiime Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qiime-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ming Liao]

Hello,

I installed MacQIIME on my macbook, and can open it when I type macqiime on the terminal.

However, when I next type whatever script in it, it turned out the same error, please see as follows:

blk192:~ mingliao$ macqiime

MacQIIME version:

MacQIIME 1.9.1-20150604

Sourcing MacQIIME environment variables...

  This is the same as a normal terminal shell, except your default

  python is DIFFERENT (/macqiime/bin/python) and there are other new

  QIIME-related things in your PATH.

  Type "exit" (return) to go back to your normal shell

MacQIIME blk192:~ $ principal_coordinates.py

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/principal_coordinates.py", line 17, in <module>

    from qiime.util import (parse_command_line_parameters, get_options_lookup,

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/util.py", line 41, in <module>

    from biom.util import compute_counts_per_sample_stats, biom_open, HAVE_H5PY

ImportError: cannot import name HAVE_H5PY

Something must be wrong here. I don't know why. Please help. Thank you.

Sincerely,

Ming

[Kyle Bittinger]

Ming,

Would you please post the output of print_qiime_config.py?

[Ming Liao]

Of course. but it comes up with the same error...

MacQIIME blk192:~ $ print_qiime_config.py

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/print_qiime_config.py", line 57, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: cannot import name HAVE_H5PY

See the QIIME Installation Guide: http://qiime.org/install/install.html

[Kyle Bittinger]

OK then!  :)

I'm pretty sure you have an old version of the biom-format library.  The variable HAVE_H5PY is only imported once in the QIIME codebase, at the top of our util module:

https://github.com/biocore/qiime/blob/master/qiime/util.py#L41

If this variable was not found in the biom-format library, you would get the observed error.

Try this:

sudo pip install --upgrade biom-format

[Ming Liao]

I tried it. but there were another problem. 

MacQIIME blk192:~ $ sudo pip install --upgrade biom-format

......

......

......

Command "/Users/mingliao/anaconda/bin/python3 -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-o1u4cr0q/numpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-pvmqkk3o-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-o1u4cr0q/numpy

According to the error, I have tried to reinstall the numpy, but nothing changed....

Ming

[Ming Liao]

I think it is better to show you the entire output:

MacQIIME blk192:~ $ sudo pip install --upgrade biom-format

Password:

The directory '/Users/mingliao/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

The directory '/Users/mingliao/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Collecting biom-format

  Downloading biom-format-2.1.5.tar.gz (307kB)

    100% |████████████████████████████████| 311kB 806kB/s 

Requirement already up-to-date: click in /macqiime/anaconda/lib/python2.7/site-packages (from biom-format)

Collecting numpy>=1.3.0 (from biom-format)

  Downloading numpy-1.10.1.tar.gz (4.0MB)

    100% |████████████████████████████████| 4.1MB 64kB/s 

Collecting future>=0.14.3 (from biom-format)

  Downloading future-0.15.2.tar.gz (1.6MB)

    100% |████████████████████████████████| 1.6MB 168kB/s 

Collecting scipy>=0.13.0 (from biom-format)

  Downloading scipy-0.16.1.tar.gz (12.2MB)

    100% |████████████████████████████████| 12.2MB 21kB/s 

Requirement already up-to-date: pyqi in /macqiime/anaconda/lib/python2.7/site-packages (from biom-format)

Installing collected packages: numpy, future, scipy, biom-format

  Found existing installation: numpy 1.9.2

    DEPRECATION: Uninstalling a distutils installed project (numpy) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.

    Uninstalling numpy-1.9.2:

      Successfully uninstalled numpy-1.9.2

  Running setup.py install for numpy

    Complete output from command /macqiime/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-IomTyE/numpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-XIsRPF-record/install-record.txt --single-version-externally-managed --compile:

    Running from numpy source directory.

    blas_opt_info:

    blas_mkl_info:

      libraries mkl,vml,guide not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    openblas_info:

      libraries openblas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    atlas_3_10_blas_threads_info:

    Setting PTATLAS=ATLAS

      libraries tatlas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    atlas_3_10_blas_info:

      libraries satlas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    atlas_blas_threads_info:

    Setting PTATLAS=ATLAS

      libraries ptf77blas,ptcblas,atlas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    atlas_blas_info:

      libraries f77blas,cblas,atlas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    /private/tmp/pip-build-IomTyE/numpy/numpy/distutils/system_info.py:635: UserWarning: Specified path  is invalid.

      warnings.warn('Specified path %s is invalid.' % d)

      FOUND:

        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']

        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

        extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']

    

    

    You have not agreed to the Xcode license agreements, please run 'xcodebuild -license' (for user-level acceptance) or 'sudo xcodebuild -license' (for system-wide acceptance) from within a Terminal window to review and agree to the Xcode license agreements.

    

    You have not agreed to the Xcode license agreements, please run 'xcodebuild -license' (for user-level acceptance) or 'sudo xcodebuild -license' (for system-wide acceptance) from within a Terminal window to review and agree to the Xcode license agreements.

    non-existing path in 'numpy/distutils': 'site.cfg'

    F2PY Version 2

    lapack_opt_info:

    openblas_lapack_info:

      libraries openblas not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

    lapack_mkl_info:

    mkl_info:

      libraries mkl,vml,guide not found in ['/macqiime/anaconda/lib', '/usr/local/lib', '/usr/lib']

      NOT AVAILABLE

    

      NOT AVAILABLE

    

    atlas_3_10_threads_info:

    Setting PTATLAS=ATLAS

      libraries tatlas,tatlas not found in /macqiime/anaconda/lib

      libraries lapack_atlas not found in /macqiime/anaconda/lib

      libraries tatlas,tatlas not found in /usr/local/lib

      libraries lapack_atlas not found in /usr/local/lib

      libraries tatlas,tatlas not found in /usr/lib

      libraries lapack_atlas not found in /usr/lib

    <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>

      NOT AVAILABLE

    

    atlas_3_10_info:

      libraries satlas,satlas not found in /macqiime/anaconda/lib

      libraries lapack_atlas not found in /macqiime/anaconda/lib

      libraries satlas,satlas not found in /usr/local/lib

      libraries lapack_atlas not found in /usr/local/lib

      libraries satlas,satlas not found in /usr/lib

      libraries lapack_atlas not found in /usr/lib

    <class 'numpy.distutils.system_info.atlas_3_10_info'>

      NOT AVAILABLE

    

    atlas_threads_info:

    Setting PTATLAS=ATLAS

      libraries ptf77blas,ptcblas,atlas not found in /macqiime/anaconda/lib

      libraries lapack_atlas not found in /macqiime/anaconda/lib

      libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib

      libraries lapack_atlas not found in /usr/local/lib

      libraries ptf77blas,ptcblas,atlas not found in /usr/lib

      libraries lapack_atlas not found in /usr/lib

    <class 'numpy.distutils.system_info.atlas_threads_info'>

      NOT AVAILABLE

    

    atlas_info:

      libraries f77blas,cblas,atlas not found in /macqiime/anaconda/lib

      libraries lapack_atlas not found in /macqiime/anaconda/lib

      libraries f77blas,cblas,atlas not found in /usr/local/lib

      libraries lapack_atlas not found in /usr/local/lib

      libraries f77blas,cblas,atlas not found in /usr/lib

      libraries lapack_atlas not found in /usr/lib

    <class 'numpy.distutils.system_info.atlas_info'>

      NOT AVAILABLE

    

      FOUND:

        extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']

        define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

        extra_compile_args = ['-msse3']

    

    /macqiime/anaconda/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros'

      warnings.warn(msg)

    running install

    running build

    running config_cc

    unifing config_cc, config, build_clib, build_ext, build commands --compiler options

    running config_fc

    unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options

    running build_src

    build_src

    building py_modules sources

    creating build

    creating build/src.macosx-10.5-x86_64-2.7

    creating build/src.macosx-10.5-x86_64-2.7/numpy

    creating build/src.macosx-10.5-x86_64-2.7/numpy/distutils

    building library "npymath" sources

    customize Gnu95FCompiler

    Could not locate executable gfortran

    Could not locate executable f95

    customize NAGFCompiler

    customize AbsoftFCompiler

    Could not locate executable f90

    Could not locate executable f77

    customize IBMFCompiler

    Could not locate executable xlf90

    Could not locate executable xlf

    customize IntelFCompiler

    Could not locate executable ifort

    Could not locate executable ifc

    customize GnuFCompiler

    Could not locate executable g77

    customize G95FCompiler

    Could not locate executable g95

    customize PGroupFCompiler

    Could not locate executable pgfortran

    don't know how to compile Fortran code on platform 'posix'

    C compiler: gcc -fno-strict-aliasing -I/macqiime/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

    

    compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/macqiime/anaconda/include/python2.7 -c'

    gcc: _configtest.c

    

    You have not agreed to the Xcode license agreements, please run 'xcodebuild -license' (for user-level acceptance) or 'sudo xcodebuild -license' (for system-wide acceptance) from within a Terminal window to review and agree to the Xcode license agreements.

    

    You have not agreed to the Xcode license agreements, please run 'xcodebuild -license' (for user-level acceptance) or 'sudo xcodebuild -license' (for system-wide acceptance) from within a Terminal window to review and agree to the Xcode license agreements.

    failure.

    removing: _configtest.c _configtest.o

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/private/tmp/pip-build-IomTyE/numpy/setup.py", line 264, in <module>

        setup_package()

      File "/private/tmp/pip-build-IomTyE/numpy/setup.py", line 256, in setup_package

        setup(**metadata)

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/core.py", line 169, in setup

        return old_setup(**new_attr)

      File "/macqiime/anaconda/lib/python2.7/distutils/core.py", line 151, in setup

        dist.run_commands()

      File "/macqiime/anaconda/lib/python2.7/distutils/dist.py", line 953, in run_commands

        self.run_command(cmd)

      File "/macqiime/anaconda/lib/python2.7/distutils/dist.py", line 972, in run_command

        cmd_obj.run()

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/install.py", line 62, in run

        r = self.setuptools_run()

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run

        return distutils_install.run(self)

      File "/macqiime/anaconda/lib/python2.7/distutils/command/install.py", line 563, in run

        self.run_command('build')

      File "/macqiime/anaconda/lib/python2.7/distutils/cmd.py", line 326, in run_command

        self.distribution.run_command(command)

      File "/macqiime/anaconda/lib/python2.7/distutils/dist.py", line 972, in run_command

        cmd_obj.run()

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/build.py", line 47, in run

        old_build.run(self)

      File "/macqiime/anaconda/lib/python2.7/distutils/command/build.py", line 127, in run

        self.run_command(cmd_name)

      File "/macqiime/anaconda/lib/python2.7/distutils/cmd.py", line 326, in run_command

        self.distribution.run_command(command)

      File "/macqiime/anaconda/lib/python2.7/distutils/dist.py", line 972, in run_command

        cmd_obj.run()

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/build_src.py", line 153, in run

        self.build_sources()

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/build_src.py", line 164, in build_sources

        self.build_library_sources(*libname_info)

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/build_src.py", line 299, in build_library_sources

        sources = self.generate_sources(sources, (lib_name, build_info))

      File "/private/tmp/pip-build-IomTyE/numpy/numpy/distutils/command/build_src.py", line 386, in generate_sources

        source = func(extension, build_dir)

      File "numpy/core/setup.py", line 669, in get_mathlib_info

        raise RuntimeError("Broken toolchain: cannot link a simple C program")

    RuntimeError: Broken toolchain: cannot link a simple C program

    

    ----------------------------------------

  Rolling back uninstall of numpy

Command "/macqiime/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-IomTyE/numpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-XIsRPF-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-IomTyE/numpy

[Ming Liao]

By the way, the error is still the same even I tried    sudo xcodebuild -license

Sincerely

Ming

[Kyle Bittinger]

I don't understand how xcodebuild is involved here, or why the error message would be the same as an error message from pip.

I'm going to suggest that you reinstall MacQIIME.  The homepage has instructions about how to move the current installation to a backup location, and has instructions for how to re-install.

http://www.wernerlab.org/software/macqiime/macqiime-installation

Best,

Kyle

[Ming Liao]

Hello, Kyle

I have tried to reinstall the MacQiime, and runned sudo pip install --upgrade biom-format

it ends up with a long long text output, the final of the output is as follows: 

Command "/macqiime/anaconda/bin/python -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-ALtT1P/scipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-FZVV6T-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-build-ALtT1P/scipy

the I tried this... 

MacQIIME Mings-MacBook-Pro:qiime_tutorial $ print_qiime_config.py

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/print_qiime_config.py", line 57, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: cannot import name HAVE_H5PY

See the QIIME Installation Guide: http://qiime.org/install/install.html

It is the same problem!

I am sure it is not related to my MAC, as it has been updated to the latest operating system.  As shown in the installation introduction, Macqiime can run on Mac OS 10.6 and up. What can I do now? Thank you.

Sincerely,

Ming

[Kyle Bittinger]

Are you running El Capitan?

[Ming Liao]

No, it is OS X Yosemite

Version 10.10.5

There seems to some problems when I updated it to El Capitan. I thought it was updated because I have downloaded the OS and installed it. But it still turned out to be version 10.10.5. As shown in the installation introduction, Macqiime can run on Mac OS 10.6 and up. 

Is my version 10.10.5 is upper than 10.6? 

[Kyle Bittinger]

Ming, would you try re-naming your current macqiime folder and re-installing? Instructions are here:

http://www.wernerlab.org/software/macqiime/macqiime-installation

[Ming Liao]

Hello, Kyle Bittinger,

I am really helpless. I have tried to re-install it, and rename the macqiime folder, it turned out the same error. 

By the way, I asked help from our colleague who is good at computer science, he tried to find out what was the problem and fix it. Unfortunately, he gave up after one hours' fixing. 

I am wondering if anyone can successfully use macqiime? 

I will keep asking for help.

Thanks, Kyle.

Sincerely,

Ming 

[Jenya Kopylov]

Hi Ming,

What is your output of these commands:

$ macqiime

$ python -c "import h5py; print h5py.__version__"

Thanks,

Jenya

[Ming Liao]

Thanks, Jenya

I have asked someone to fix the problem of h5py. I am not sure whether it is fixed, but after that it seems that h5py can be successfully imported. However, it comes up with another problem. Please see as follows:

MacQIIME blk147:qiime $ python -c "import h5py; print h5py.__version__"

2.5.0

MacQIIME blk147:qiime $ make_otu_table.py

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/make_otu_table.py", line 22, in <module>

    from qiime.filter import (get_seq_ids_from_seq_id_file,

  File "/macqiime/anaconda/lib/python2.7/site-packages/qiime/filter.py", line 20, in <module>

    from biom import load_table

ImportError: cannot import name load_table

MacQIIME blk147:qiime $ python -c "import load_table; print load_table.__version__"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

ImportError: No module named load_table

[Jenya Kopylov]

The load_table function is from the biom format. You can check whether it's installed with the command:

$ python -c "import biom; print biom.__version__"

Can you paste here the output of,

$ print_qiime_config.py

Thanks!

[Ming Liao]

MacQIIME blk147:qiime $ python -c "import biom; print biom.__version__"

1.3.1

MacQIIME blk147:qiime $ print_qiime_config.py

Traceback (most recent call last):

  File "/macqiime/anaconda/bin/print_qiime_config.py", line 57, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: cannot import name load_table

See the QIIME Installation Guide: http://qiime.org/install/install.html

[Jenya Kopylov]

Thanks!

The biom-format package needs to be >= 2.1.4, < 2.2.0 for QIIME 1.9.1.

What do you get in your run the command,

$ pip install biom-format==2.1.4

[Ming Liao]

I got the output as follows

.....

    Uninstalling biom-format-1.3.1:

      Successfully uninstalled biom-format-1.3.1

  Running setup.py install for biom-format

Successfully installed biom-format-2.1.4

The biom-format has been updated, but the problem is still there as follows:

[Jenya Kopylov]

Can you output the results of these commands? Make sure you called 'macqiime' beforehand.

$ pip show biom-format

and

$ echo $PATH

$ echo $PYTHONPATH

[Ming Liao]

Of course.  I am sure that "macqiime" was called at first. Please see the following output. Thanks.

MacQIIME blk147:qiime $ pip show biom-format

---

Metadata-Version: 1.1

Name: biom-format

Version: 2.1.4

Summary: Biological Observation Matrix (BIOM) format

Home-page: http://www.biom-format.org

Author: Daniel McDonald

Author-email: mcdo...@colorado.edu

License: BSD

Location: /macqiime/anaconda/lib/python2.7/site-packages

Requires: numpy, pyqi, scipy

MacQIIME blk147:qiime $  echo $PATH

/macqiime/QIIME/bin:/macqiime/bin:/macqiime/rtax-0.984:/macqiime/rtax-0.984/scripts:/macqiime/anaconda/bin:/macqiime/microbiomeutil_2010-04-29/ChimeraSlayer:/macqiime/microbiomeutil_2010-04-29/NAST-Ier:/macqiime/microbiomeutil_2010-04-29/WigeoN:/macqiime/blat:/opt/local/bin:/opt/local/sbin:/Users/mingliao/anaconda/bin://anaconda/bin:/Users/mingliao/perl5/perlbrew/bin:/Users/mingliao/perl5/perlbrew/perls/perl-5.16.0/bin:/Users/mingliao/bin/:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

MacQIIME blk147:qiime $ echo $PYTHONPATH

/Users/mingliao/lib/:

[Jenya Kopylov]

Alright, I'm not sure why your $PYTHONPATH wasn't updated to include "/macqiime/anaconda/lib/python2.7/site-packages", seems to be an error seen before here. In fact, my own $PYTHONPATH doesn't update when I run 'macqiime', however as I already had the correct biom-format version installed elsewhere, I didn't receive your errors.

To resolve this,

$ export PYTHONPATH=/macqiime/anaconda/lib/python2.7/site-packages:$PYTHONPATH

Let me know if it's working.

[Ming Liao]

Great! It seems working after your instruction. Do you think my MacQIIME can work now?

MacQIIME blk147:qiime $ print_qiime_config.py

System information

==================

         Platform: darwin

   Python version: 2.7.10 |Anaconda 2.2.0 (x86_64)| (default, May 28 2015, 17:04:42)  [GCC 4.2.1 (Apple Inc. build 5577)]

Python executable: /macqiime/anaconda/bin/python

QIIME default reference information

===================================

For details on what files are used as QIIME's default references, see here:

 https://github.com/biocore/qiime-default-reference/releases/tag/0.1.2

Dependency versions

===================

          QIIME library version: 1.9.1

           QIIME script version: 1.9.1

qiime-default-reference version: 0.1.2

                  NumPy version: 1.10.1

                  SciPy version: 0.15.1

                 pandas version: 0.16.1

             matplotlib version: 1.4.3

            biom-format version: 2.1.4

                   h5py version: 2.5.0 (HDF5 version: 1.8.14)

                   qcli version: 0.1.1

                   pyqi version: 0.3.2

             scikit-bio version: 0.2.3

                 PyNAST version: 1.2.2

                Emperor version: 0.9.51

                burrito version: 0.9.1

       burrito-fillings version: 0.1.1

              sortmerna version: SortMeRNA version 2.0, 29/11/2014

              sumaclust version: SUMACLUST Version 1.0.00

                  swarm version: Swarm 1.2.19 [Jun  2 2015 14:40:16]

                          gdata: Installed.

QIIME config values

===================

For definitions of these settings and to learn how to configure QIIME, see here:

 http://qiime.org/install/qiime_config.html

 http://qiime.org/tutorials/parallel_qiime.html

                     blastmat_dir: None

      pick_otus_reference_seqs_fp: /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                         sc_queue: all.q

      topiaryexplorer_project_dir: None

     pynast_template_alignment_fp: /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set_aligned/85_otus.pynast.fasta

                  cluster_jobs_fp: start_parallel_jobs.py

pynast_template_alignment_blastdb: None

assign_taxonomy_reference_seqs_fp: /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/rep_set/97_otus.fasta

                     torque_queue: friendlyq

                    jobs_to_start: 1

                       slurm_time: None

            denoiser_min_per_core: 50

assign_taxonomy_id_to_taxonomy_fp: /macqiime/anaconda/lib/python2.7/site-packages/qiime_default_reference/gg_13_8_otus/taxonomy/97_otu_taxonomy.txt

                         temp_dir: /tmp/

                     slurm_memory: None

                      slurm_queue: None

                      blastall_fp: blastall

                 seconds_to_sleep: 60

MacQIIME blk147:qiime $ 

[Jenya Kopylov]

Yes, everything should be working now. However, unless you add the line:

export PYTHONPATH=/macqiime/anaconda/lib/python2.7/site-packages:$PYTHONPATH

to your ~/.bash_profile file, you will need to run that command every time after you call "$ macqiime" since it is a temporary setting.

Let us know if you have other questions!

Jenya

[Ming Liao]

Jenya, thank you so much. I have added the line as you mentioned. Hopefully it will work now.

Actually, I have been wasting too much time on the alternative ways of running qiime, like installing virtual box on Mac, or using the qiime source, but nothing works well. I really appreciate your help. Thanks again.

Best regards,

Ming

[Jeff Werner]

Wow, what a mess, sorry about all that! I'm confused why I've never seen this problem come up before, but I will make sure it is fixed in the next release.

[Ming Liao]

Hello, Jenya,

I have a question about PYTHONPATH. 

When I did not run macqiime at first, I can NOT run the other python scripts not related to qiime. After I run macqiime at first, the other python scripts can work.

How I can run other python scripts out of macqiime? like the python script "print "Hello, world" " 

Sincerely,

Ming

[Ming Liao]

Let me show you in details:

When it is in the MacQiime, the test.py can work 

MacQIIME blk147:metaphlan2 $ python test.py

Hello, World

When it out of MacQiime, the test.py can not work 

MacQIIME blk147:metaphlan2 $ exit

exit

Forgetting MacQIIME environment variable settings...

blk147:~ mingliao$ python test.py

Failed to import the site module

......

Then I checked the python version in two ways, one is out of MacQiime:

blk147:~ mingliao$ python -V

Python 3.4.3 :: Anaconda 2.2.0 (x86_64)

the other one is in MacQiime: 

blk147:~ mingliao$ macqiime

MacQIIME version:

MacQIIME 1.9.1-20150604

Sourcing MacQIIME environment variables...

  This is the same as a normal terminal shell, except your default

  python is DIFFERENT (/macqiime/bin/python) and there are other new

  QIIME-related things in your PATH.

  Type "exit" (return) to go back to your normal shell

MacQIIME blk147:~ $ python -V

Python 2.7.10 :: Anaconda 2.2.0 (x86_64)

Is the problem related to the python version? I searched on google. Someone said the pre-install Python in Mac is old, and the apple not gonna to update it. But I don't care about that. I am just wondering,  how can I run some python scripts out of Macqiime?

Have a great thanksgiving day!

Ming

[Jenya Kopylov]

Hi Ming,

When you call macqiime, it prepends certain directories for executables and libraries to your PATH and PYTHONPATH env variables needed to run QIIME.

For example, try calling $PATH before and after you call macqiime. After calling macqiime, you should see the following directories prepended to the original $PATH directories:

/macqiime/QIIME/bin:/macqiime/bin:/macqiime/rtax-0.984:/macqiime/rtax-0.984/scripts:/macqiime/anaconda/bin:/macqiime/microbiomeutil_2010-04-29/ChimeraSlayer:/macqiime/microbiomeutil_2010-04-29/NAST-Ier:/macqiime/microbiomeutil_2010-04-29/WigeoN:/macqiime/blat:$PATH

[Ming Liao]

Thanks, Jenya, I can understand the difference between PATH and PYTHONPATH now. 

But I am still confused about that, why the other python that can run in Macqiime but can not work outside Macqiime?

blk147:metaphlan2 mingliao$ python test.py

Failed to import the site module

Traceback (most recent call last):

  File "/macqiime/anaconda/lib/python2.7/site-packages/site.py", line 75, in <module>

    __boot()

  File "/macqiime/anaconda/lib/python2.7/site-packages/site.py", line 23, in __boot

    loader.load_module('site')

  File "/Users/mingliao/anaconda/lib/python3.4/site.py", line 562, in <module>

    main()

  File "/Users/mingliao/anaconda/lib/python3.4/site.py", line 548, in main

    known_paths = addusersitepackages(known_paths)

  File "/Users/mingliao/anaconda/lib/python3.4/site.py", line 281, in addusersitepackages

    user_site = getusersitepackages()

  File "/Users/mingliao/anaconda/lib/python3.4/site.py", line 257, in getusersitepackages

    user_base = getuserbase() # this will also set USER_BASE

  File "/Users/mingliao/anaconda/lib/python3.4/site.py", line 247, in getuserbase

    USER_BASE = get_config_var('userbase')

  File "/Users/mingliao/anaconda/lib/python3.4/sysconfig.py", line 589, in get_config_var

    return get_config_vars().get(name)

  File "/Users/mingliao/anaconda/lib/python3.4/sysconfig.py", line 568, in get_config_vars

    import _osx_support

  File "/Users/mingliao/anaconda/lib/python3.4/_osx_support.py", line 4, in <module>

    import re

  File "/Users/mingliao/anaconda/lib/python3.4/re.py", line 336, in <module>

    import copyreg

  File "/macqiime/anaconda/lib/python2.7/site-packages/copyreg/__init__.py", line 7, in <module>

    raise ImportError('This package should not be accessible on Python 3. '

ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.

[Jenya Kopylov]

Hi Ming,

When you exit from macqiime and type $PYTHONPATH, do you still see "/macqiime/anaconda/lib/python2.7/site-packages" at the beginning of the string?

[Ming Liao]

Yes, I see it, then?

blk147:metaphlan2 mingliao$ echo $PYTHONPATH

/Users/mingliao/Downloads/nsegata-graphlan:/macqiime/anaconda/lib/python2.7/site-packages:/Users/mingliao/lib/:

[Jenya Kopylov]

Seems that macqiime doesn't remove this path from PYTHONPATH upon exit, try removing it yourself and running python test.py again.

Here is a command to remove /macqiime/anaconda/lib/python2.7/site-packages, it should work however echo your PYTHONPATH first so in case it fails you can reset manually.

$ echo $PYTHONPATH

/example/path/1:/example/path/2

$ PYTHONPATH=$(echo "$PYTHONPATH" | sed 's|/macqiime/anaconda/lib/python2.7/site-packages:||g')

The check,

$ echo $PYTHONPATH

If /macqiime/anaconda/lib/python2.7/site-packages was removed but the rest of the directories are there, you're good! If you see something else, then use the output of your first $PYTHONPATH to reset it,

$ PYTHONPATH=/example/path/1:/example/path/2

and search a google more to figure out the correct command for removing paths from env variables (ex. PYTHONPATH).

[Ming Liao]

Thanks Jenya, It works. The test python script works outside the MacQiime.

Can  I ask more about the PYTHONPATH? When the MacQiime exits, does the PYTHONPATH have the same function as PATH?  

or When I install python software outside the macqiime, which path should I export to? 

I really appreciate your help.

sincerely,

Ming

[Jenya Kopylov]

Hi Ming,

Here is a good thread on stack overflow describing the differences between PATH and PYTHONPATH.

I'm not sure exactly what you mean by 'which paths you should export to' regarding installation of new software.

The default directories where software are installed vary depending on the method you used to install them and whether you have sudo access.

For example, if you install python packages using pip, here is a thread discussing ways you can determine where these packages were installed.

For questions outside of QIIME, it's probably best to search google first as it contains a lot of good information & generally the first place I go myself.

Good luck with your work!

Jenya

[Jeff Werner]

The issue here is that MacQIIME has its own built-in version of Python, and you have another version of Python on your computer that you use otherwise when MAcQIIME isn't sourced. Some Python scripts may work in both versions, but some may only work in MacQIIME's version of python (e.g. all the QIIME scripts) and some scripts may only work in your regular native python, if they require other libraries etc that you have in your regular Python. There isn't a good way to make everything work everywhere because some stuff requires either different versions of Python or different versions of Python libraries. This is why I put MacQIIME in a "bottle" with its own Python installation, so that it won't mess with anything else you normally use in your regular Python install.  

The missing PYTHONPATH variable is a mistake on my part, sorry about that.  The way I will fix it in the future, and how you could fix it now, is to put this line into the /macqiime/configs/bash_profile.txt file:

export PYTHONPATH=/macqiime/anaconda/lib/python2.7/site-packages/

This will replace your normal PYTHONPATH only when you have macqiime sourced, which is how it is meant to work.  And, don't have that folder sourced in your regular environment (which I think you have right now).  That way MacQIIME stays on its own, and your regular python stuff will stay on its own, without mixing the two (which will very likely cause errors and incompatibilities).

If you want to add software to the MacQIIME environment, you can do that while you have macqiime sourced and it should all end up in the macqiime anaconda folder. If the package you want to add to macqiime is available through anaconda, the best way to install it is to use the "conda" script, or pip if it isn't available through conda.  If you want to install software to your regular default python outside of macqiime, just do it when you don't have macqiime sourced. They're separate python installs and it's good to keep track of what is installed in each, without mixing them up.

Cheers,

Jeff

[Ming Liao]

Great, Jeff, Thank you so much

I am now much more aware of what I was doing before. Your information is very helpful for me, who has no idea about the how python works, what is condo or pip, or  other things like that. The way I am learning to use a software like qiime is to follow the instruction in the tutorial. If some problems happen when using the software, I don't know how to find it out, even though it seems very simple technically. 

Thank you for your patient and explanation in such a details. 

Sincerely,

Ming