Qiime installation in virtualenv on HPC fails because numpy isn't found

[Michael Baron]

Hi everybody,

This is a bit niche and I might not find an answer here.

I'm trying to get Qiime running on our HPC cluster. Alas, 'pip install --user qiime' failed, as the sys.path wouldn't work.

Will not install to the user site because it will lack sys.path precedence to setuptools in /shared/ucl/apps/python/bundles/1453203686-2/venv/lib/python2.7/site-packages

So I moved on to a virtualenv.

virtualenv qiime
source qiime/bin/activate
pip install numpy

Unfortunately, the virtualenv installation fails as well because -- despite Numpy being installed first -- pip throws an error during the cogent installation:

pip install qiime

Collecting qiime

  Using cached qiime-1.9.1.tar.gz

Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in ./qiime/lib/python2.7/site-packages (from qiime)

Collecting scipy>=0.14.0 (from qiime)

  Using cached scipy-0.17.0.tar.gz

Collecting cogent==1.5.3 (from qiime)

  Using cached cogent-1.5.3.tgz

    Complete output from command python setup.py egg_info:

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/tmp/pip-build-2Mcziv/cogent/setup.py", line 25, in <module>

        raise RuntimeError("Numpy required but not found.")

    RuntimeError: Numpy required but not found.

    ----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2Mcziv/cogent

Any clues? Must be my environment, but I'd like to give it a shot 'myself' before I bother the HPC support people.

Thanks,

Michael

[Colin Brislawn]

Hello Michael,

I think trying a virtual environment is the right approach, but the one that comes with python is a bit week. Have you tried out Anaconda? I use that deploy qiime on several different clusters with great success (no sudo needed).

Install Miniconda from here: http://conda.pydata.org/miniconda.html

Then install qiime like I do here: https://github.com/biocore/qiime/issues/2108#issuecomment-161790019 

Let me know how this works. (This is going to be the default way to install qiime 2 so your feedback is appreciated.)

Colin Brislawn

[Michael Baron]

Hi Colin,

Thanks for helping me out (again)! :)

The Anaconda installation went without a hitch (just wget'ed the .sh file and executed it). However, your command in the gist throws an error:

conda -n create qiime191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata h5py
usage: conda [-h] [-V] [--debug] command ...
conda: error: unrecognized arguments: -n

I guess conda-env is the equivalent to virtualenv. And from a quick google search 'conda create env_name env_packages' would set it up. Is the -n tag deprecated?

Thanks,
Michael

edit: I see, it was just an order issue. -n stand for name.

[Michael Baron]

A short follow up. I ran the following code:

conda create -n qiime191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata h5py
source activate qiime191
pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz

And it seems the installation went through:

Collecting https://github.com/biocore/qiime/archive/1.9.1.tar.gz

  Downloading https://github.com/biocore/qiime/archive/1.9.1.tar.gz

     | 20.3MB 10.4MB/s

Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)

Requirement already satisfied (use --upgrade to upgrade): scipy>=0.14.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)

Collecting cogent==1.5.3 (from qiime==1.9.1)

  Using cached cogent-1.5.3.tgz

Collecting natsort<4.0.0 (from qiime==1.9.1)

  Using cached natsort-3.5.6-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): matplotlib!=1.4.2,>=1.1.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)

Collecting pynast==1.2.2 (from qiime==1.9.1)

  Using cached pynast-1.2.2.tar.gz

Collecting qcli<0.2.0,>=0.1.1 (from qiime==1.9.1)

  Using cached qcli-0.1.1.tar.gz

Requirement already satisfied (use --upgrade to upgrade): gdata in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)

Collecting biom-format<2.2.0,>=2.1.4 (from qiime==1.9.1)

  Using cached biom-format-2.1.5.tar.gz

Collecting emperor<1.0.0,>=0.9.51 (from qiime==1.9.1)

  Using cached emperor-0.9.51.tar.gz

Collecting scikit-bio<0.3.0,>=0.2.3 (from qiime==1.9.1)

  Using cached scikit-bio-0.2.3.tar.gz

Collecting burrito-fillings<0.2.0,>=0.1.1 (from qiime==1.9.1)

  Using cached burrito-fillings-0.1.1.tar.gz

Requirement already satisfied (use --upgrade to upgrade): pandas>=0.13.1 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)

Collecting burrito<1.0.0,>=0.9.1 (from qiime==1.9.1)

  Using cached burrito-0.9.1.tar.gz

Collecting qiime-default-reference<0.2.0,>=0.1.2 (from qiime==1.9.1)

  Using cached qiime-default-reference-0.1.3.tar.gz

Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)

Requirement already satisfied (use --upgrade to upgrade): pytz in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)

Requirement already satisfied (use --upgrade to upgrade): cycler in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)

Requirement already satisfied (use --upgrade to upgrade): pyparsing!=2.0.4,>=1.5.6 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)

Collecting click (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)

  Using cached click-6.2-py2.py3-none-any.whl

Requirement already satisfied (use --upgrade to upgrade): future>=0.14.3 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)

Collecting pyqi (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)

  Using cached pyqi-0.3.2.tar.gz

Requirement already satisfied (use --upgrade to upgrade): six in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from scikit-bio<0.3.0,>=0.2.3->qiime==1.9.1)

Requirement already satisfied (use --upgrade to upgrade): IPython in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from scikit-bio<0.3.0,>=0.2.3->qiime==1.9.1)

Building wheels for collected packages: cogent, pynast, qcli, biom-format, emperor, scikit-bio, burrito-fillings, burrito, qiime-default-reference, pyqi

  Running setup.py bdist_wheel for cogent ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/31/4a/1e/5e89366707d4ed8d510521c6185662503e6098f2234721ecf4

  Running setup.py bdist_wheel for pynast ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/f5/c6/15/140e55821648e3edbdf539bdcd13973b72eabf7a6037f7ba17

  Running setup.py bdist_wheel for qcli ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/fd/59/31/4c48e4ecfb3ad3c5cc98614ddf92d332b24431dc9afd64644b

  Running setup.py bdist_wheel for biom-format ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/5a/e4/11/ef896761f31e4d9a8a6a5216136d46634ae436f5da3fd5dad3

  Running setup.py bdist_wheel for emperor ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/41/12/db/a1085947c85572878b0b7c6ea18f4c62110ad2eee1af89d80f

  Running setup.py bdist_wheel for scikit-bio ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/cf/81/f7/5f307d109d3dfb5ce16d112541407b569197373abb386a7e5b

  Running setup.py bdist_wheel for burrito-fillings ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/4e/a7/d5/3995f264f46c3bedb065384beafcbec3b2a1b893c1afa6ba21

  Running setup.py bdist_wheel for burrito ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/04/6e/03/b773992d32f3761527a3f093f1857f86b27f83076f82084fa7

  Running setup.py bdist_wheel for qiime-default-reference ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/b7/ce/c9/9b7a13ec1c9a4fc7faf3da0921d4cf94eee040173c1f9fc8f0

  Running setup.py bdist_wheel for pyqi ... done

  Stored in directory: /home/sejj036/.cache/pip/wheels/9e/d2/3a/f21057d3aed9a859cead95630f30de1144b4235ec36884d188

Successfully built cogent pynast qcli biom-format emperor scikit-bio burrito-fillings burrito qiime-default-reference pyqi

Installing collected packages: cogent, natsort, pynast, qcli, click, pyqi, biom-format, scikit-bio, emperor, burrito, burrito-fillings, qiime-default-reference, qiime

  Found existing installation: natsort 4.0.3

    Uninstalling natsort-4.0.3:

      Successfully uninstalled natsort-4.0.3

  Found existing installation: scikit-bio 0.4.0

    Uninstalling scikit-bio-0.4.0:

      Successfully uninstalled scikit-bio-0.4.0

  Running setup.py install for qiime ... done

Successfully installed biom-format-2.1.5 burrito-0.9.1 burrito-fillings-0.1.1 click-6.2 cogent-1.5.3 emperor-0.9.51 natsort-3.5.6 pynast-1.2.2 pyqi-0.3.2 qcli-0.1.1 qiime-1.9.1 qiime-default-reference-0.1.3 scikit-bio-0.2.3

When I try to print the qiime config to check the installation, it seems to get its knickers into a twist:

print_qiime_config.py

Traceback (most recent call last):

  File "/home/sejj036/miniconda2/envs/qiime191/bin/print_qiime_config.py", line 40, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: /home/sejj036/miniconda2/envs/qiime191/lib/python2.7/site-packages/skbio/alignment/_ssw_wrapper.so: undefined symbol: _intel_fast_memcpy

See the QIIME Installation Guide: http://qiime.org/install/install.html

Did I pick the wrong compiler? I vaguely remember there was a caveat somewhere in the HPC documentation that I should pick the GCC, but that would imply C-code...

Thanks,

Michael

[Colin Brislawn]

Good catch. I've edited my comment. 

Colin

[Colin Brislawn]

That's odd... I'm not totally sure what would be causing that. What OS is on your supercomputer? For example, when I run 'cat /etc/issue' it tells me:

Red Hat Enterprise Linux Workstation release 6.6 (Santiago)

At one point, I remember having to manually load a gcc compiler to make some python software happy. You could try the same to see if it helps. I ran something like 'module load gcc' then tried running python again.

Maybe a qiime dev can help more...

Colin

[Michael Baron]

It runs Red Hat as well.

And as it turns out it likely has been the compiler. This is from our HPC documentation:

Set compiler module
The Python versions on Legion were built with GCC. You can run them with the default Intel compilers loaded because everything depends on the gcc-libs/4.9.2 module. When you are building your own Python packages you should have the GCC compiler module loaded however, to avoid the situation where you build a package with the Intel compiler and then try to run it with GCC, in which case it will be unable to find Intel-specific instructions.
module unload compilers
module load compilers/gnu/4.9.2
If you get an error like this when trying to run something, recheck what compiler you used.
undefined symbol: __intel_sse2_strrchr

I'll rebuild the virtual environment later today and report back.

Lesson learned: double check the documentation!

Thanks again, Colin!

[Michael Baron]

Alas, rebuilding the environment with the GNU compiler, or even reinstalling miniconda did not help.

I'll have to get in touch with our HPC support, however, if someone else has any insight of what is failing, I'd be grateful.

Thanks,

Michael

[Michael Baron]

Caching was the culprit. Once I cleared out all the previously compiled python modules (~./cache/), it went through without a hitch.

I even got it working in the standard python virtualenv (the --no-cache-dir is a little redundant):

module unload compilers
module load compilers/gnu/4.9.2

virtualenv qiime191
source qiime191/bin/activate
pip install --no-cache-dir numpy
# paths to blas and lapack
export BLAS=/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/libblas.so
export LAPACK=/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/liblapack.so
pip install --no-cache-dir scipy
pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz --no-cache-dir

Thus always make sure the correct compiler is selected.

[Colin Brislawn]

Very good troubleshooting on your part. Glad you figured that out!

Let us know if you have any more questions,

Colin