Qiime installation in virtualenv on HPC fails because numpy isn't found

338 views
Skip to first unread message

Michael Baron

unread,
Feb 16, 2016, 2:02:47 PM2/16/16
to Qiime 1 Forum
Hi everybody,

This is a bit niche and I might not find an answer here.

I'm trying to get Qiime running on our HPC cluster. Alas, 'pip install --user qiime' failed, as the sys.path wouldn't work.

Will not install to the user site because it will lack sys.path precedence to setuptools in /shared/ucl/apps/python/bundles/1453203686-2/venv/lib/python2.7/site-packages

So I moved on to a virtualenv.

virtualenv qiime
source qiime/bin/activate
pip install numpy

Unfortunately, the virtualenv installation fails as well because -- despite Numpy being installed first -- pip throws an error during the cogent installation:

pip install qiime
Collecting qiime
  Using cached qiime-1.9.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in ./qiime/lib/python2.7/site-packages (from qiime)
Collecting scipy>=0.14.0 (from qiime)
  Using cached scipy-0.17.0.tar.gz
Collecting cogent==1.5.3 (from qiime)
  Using cached cogent-1.5.3.tgz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-2Mcziv/cogent/setup.py", line 25, in <module>
        raise RuntimeError("Numpy required but not found.")
    RuntimeError: Numpy required but not found.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2Mcziv/cogent

Any clues? Must be my environment, but I'd like to give it a shot 'myself' before I bother the HPC support people.

Thanks,
Michael

Colin Brislawn

unread,
Feb 16, 2016, 2:50:26 PM2/16/16
to Qiime 1 Forum
Hello Michael,

I think trying a virtual environment is the right approach, but the one that comes with python is a bit week. Have you tried out Anaconda? I use that deploy qiime on several different clusters with great success (no sudo needed).

Install Miniconda from here: http://conda.pydata.org/miniconda.html

Let me know how this works. (This is going to be the default way to install qiime 2 so your feedback is appreciated.)
Colin Brislawn

Michael Baron

unread,
Feb 16, 2016, 4:36:04 PM2/16/16
to qiime...@googlegroups.com
Hi Colin,

Thanks for helping me out (again)! :)

The Anaconda installation went without a hitch (just wget'ed the .sh file and executed it). However, your command in the gist throws an error:

conda -n create qiime191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata h5py
usage
: conda [-h] [-V] [--debug] command ...
conda
: error: unrecognized arguments: -n

I guess conda-env is the equivalent to virtualenv. And from a quick google search 'conda create env_name env_packages' would set it up. Is the -n tag deprecated?

Thanks,
Michael

edit: I see, it was just an order issue. -n stand for name.

Michael Baron

unread,
Feb 16, 2016, 4:54:25 PM2/16/16
to Qiime 1 Forum
A short follow up. I ran the following code:

conda create -n qiime191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata h5py
source activate qiime191
pip install https
://github.com/biocore/qiime/archive/1.9.1.tar.gz

And it seems the installation went through:

     | 20.3MB 10.4MB/s
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.14.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)
Collecting cogent==1.5.3 (from qiime==1.9.1)
  Using cached cogent-1.5.3.tgz
Collecting natsort<4.0.0 (from qiime==1.9.1)
  Using cached natsort-3.5.6-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): matplotlib!=1.4.2,>=1.1.0 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)
Collecting pynast==1.2.2 (from qiime==1.9.1)
  Using cached pynast-1.2.2.tar.gz
Collecting qcli<0.2.0,>=0.1.1 (from qiime==1.9.1)
  Using cached qcli-0.1.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): gdata in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)
Collecting biom-format<2.2.0,>=2.1.4 (from qiime==1.9.1)
  Using cached biom-format-2.1.5.tar.gz
Collecting emperor<1.0.0,>=0.9.51 (from qiime==1.9.1)
  Using cached emperor-0.9.51.tar.gz
Collecting scikit-bio<0.3.0,>=0.2.3 (from qiime==1.9.1)
  Using cached scikit-bio-0.2.3.tar.gz
Collecting burrito-fillings<0.2.0,>=0.1.1 (from qiime==1.9.1)
  Using cached burrito-fillings-0.1.1.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pandas>=0.13.1 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from qiime==1.9.1)
Collecting burrito<1.0.0,>=0.9.1 (from qiime==1.9.1)
  Using cached burrito-0.9.1.tar.gz
Collecting qiime-default-reference<0.2.0,>=0.1.2 (from qiime==1.9.1)
  Using cached qiime-default-reference-0.1.3.tar.gz
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)
Requirement already satisfied (use --upgrade to upgrade): pytz in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)
Requirement already satisfied (use --upgrade to upgrade): cycler in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)
Requirement already satisfied (use --upgrade to upgrade): pyparsing!=2.0.4,>=1.5.6 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from matplotlib!=1.4.2,>=1.1.0->qiime==1.9.1)
Collecting click (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)
  Using cached click-6.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): future>=0.14.3 in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)
Collecting pyqi (from biom-format<2.2.0,>=2.1.4->qiime==1.9.1)
  Using cached pyqi-0.3.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): six in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from scikit-bio<0.3.0,>=0.2.3->qiime==1.9.1)
Requirement already satisfied (use --upgrade to upgrade): IPython in ./miniconda2/envs/qiime191/lib/python2.7/site-packages (from scikit-bio<0.3.0,>=0.2.3->qiime==1.9.1)
Building wheels for collected packages: cogent, pynast, qcli, biom-format, emperor, scikit-bio, burrito-fillings, burrito, qiime-default-reference, pyqi
  Running setup.py bdist_wheel for cogent ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/31/4a/1e/5e89366707d4ed8d510521c6185662503e6098f2234721ecf4
  Running setup.py bdist_wheel for pynast ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/f5/c6/15/140e55821648e3edbdf539bdcd13973b72eabf7a6037f7ba17
  Running setup.py bdist_wheel for qcli ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/fd/59/31/4c48e4ecfb3ad3c5cc98614ddf92d332b24431dc9afd64644b
  Running setup.py bdist_wheel for biom-format ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/5a/e4/11/ef896761f31e4d9a8a6a5216136d46634ae436f5da3fd5dad3
  Running setup.py bdist_wheel for emperor ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/41/12/db/a1085947c85572878b0b7c6ea18f4c62110ad2eee1af89d80f
  Running setup.py bdist_wheel for scikit-bio ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/cf/81/f7/5f307d109d3dfb5ce16d112541407b569197373abb386a7e5b
  Running setup.py bdist_wheel for burrito-fillings ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/4e/a7/d5/3995f264f46c3bedb065384beafcbec3b2a1b893c1afa6ba21
  Running setup.py bdist_wheel for burrito ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/04/6e/03/b773992d32f3761527a3f093f1857f86b27f83076f82084fa7
  Running setup.py bdist_wheel for qiime-default-reference ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/b7/ce/c9/9b7a13ec1c9a4fc7faf3da0921d4cf94eee040173c1f9fc8f0
  Running setup.py bdist_wheel for pyqi ... done
  Stored in directory: /home/sejj036/.cache/pip/wheels/9e/d2/3a/f21057d3aed9a859cead95630f30de1144b4235ec36884d188
Successfully built cogent pynast qcli biom-format emperor scikit-bio burrito-fillings burrito qiime-default-reference pyqi
Installing collected packages: cogent, natsort, pynast, qcli, click, pyqi, biom-format, scikit-bio, emperor, burrito, burrito-fillings, qiime-default-reference, qiime
  Found existing installation: natsort 4.0.3
    Uninstalling natsort-4.0.3:
      Successfully uninstalled natsort-4.0.3
  Found existing installation: scikit-bio 0.4.0
    Uninstalling scikit-bio-0.4.0:
      Successfully uninstalled scikit-bio-0.4.0
  Running setup.py install for qiime ... done
Successfully installed biom-format-2.1.5 burrito-0.9.1 burrito-fillings-0.1.1 click-6.2 cogent-1.5.3 emperor-0.9.51 natsort-3.5.6 pynast-1.2.2 pyqi-0.3.2 qcli-0.1.1 qiime-1.9.1 qiime-default-reference-0.1.3 scikit-bio-0.2.3

When I try to print the qiime config to check the installation, it seems to get its knickers into a twist:

print_qiime_config.py
Traceback (most recent call last):
  File "/home/sejj036/miniconda2/envs/qiime191/bin/print_qiime_config.py", line 40, in <module>
    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))
ImportError: /home/sejj036/miniconda2/envs/qiime191/lib/python2.7/site-packages/skbio/alignment/_ssw_wrapper.so: undefined symbol: _intel_fast_memcpy
See the QIIME Installation Guide: http://qiime.org/install/install.html

Did I pick the wrong compiler? I vaguely remember there was a caveat somewhere in the HPC documentation that I should pick the GCC, but that would imply C-code...

Thanks,
Michael

Colin Brislawn

unread,
Feb 16, 2016, 4:54:46 PM2/16/16
to Qiime 1 Forum
Good catch. I've edited my comment. 

Colin

Colin Brislawn

unread,
Feb 16, 2016, 5:19:40 PM2/16/16
to qiime...@googlegroups.com
That's odd... I'm not totally sure what would be causing that. What OS is on your supercomputer? For example, when I run 'cat /etc/issue' it tells me:
Red Hat Enterprise Linux Workstation release 6.6 (Santiago)


At one point, I remember having to manually load a gcc compiler to make some python software happy. You could try the same to see if it helps. I ran something like 'module load gcc' then tried running python again.

Maybe a qiime dev can help more...
Colin

Michael Baron

unread,
Feb 17, 2016, 3:44:53 AM2/17/16
to Qiime 1 Forum
It runs Red Hat as well.

And as it turns out it likely has been the compiler. This is from our HPC documentation:

Set compiler module
The Python versions on Legion were built with GCC. You can run them with the default Intel compilers loaded because everything depends on the gcc-libs/4.9.2 module. When you are building your own Python packages you should have the GCC compiler module loaded however, to avoid the situation where you build a package with the Intel compiler and then try to run it with GCC, in which case it will be unable to find Intel-specific instructions.
module unload compilers
module load compilers/gnu/4.9.2
If you get an error like this when trying to run something, recheck what compiler you used.
undefined symbol: __intel_sse2_strrchr

I'll rebuild the virtual environment later today and report back.

Lesson learned: double check the documentation!

Thanks again, Colin!

Michael Baron

unread,
Feb 17, 2016, 10:14:53 AM2/17/16
to Qiime 1 Forum
Alas, rebuilding the environment with the GNU compiler, or even reinstalling miniconda did not help.

I'll have to get in touch with our HPC support, however, if someone else has any insight of what is failing, I'd be grateful.

Thanks,
Michael

Michael Baron

unread,
Feb 17, 2016, 2:12:10 PM2/17/16
to Qiime 1 Forum
Caching was the culprit. Once I cleared out all the previously compiled python modules (~./cache/), it went through without a hitch.

I even got it working in the standard python virtualenv (the --no-cache-dir is a little redundant):

module unload compilers
module load compilers/gnu/4.9.2

virtualenv qiime191
source qiime191
/bin/activate
pip install
--no-cache-dir numpy
# paths to blas and lapack
export BLAS=/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/libblas.so
export LAPACK=/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/liblapack.so
pip install
--no-cache-dir scipy
pip install https
://github.com/biocore/qiime/archive/1.9.1.tar.gz --no-cache-dir

Thus always make sure the correct compiler is selected.

Colin Brislawn

unread,
Feb 17, 2016, 6:22:54 PM2/17/16
to Qiime 1 Forum
Very good troubleshooting on your part. Glad you figured that out!
Let us know if you have any more questions,
Colin
Reply all
Reply to author
Forward
0 new messages