QIIME install with Conda on a CentOS supercomputer

[Emma Dietrich]

Hi all,

I am attempting to install QIIME in a miniconda environment on a CentOS supercomputer using the instructions posted here:

https://github.com/biocore/qiime/issues/1759#issuecomment-88633371

I'm new-ish to Linux so it took me a bit to figure out how to set up the miniconda environment, but I finally got that figured out. Anyway, I finally have the environment created and running and was able to install qiime (I had some errors with gcc not being loaded and my PYTHONPATH pointing to the wrong places but those are taken care of now).

When I run:

(QIIME191)tacc:/work/03071/dietrich$ pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz

I get no errors, seems like everything installs fine. However, when I run:

(QIIME191)tacc:/work/03071/dietrich$ print_qiime_config.py

I get the following:

Traceback (most recent call last):

  File "/work/03071/dietrich/miniconda2/envs/QIIME191/bin/print_qiime_config.py", line 40, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: cannot import name solve

It's not exactly clear to me what this means, but googling around it seems to be a common error when things are installed incorrectly, and/or packages are not the right version, etc. Kind of a multitude of possible problems that it could indicate. 

Any idea of how to troubleshoot this? 

Here is a list of package versions: 

(QIIME191)tacc:/work/03071/dietrich$ pip freeze > pkgs.txt

(QIIME191)tacc:/work/03071/dietrich$ less pkgs.txt

biom-format==2.1.5

burrito==0.9.1

burrito-fillings==0.1.1

bz2file==0.98

CacheControl==0.11.5

click==6.2

cogent==1.5.3

contextlib2==0.4.0

cycler==0.9.0

decorator==4.0.6

emperor==0.9.51

future==0.15.2

gdata==2.0.18

ipython==4.0.3

ipython-genutils==0.1.0

lockfile==0.10.2

matplotlib==1.5.1

natsort==3.5.6

nose==1.3.7

numpy==1.10.2

pandas==0.17.1

path.py==0.0.0

pexpect==3.3

pickleshare==0.5

pycairo==1.10.0

pynast==1.2.2

pyparsing==2.0.3

pyqi==0.3.2

python-dateutil==2.4.2

pytz==2015.7

qcli==0.1.1

qiime==1.9.1

qiime-default-reference==0.1.3

requests==2.9.1

scikit-bio==0.2.3

scipy==0.16.1

simplegeneric==0.8.1

six==1.10.0

traitlets==4.1.0

wheel==0.26.0

[Daniel McDonald]

Hi Emma,

Can you provide the list of commands you used to setup the environment? What's worked for me on a variety of Linux clusters is the following:

$ conda create --name=qiime191 python=2

$ conda install numpy scipy jupyter matplotlib

$ pip install qiime

QIIME currently is not in conda, but it is in pypi which is why you shouldn't need to specify the github URL.

What I suspect is that conda and pip are battling each other -- sometimes they don't play well together. 

Let me know how it goes!

Best,

Daniel 

[Emma Dietrich]

Here are the commands I used to create the environment:

tacc:/work/03071/dietrich$ conda create -n QIIME191 pip numpy matplotlib IPython future natsort scipy pandas scikit-bio gdata 

tacc:/work/03071/dietrich$ source activate QIIME191

Then I have to reset my PYTHONPATH because otherwise I get errors with numpy in the qiime install:

(QIIME191)tacc:/work/03071/dietrich$ unset PYTHONPATH

(QIIME191)tacc:/work/03071/dietrich$ export PYTHONPATH=/work/03071/dietrich/miniconda2/envs/QIIME191/lib:$PYTHONPATH

(QIIME191)tacc:/work/03071/dietrich$ pip install https://github.com/biocore/qiime/archive/1.9.1.tar.gz

No obvious errors here though. Only when I go to print the qiime config information does it give me an error (as posted above).

I'll create a new environment using your advice and let you know how that goes! Thanks!!!

[Emma Dietrich]

OK, so I created a new environment using your advice and still get an error with print_qiime_config.py.

So here's what I did (pretty much as you said, but I activated the environment and installed stuff in there - was that wrong to do? I wasn't sure why I would make a new environment otherwise):

tacc:/work/03071/dietrich$ conda create --name=qiime191 python=2

tacc:/work/03071/dietrich$ source activate qiime191

(qiime191)tacc:/work/03071/dietrich$ conda install numpy scipy jupyter matplotlib

(qiime191)tacc:/work/03071/dietrich$ pip install qiime

Now I get a new error with print_qiime_config.py --- Haven't googled around much yet to see what it means but will do so now!

(qiime191)tacc:/work/03071/dietrich$ print_qiime_config.py 

Traceback (most recent call last):

  File "/work/03071/dietrich/miniconda2/envs/qiime191/bin/print_qiime_config.py", line 28, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: /corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString

See the QIIME Installation Guide: http://qiime.org/install/install.html

[Daniel McDonald]

Hi Emma,

Sorry for the delayed response. On your initial install, I strongly suspect the issue is associated with modifying PYTHONPATH -- when using conda, you really should not have to do that.

With the second install, the PYTHONPATH appears to also be wrong. Notice that the path associated with the library is not part of your environment, but is external to it. From within the qiime191 environment, can you run the following please?

python -c "import numpy; print(numpy.__file__); print(numpy.__version__)"

And just to check, are there any "module load" or dotkit statements in your bash_profile or bashrc? 

Best,

Daniel

[Emma Dietrich]

Hi Daniel,

I think you were right, my PYTHONPATH got super messed up when trying to create the conda environment, and I never fixed it entirely. 

Our university actually just updated its systems on our supercomputers, and I finally transitioned my account to the new system. Not sure what the specific differences are between the systems but I went to install qiime as you said and the install worked perfectly! Probably a combination of updated packages, and a less messy PYTHONPATH... So, I'm hoping that's the last hurdle I'll face (hah....). Thanks again for you help! 

[Emma Dietrich]

Hmm ok so for some reason when I logged into my new qiime environment (which was working fine yesterday, for the hour or so I tried it out), I now get the same error as I did previously. For example:

tacc:~$ source activate qiime191

discarding /work/03071/dietrich/miniconda2/bin from PATH

prepending /work/03071/dietrich/miniconda2/envs/qiime191/bin to PATH

(qiime191)tacc:~$ print_qiime_config.py 

Traceback (most recent call last):

  File "/work/03071/dietrich/miniconda2/envs/qiime191/bin/print_qiime_config.py", line 28, in <module>

    raise ImportError("%s\n%s" % (e, core_dependency_missing_msg))

ImportError: /corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString

See the QIIME Installation Guide: http://qiime.org/install/install.html

This install (on a whole new computer, essentially), I did not change PYTHONPATH, or anything. But, as you said, it seems to be point to the wrong Python. I checked my .bashrc and no module load or dot kit statements are present.

I run the import line and I get this error:

(qiime191)tacc:~$ python -c "import numpy; print(numpy.__file__); print(numpy.__version__)"

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File "/corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module>

    from . import add_newdocs

  File "/corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module>

    from numpy.lib import add_newdoc

  File "/corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module>

    from .type_check import *

  File "/corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module>

    import numpy.core.numeric as _nx

  File "/corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module>

    from . import multiarray

ImportError: /corral-repl/utexas/BioITeam/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsASCIIString

I'm not sure why the PYTHONPATH points outside of the environment. I redid everything in a clean environment and am getting the same exact errors, so I have no idea what has changed from yesterday, when it was working, to today. 

[Daniel McDonald]

Hi Emma,

That is frustrating. When you have a moment, could you send the output from the following commands?

# from outside of your qiime191 environment

$ which python

$ echo $PYTHONPATH

$ echo $PATH

# from within your qiime191 environment

$ which python

$ echo $PYTHONPATH

$ echo $PATH

Best,

Daniel

[Emma Dietrich]

Here you go!

From outside the environment:

tacc:/work/03071/dietrich/lonestar$ which python

/opt/apps/intel16/python/2.7.10/bin/python

tacc:/work/03071/dietrich/lonestar$ echo $PYTHONPATH

/opt/apps/intel16/cray_mpich_7_3/python/2.7.10/lib/python2.7/site-packages:/corral-repl/utexas/BioITeam/python2.7/site-packages:/corral-repl/utexas/BioITeam/lib/python2.7/site-packages:/work/03071/dietrich/lonestar/python/lib/python2.7/site-packages

tacc:/work/03071/dietrich/lonestar$ echo $PATH

/opt/apps/tacc/bin:/opt/apps/intel16/cray_mpich_7_3/python/2.7.10/lib/python2.7/site-packages/mpi4py/bin:/opt/apps/intel16/python/2.7.10/bin:/corral-repl/utexas/BioITeam/bin:/home1/03071/dietrich/local/bin:/corral-repl/utexas/BioITeam/breseq/bin:/work/03071/dietrich/miniconda2/bin:/opt/apps/intel16/cray_mpich/7.3.0/bin:/opt/cray/mpt/default/gni/bin:/opt/apps/intel/16.0.1.150/compilers_and_libraries_2016.1.150/linux/bin/intel64:/opt/apps/gcc/4.9.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:.:/opt/slurm/default/bin

From inside the environment:

tacc:/work/03071/dietrich/lonestar$ source activate qiime191

discarding /work/03071/dietrich/miniconda2/bin from PATH

prepending /work/03071/dietrich/miniconda2/envs/qiime191/bin to PATH

(qiime191)tacc:/work/03071/dietrich/lonestar$ which python

/work/03071/dietrich/miniconda2/envs/qiime191/bin/python

(qiime191)tacc:/work/03071/dietrich/lonestar$ echo $PATH

/work/03071/dietrich/miniconda2/envs/qiime191/bin:/opt/apps/tacc/bin:/opt/apps/intel16/cray_mpich_7_3/python/2.7.10/lib/python2.7/site-packages/mpi4py/bin:/opt/apps/intel16/python/2.7.10/bin:/corral-repl/utexas/BioITeam/bin:/home1/03071/dietrich/local/bin:/corral-repl/utexas/BioITeam/breseq/bin:/opt/apps/intel16/cray_mpich/7.3.0/bin:/opt/cray/mpt/default/gni/bin:/opt/apps/intel/16.0.1.150/compilers_and_libraries_2016.1.150/linux/bin/intel64:/opt/apps/gcc/4.9.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/lib/qt3/bin:.:/opt/slurm/default/bin

(qiime191)tacc:/work/03071/dietrich/lonestar$ echo $PYTHONPATH

/opt/apps/intel16/cray_mpich_7_3/python/2.7.10/lib/python2.7/site-packages:/corral-repl/utexas/BioITeam/python2.7/site-packages:/corral-repl/utexas/BioITeam/lib/python2.7/site-packages:/work/03071/dietrich/lonestar/python/lib/python2.7/site-packages

So, even though the path points within the environment, pythonpath does not. This is the same problem I was dealing with previously, which is why I tried to change the pythonpath to point within the environment, but then that give me a different sort of error. 

[Daniel McDonald]

I think I know what's going on. Conda doesn't modify the $PYTHONPATH normally as it doesn't need to: python is installed within the environment already. Can you try "unset PYTHONPATH" either prior to or following the activation of the qiime191 environment, and then attempt running "print_qiime_config.py"? 

If this solves the problem, then I suspect there may be entries setting your PYTHONPATH inside of either ~/.bashrc or ~/.bash_profile?

Best,

Daniel 

[Emma Dietrich]

Unsetting PYTHONPATH worked! I guess I never tried just doing unset PYTHONPATH, I thought you always had to set it again or something. 

I really don't see anything in my ~/.bashrc or ~/.profile  (I don't have a file called ~/.bash_profile..... is there a difference?)

[Daniel McDonald]

I don't recall the specific differences between ~/.profile and ~/.bash_profile right now but I think has to do with whether its actually a login shell or not. I suspect there's something at a system level setting the python path for all users. It might be worth contacting the administrators to inquire? Or, just forget and qiime on :)

Glad to hear things are working now!

Best,

Daniel

[Emma Dietrich]

So far so good, as long as I just unset PYTHONPATH after I activate the environment! Thanks for all your help Daniel. 

For anyone else that ever uses this thread in the future:
The only other glitch I have experienced since getting print_qiime_config.py to work, was that I needed to download 1) fastq-join (in ea-utils) and 2) Seqprep in order to get multiple_join_paired_ends.py to work. There's lots of advice around this thread/the net on how to download and install each of these, but mostly, don't forgot to add them to you PATH once you've downloaded them! 

Since then I have successfully run:

multiple_join_paired_ends.py

multiple_extract_barcodes.py

multiple_split_libraries_fastq.py

pick_open_reference_otus,py (with enable_rev_strand_match; this worked on a test data set, currently running it on my full dataset!)

[Colin Brislawn]

I'm glad you got it working Emma!

Historically, qiime has been pretty hard to install. We are working to improve it with easier to install software packages. Adding ea-utils, one of the qiime defaults, would be a great piece of software to add! 

Colin