Installation on hpc:

[Michael Davies]

Hi,

I am trying to install MDAnalysis on a hpc but I am getting an error when I try to import it.

Python 2.7.5

Installed via:

pip install --user --upgrade MDAnalysis

Version: MDAnalysis 0.19.2

Error when import:

>>> import MDAnalysis

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/transformations.py:1813: UserWarning: failed to import module _transformations

  warnings.warn("failed to import module " + module_name)

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/__init__.py", line 183, in <module>

    from .lib import log

  File "/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/__init__.py", line 38, in <module>

    from . import distances  # distances relies on mdamath

  File "/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/distances.py", line 85, in <module>

    package="MDAnalysis.lib")

  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module

    __import__(name)

ImportError: /home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/c_distances.so: undefined symbol: __svml_round2_mask

I then ran the installation test and saw that this gives many errors.

Installation test:

pytest --disable-pytest-warnings --pyargs MDAnalysisTests

Output of this is in the log file attached. Basically two errors with the second repeated:

1) ImportError: /home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/c_distances.so: undefined symbol: __svml_round2_mask

2) ImportError: cannot import name log

I'm not sure how to fix this and any help would be greatly appreciated.

Thank you

Michael

[Oliver Beckstein]

Hi Michael,

welcome to the MDAnalysis list!

I think there are issues with numpy and the math libraries. Did you install numpy and scipy yourself? Are there numpy/scipy versions in .local that collide with globally installed ones? 

You are also using a terribly outdated Python version (presumably on a cluster that hasn’t beed updated in ages). Numpy and Scipy stopped supporting Python 2.7 and MDAnalysis (with the rest of the Python eco system will stop supporting 2.7 in 2020).

When we use MDA on HPC we just install a fresh conda environment to get rid of all the outdated cluster software, see https://www.mdanalysis.org/pages/installation_quick_start/#conda and https://www.mdanalysis.org/2017/04/07/environments/#managing-software-versioning-using-conda-environments . You could probably debug your Python installation and ultimately make it work but unless you have a really good reason why you absolutely need to use the cluster versions, I would not do it and instead use conda. If you REALLY want to use your cluster, I would use virtual environments to separate cluster packages from installed packages, see https://www.mdanalysis.org/2017/04/07/environments/#python-virtual-environments

Oliver

--
You received this message because you are subscribed to the Google Groups "MDnalysis discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mdnalysis-discus...@googlegroups.com.
To post to this group, send email to mdnalysis-...@googlegroups.com.
Visit this group at https://groups.google.com/group/mdnalysis-discussion.
To view this discussion on the web visit https://groups.google.com/d/msgid/mdnalysis-discussion/f9352005-418e-451c-8b32-96369f99d7c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<log.txt>

--

Oliver Beckstein

orbe...@gmail.com * orbe...@gmx.net

[Michael Davies]

Hi Oliver,

Thanks for your help.

I installed python, numpy and scripy myself locally, using pip install --user. My knowledge on this is a little weak, but I imagine "global" versions on a HPC would be the modules I have loaded? I don't think there will be a conflict between local and globally installed versions as I don't load any python modules, and I can't install globally on the HPC because I am not root and I don't think there is a default python that is installed globally on the HPC.

Anyway, I have now installed conda as you suggested and created an environment called "mdaenv" however I get the same error message on import and the same errors when I run the test. I'm new to conda so maybe there is a mistake in how I made it:

conda config --add channels conda-forge

conda update --yes conda

#create python 2.7.16 env called "mdaenv" with all the packages I need

conda create --yes -n mdaenv python=2.7.16 joblib ase numpy scipy matplotlib ipython jupyter pandas sympy nose MDAnalysis MDAnalysisTests

#activate the environment

source activate mdaenv

#try to import

(mdaenv) [zcapmbd@login01 ~]$ ipython

Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 21:00:58) 

Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.

?         -> Introduction and overview of IPython's features.

%quickref -> Quick reference.

help      -> Python's own help system.

object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import MDAnalysis

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/transformations.py:1813: UserWarning: failed to import module _transformations

  warnings.warn("failed to import module " + module_name)

---------------------------------------------------------------------------

ImportError                               Traceback (most recent call last)

<ipython-input-1-4c4262a6b4c9> in <module>()

----> 1 import MDAnalysis

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/__init__.py in <module>()

    181 )

    182 

--> 183 from .lib import log

    184 from .lib.log import start_logging, stop_logging

    185 

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/__init__.py in <module>()

     36 from . import util

     37 from . import mdamath

---> 38 from . import distances  # distances relies on mdamath

     39 from . import NeighborSearch

     40 from . import formats

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/distances.py in <module>()

     83 _distances = {}

     84 _distances['serial'] = importlib.import_module(".c_distances",

---> 85                                          package="MDAnalysis.lib")

     86 try:

     87     _distances['openmp'] = importlib.import_module(".c_distances_openmp",

/home/zcapmbd/anaconda2/envs/mdaenv/lib/python2.7/importlib/__init__.pyc in import_module(name, package)

     35             level += 1

     36         name = _resolve_name(name[level:], package, level)

---> 37     __import__(name)

     38     return sys.modules[name]

ImportError: /home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/c_distances.so: undefined symbol: __svml_round2_mask

I have attached the environment I created in case this could be of help. Do you know what could be going wrong here?

Also, I am using python 2.7 still as I simply haven't yet made the swap to python 3. I find numpy, scipy, and MDAnalysis work perfectly for me on my desktop with python 2 (using the versions that support it). I don't wish to switch to python 3 as I've now written a large python 2 script that I want to run on the HPC.

Michael

[Farshad Saberi]

Hi,

I used to have similar issues. I recommend that you install your python in a virtual environment at HPC. Then, install MDAnalysis in your virtual environment. Just make sure that you activate the virtual environment before installing MDAnalysis.

Cheers,

Farshad

--

You received this message because you are subscribed to the Google Groups "MDnalysis discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mdnalysis-discus...@googlegroups.com.
To post to this group, send email to mdnalysis-...@googlegroups.com.
Visit this group at https://groups.google.com/group/mdnalysis-discussion.

To view this discussion on the web visit https://groups.google.com/d/msgid/mdnalysis-discussion/0e754df1-2343-4005-b9db-f885c0b8b3aa%40googlegroups.com.

[Oliver Beckstein]

Hi Michael,

Also, I am using python 2.7 still as I simply haven't yet made the swap to python 3. I find numpy, scipy, and MDAnalysis work perfectly for me on my desktop with python 2 (using the versions that support it). I don't wish to switch to python 3 as I've now written a large python 2 script that I want to run on the HPC.

I totally understand. This is why we support 2.7 for as long as we can.

Your error is almost certainly a linking problem so there is somehow the wrong library in your LD_LIBRARY_PATH. Apparently, the error came up before https://github.com/MDAnalysis/mdanalysis/issues/1350 but we didn’t get enough feedback to resolve it.

I followed your exact instructions on one of our local clusters and have no problems installing and running MDAnalysis.

What does 

module list

show? 

Do you still have a ~/.local directory? If yes, I would rename it for the time being

mv ~/.local ~/.local_DEACTIVATED

I think you are still wrongly importing from .local: the trace shows 

/home/zcapmbd/.local/lib/python2.7/site-packages/MDAnalysis/lib/distances.py

and this should instead show a path inside mdaenv.

Also make sure that ipython (and Jupyter) are installed *inside* the mdaenv environment. If not they can pick up the wrong modules. If in doubt try just

which python   # should show a path inside mdaenv

python

and try importing inside the normal interpreter.

If this does not work, let me know.

Oliver

[Michael Davies]

Hi Oliver and Farshad,

Thanks for your suggestions.

It was indeed as you said Oliver, it was still accessing the MDAnalysis I had in the ~/.local whilst I was in the virtual environment.

I removed this via pip uninstall MDAnalysis and pip uninstall MDAnalysisTests whilst outside of my virtual environment. This removed them from ~/.local and it now correctly follows the path to the MDAnalysis stored in my virtual environment, as well as python, ipython, numpy etc. The test now completes fine and I can import MDAnalysis in the python interpreter and use it for what I need. So it works!

Thanks very much for your help! It's greatly appreciated.

Michael

[Richard Gowers]

For what it’s worth, pip local is evil for this reason :)

--
You received this message because you are subscribed to the Google Groups "MDnalysis discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mdnalysis-discus...@googlegroups.com.
To post to this group, send email to mdnalysis-...@googlegroups.com.
Visit this group at https://groups.google.com/group/mdnalysis-discussion.

To view this discussion on the web visit https://groups.google.com/d/msgid/mdnalysis-discussion/8134a528-8a17-4d6d-81de-8c2f13f07577%40googlegroups.com.

[Tyler Reddy]

SciPy 1.2.x (LTS) supports 2.7 until 2020

Likewise for NumPy 1.16.x (LTS) series

To view this discussion on the web visit https://groups.google.com/d/msgid/mdnalysis-discussion/29a6b1b8-a9a7-43f9-8508-9ac899c626e4%40Richards-iPhone.