Correct way to install nomkl version of cntk?

0 views

Skip to first unread message

Geoffrey Anderson

unread,

Jan 2, 2018, 12:55:08 PM1/2/18

to conda - Public

What is the correct way to install the nomkl version of cntk? At the bottom I put up several possible hypotheses. Perhaps one of them is close to correct.

I prefer to only correct the underlying cause of the crashing of cntk at this time, and accordingly, only change things that would affect cntk. I prefer to avoid if possible global python changes on this host, but will do it anyway if what I prefer is not actually possible. I prefer NOT to use env var KMP_DUPLICATE_LIB_OK=TRUE. I prefer to usually use OpenBLAS or LIBNVBLAS on this host whenever a math library is allowed because I have much excellent prior experience using OpenBLAS for the CPUs and nvidia's BLAS for the GPU. I use them both successfully. I do not want Intel MKS.

WHY I WANT TO DO THIS:

CNTK is the microsoft deep learning tool kit, and as well the nomkl version of certain other packages going forward, given that I have a system already installed with lots of conda packages apparently referencing mkl. Improper double initialization of OpenMP library libiomp5.so which always results in a crash is why I feel I need the nomkl version of cntk.

In the terminal where I started jupyter notebook server:

[I 12:22:44.489 NotebookApp] KernelRestarter: restarting kernel (1/5)

WARNING:root:kernel -91cb-47e5-bed7-2406fd30c93d restarted

[W 12:24:18.246 NotebookApp] Notebook CNTK_101_LogisticRegression.ipynb is not trusted

Selected GPU[0] GeForce GTX 1080 as the process wide default device.

OMP: Error #15: Initializing libiomp5.so, but found libiomp5.so already initialized.

OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

[I 12:24:50.497 NotebookApp] KernelRestarter: restarting kernel (1/5)

SEVERAL DIFFERENT WAYS THAT MIGHT BE THE CORRECT WAY:

Maybe the correct fix is to run:

conda install -c anaconda nomkl

As referenced here: https://anaconda.org/anaconda/nomkl

Or else myabe the corect fix is to run:

conda install nomkl

As referenced here: https://github.com/BVLC/caffe/issues/3884 by wgong Aug 14, 2017

And then after choosing 1 of the 2 above commands, maybe the correct next steps would be to change only the installed CNTK to be the nomkl version:

(How?)

Or else maybe the correct fix is to change all my installed packages on the entire host, to be the nomkl version:

(How?)

Or else maybe everything for all the installed packages will have been already fixed by just the command conda install nomkl, hypothetically.

Or else maybe conda install -c anaconda nomkl should not be used at all, and neither should conda install nomkl and instead, just remove the mkl version of of CNTK, and then install the nomkl version of CNTK.

Please Advise. I wish to avoid damaging and redownloading and reinstalling the entire python system and installed packages unless it is necessary. Currently the only crashing is by all those python notebooks which use libraries Tensorflow or CNTK and possibly numpy together. Numpy alone never crashes.

Thank you for reading.

My current environment is shown below.

$ conda info

Current conda install:

platform : linux-64

conda version : 4.3.30

conda is private : False

conda-env version : 4.3.30

conda-build version : 2.1.8

python version : 3.5.4.final.0

requests version : 2.13.0

root environment : /home/ga/anaconda3 (writable)

default environment : /home/ga/anaconda3

envs directories : /home/ga/anaconda3/envs

/home/ga/.conda/envs

package cache : /home/ga/anaconda3/pkgs

/home/ga/.conda/pkgs

channel URLs : https://conda.anaconda.org/bioconda/linux-64

https://conda.anaconda.org/bioconda/noarch

https://repo.continuum.io/pkgs/main/linux-64

https://repo.continuum.io/pkgs/main/noarch

https://repo.continuum.io/pkgs/free/linux-64

https://repo.continuum.io/pkgs/free/noarch

https://repo.continuum.io/pkgs/r/linux-64

https://repo.continuum.io/pkgs/r/noarch

https://repo.continuum.io/pkgs/pro/linux-64

https://repo.continuum.io/pkgs/pro/noarch

config file : /home/ga/.condarc

netrc file : None

offline mode : False

user-agent : conda/4.3.30 requests/2.13.0 CPython/3.5.4 Linux/4.4.0-104-generic debian/stretch/sid glibc/2.23

UID:GID : 1000:1000

$ conda list

# packages in environment at /home/ga/anaconda3:

_license 1.1 py35_1

_nb_ext_conf 0.3.0 py35_0

alabaster 0.7.10 py35_0

anaconda custom py35_0

anaconda-clean 1.1.0 py35_0

anaconda-client 1.6.2 py35_0

anaconda-navigator 1.5.1 py35_0

anaconda-project 0.4.1 py35_0

argcomplete 1.0.0 py35_1

astroid 1.4.9 py35_0

astropy 2.0.2 py35h2d2a8a6_4

babel 2.4.0 py35_0

backports 1.0 py35_0

backports.weakref 1.0rc1 py35_0

beautifulsoup4 4.5.3 py35_0

bitarray 0.8.1 py35_0

blaze 0.10.1 py35_0

bleach 1.5.0 py35_0

bokeh 0.12.4 py35_0

boto 2.46.1 py35_0

bottleneck 1.2.1 py35he1b16f1_0

bzip2 1.0.6 3

ca-certificates 2017.08.26 h1d4fec5_0

cairo 1.14.8 0

cffi 1.9.1 py35_0

chainer 3.2.0 <pip>

chardet 2.3.0 py35_0

chest 0.2.3 py35_0

click 6.7 py35_0

cloudpickle 0.2.2 py35_0

clyent 1.2.2 py35_0

cntk 2.3.1 <pip>

colorama 0.3.7 py35_0

conda 4.3.30 py35hf9359ed_0

conda-build 2.1.8 py35_0

conda-env 2.6.0 h36134e3_1

conda-verify 2.0.0 py35_0

configobj 5.0.6 py35_0

contextlib2 0.5.4 py35_0

cryptography 1.7.1 py35_0

curl 7.52.1 0

cycler 0.10.0 py35_0

cython 0.27.3 py35h6cdc64b_0

cytoolz 0.8.2 py35_0

dask 0.16.0 py35hcb8ecc8_0

dask-core 0.16.0 py35hfc66869_0

datashape 0.5.4 py35_0

dbus 1.10.10 0

decorator 4.0.11 py35_0

dill 0.2.5 py35_0

distributed 1.20.2 py35_0

docutils 0.13.1 py35_0

dynd-python 0.7.2 py35_0

entrypoints 0.2.2 py35_1

et_xmlfile 1.0.1 py35_0

expat 2.1.0 0

fastcache 1.0.2 py35_1

filelock 2.0.7 py35_0

flask 0.12 py35_0

flask-cors 3.0.2 py35_0

fontconfig 2.12.1 3

freetype 2.5.5 2

future 0.16.0 <pip>

get_terminal_size 1.0.0 py35_0

gevent 1.2.1 py35_0

glib 2.50.2 1

gmp 6.1.0 0

greenlet 0.4.12 py35_0

gsl 2.2.1 0

gst-plugins-base 1.8.0 0

gstreamer 1.8.0 0

gym 0.9.4 <pip>

h5py 2.7.0 np113py35_0

harfbuzz 0.9.39 2

hdf5 1.8.17 1

heapdict 1.0.0 py35_1

html5lib 0.9999999 py35_0

icu 54.1 0

idna 2.2 py35_0

imageio 2.2.0 py35hd0a6de2_0

imagesize 0.7.1 py35_0

intel-openmp 2018.0.0 hc7b2577_8

ipykernel 4.5.2 py35_0

ipython 6.2.1 py35hd850d2a_1

ipython_genutils 0.2.0 py35_0

ipywidgets 6.0.0 py35_0

isort 4.2.5 py35_0

itsdangerous 0.24 py35_0

jbig 2.1 0

jdcal 1.3 py35_0

jedi 0.11.0 py35_2

jinja2 2.9.5 py35_0

jpeg 9b 0

jsonschema 2.5.1 py35_0

jupyter 1.0.0 py35_3

jupyter_client 5.0.0 py35_0

jupyter_console 5.1.0 py35_0

jupyter_core 4.3.0 py35_0

keras 2.0.8 py35h4bddecc_0

lazy-object-proxy 1.2.2 py35_0

libdynd 0.7.2 0

libedit 3.1 heed3624_0

libffi 3.2.1 1

libgcc 5.2.0 0

libgcc-ng 7.2.0 h7cc24e2_2

libgfortran 3.0.0 1

libgfortran-ng 7.2.0 h9f7466a_2

libiconv 1.14 0

libpng 1.6.27 0

libprotobuf 3.4.1 h5b8497f_0

libsodium 1.0.10 0

libstdcxx-ng 7.2.0 h7a57d05_2

libtiff 4.0.6 3

libxcb 1.12 1

libxml2 2.9.4 0

libxslt 1.1.29 0

llvmlite 0.21.0 py35hcfd7307_0

locket 0.2.0 py35_1

lxml 3.7.3 py35_0

markdown 2.6.9 py35_0

markupsafe 0.23 py35_2

matplotlib 2.0.2 np113py35_0

mistune 0.7.4 py35_0

mkl 2018.0.1 h19d6760_4

mkl-service 1.1.2 py35_3

mpmath 0.19 py35_1

msgpack-python 0.4.8 py35h783f4c8_0

multipledispatch 0.4.9 py35_0

nb_anacondacloud 1.2.0 py35_0

nb_conda 2.0.0 py35_0

nb_conda_kernels 2.0.0 py35_0

nbconvert 5.1.1 py35_0

nbformat 4.3.0 py35_0

nbpresent 3.0.2 py35_0

ncurses 6.0 h9df7e31_2

networkx 1.11 py35_0

nltk 3.2.2 py35_0

nose 1.3.7 py35_1

notebook 4.4.1 py35_0

numba 0.36.1 np113py35hf1afa12_0

numexpr 2.6.4 py35h119f745_0

numpy 1.13.3 py35hd829ed6_0

numpydoc 0.6.0 py35_0

odo 0.5.0 py35_1

olefile 0.44 py35_0

openjdk 8.17.0.3 1

openpyxl 2.4.1 py35_0

openssl 1.0.2n hb7f436b_0

pandas 0.21.1 py35h20b78c2_0

pandas-datareader 0.5.0 <pip>

pandoc 1.15.0.6 0

pandocfilters 1.4.1 py35_0

pango 1.40.3 1

parso 0.1.1 py35h1b200a3_0

partd 0.3.8 py35h68187f2_0

patchelf 0.9 0

path.py 10.1 py35_0

pathlib2 2.2.0 py35_0

patsy 0.4.1 py35_0

pcre 8.39 1

pep8 1.7.0 py35_0

pexpect 4.2.1 py35_0

pickleshare 0.7.4 py35_0

pillow 4.0.0 py35_1

pip 9.0.1 py35_1

pixman 0.34.0 0

pkginfo 1.4.1 py35_0

ply 3.10 py35_0

prompt_toolkit 1.0.13 py35_0

protobuf 3.4.1 py35he6b9134_0

psutil 5.2.1 py35_0

ptyprocess 0.5.1 py35_0

py 1.4.31 py35_0

pyasn1 0.2.3 py35_0

pycosat 0.6.2 py35_0

pycparser 2.17 py35_0

pycrypto 2.6.1 py35_4

pycurl 7.43.0 py35_2

pyflakes 1.5.0 py35_0

pyglet 1.3.0 <pip>

pygments 2.2.0 py35_0

pylint 1.6.4 py35_1

pyopenssl 16.2.0 py35_0

pyparsing 2.1.4 py35_0

pyqt 5.6.0 py35_2

pytables 3.4.2 np113py35_0

pytest 3.0.7 py35_0

python 3.5.4 h417fded_24

python-dateutil 2.6.0 py35_0

pytz 2017.2 py35_0

pywavelets 0.5.2 py35h53ec731_0

pyyaml 3.12 py35_0

pyzmq 16.0.2 py35_0

qt 5.6.2 3

qtawesome 0.4.4 py35_0

qtconsole 4.3.0 py35_0

qtpy 1.2.1 py35_0

readline 7.0 ha6073c6_4

redis 3.2.0 0

redis-py 2.10.5 py35_0

requests 2.13.0 py35_0

requests-file 1.4.2 <pip>

requests-ftp 0.3.1 <pip>

rope 0.9.4 py35_1

ruamel_yaml 0.11.14 py35_1

scikit-image 0.13.1 py35h7a281a6_0

scikit-learn 0.19.1 py35hbf1f462_0

scipy 1.0.0 py35hcbbe4a2_0

seaborn 0.8.1 <pip>

setuptools 27.2.0 py35_0

simplegeneric 0.8.1 py35_1

singledispatch 3.4.0.3 py35_0

sip 4.18 py35_0

six 1.10.0 py35_0

snowballstemmer 1.2.1 py35_0

sockjs-tornado 1.0.3 py35_0

sortedcontainers 1.5.7 py35h683703c_0

sphinx 1.5.1 py35_0

spyder 3.1.3 py35_0

sqlalchemy 1.1.8 py35_0

sqlite 3.20.1 hb898158_2

statsmodels 0.8.0 py35haa9d50b_0

sympy 1.0 py35_0

tblib 1.3.2 py35hf1eb0b4_0

tensorflow 1.3.0 0

tensorflow-base 1.3.0 py35h79a3156_1

tensorflow-tensorboard 0.1.5 py35_0

terminado 0.6 py35_0

testpath 0.3 py35_0

tk 8.6.7 hc745277_3

toolz 0.8.2 py35h90f1797_0

tornado 4.5.2 py35hf879e1d_0

traitlets 4.3.2 py35_0

unicodecsv 0.14.1 py35_0

wcwidth 0.1.7 py35_0

werkzeug 0.12.1 py35_0

wheel 0.29.0 py35_0

widgetsnbextension 2.0.0 py35_0

wrapt 1.10.8 py35_0

xgboost 0.6a2 py35_0 bioconda

xlrd 1.0.0 py35_0

xlsxwriter 0.9.6 py35_0

xlwt 1.2.0 py35_0

xz 5.2.3 h55aa19d_2

yaml 0.1.6 0

zeromq 4.1.5 0

zict 0.1.3 py35h29275ca_0

zlib 1.2.11 ha838bed_2

Here is my OpenBLAS, and LIBNVBLAS (from Nvidia) which are (were) both working great before today's problem, as you can see:

$ sudo update-alternatives --config libblas.so.3

There are 3 choices for the alternative libblas.so.3 (providing /usr/lib/libblas.so.3).

Selection Path Priority Status

------------------------------------------------------------

0 /usr/lib/openblas-base/libblas.so.3 40 auto mode

1 /usr/lib/libblas/libblas.so.3 10 manual mode

* 2 /usr/lib/openblas-base/libblas.so.3 40 manual mode

3 /usr/local/cuda/lib64/libnvblas.so 5 manual mode

Reply all

Reply to author

Forward

0 new messages