Error while running tests: error while loading shared libraries: libhdf5_hl.so.8: cannot open shared

[Nishan Wickrama]

Hi all,

I download the release candidate of Caffe and built it successfully.

"make all" and "make test" commands completed without a problem. But when I run "make runtest" I get the following error:

.build_release/test/test_all.testbin 0 --gtest_shuffle 

.build_release/test/test_all.testbin: error while loading shared libraries: libhdf5_hl.so.8: cannot open shared object file: No such file or directory

make: *** [runtest] Error 12

I get the same error when I try to run LeNet sample. My Makefile.config file is given below (only uncommented lines are shown). I'm using the anaconda Python distribution. I have already removed libm.so.* files from /home/caffe/anaconda/lib. My LD_LIBRARY_PATH contains ./home/caffe/anaconda/lib.

Any help would be greatly appreciated!

Thanks,

Nish

CUDA_DIR := /usr/local/cuda

CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \

-gencode arch=compute_20,code=sm_21 \

-gencode arch=compute_30,code=sm_30 \

-gencode arch=compute_35,code=sm_35 \

#-gencode arch=compute_50,code=sm_50 \

#-gencode arch=compute_50,code=compute_50

BLAS := mkl

MATLAB_DIR := /usr/local/MATLAB/R2014a

PYTHON_INCLUDE := $(HOME)/anaconda/include \

$(HOME)/anaconda/include/python2.7 \

$(HOME)/anaconda/lib/python2.7/site-packages/numpy/core/include

PYTHON_LIB := $(HOME)/anaconda/lib

INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include

LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

BUILD_DIR := build

DISTRIBUTE_DIR := distribute

TEST_GPUID := 0

[Friendly User]

after 

sudo apt-get install libhdf5-dev

does

 locate libhdf5_hl.so

give you any clues?

[Nishan Wickrama]

Hello,

It turns out that my LD_LIBRARY_PATH configuration inside .bashrc wasn't working due to some reason. 

I set it as described in the top answer in http://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux and it worked!

For the record I added a new file to sudo gedit /etc/ld.so.conf.d/caffeLibs.conf

and inside the file wrote

/home/caffe/anaconda/lib

/opt/intel/mkl/lib/intel64

Thanks,

Nish

--
You received this message because you are subscribed to a topic in the Google Groups "Caffe Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/caffe-users/tjRiYa8-VjI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/67171891-549f-48a9-be56-66084b620e49%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sujit Pal]

I know this is a solved issue, but I encountered this recently and I had a slightly different resolution path, so putting it here for posterity in case it helps someone.

When trying to do "make runtest" myself. I am also using Anaconda Python. My locate command (based on reply from Friendly User) initially returned an error complaining about a missing mlocate.db, so based on a tip on linuxquestions.org[1] I ran "sudo updatedb", after which the "locate libhdf5_hl.so.9" command returned files under my Anaconda lib directory. Adding the path $ANACONDA_HOME/lib to LD_LIBRARY_PATH in .bashrc fixed it, and "make runtest" ran without error.

-sujit

[1] http://www.linuxquestions.org/questions/linux-newbie-8/locate-can-not-open-%60-var-lib-mlocate-mlocate-db%27-no-such-file-or-directory-665848/

[Zizhao Zhang]

Hi,

I solve this problem with the same method as you told. But when I try to run matlab interface. I got an error caffe_.mexa64': libhdf5_hl.so.9: cannot open shared object file: No such file or directory. It seems there is a library mismatch between matlab and compile. Do you have any idea how to solve that?

[Sebastián Ramírez]

Using the Anaconda LD_LIBRARY_PATH can work for make runtest, but can create problems later, I just compiled with the current version and had problems with other libraries later.

To compile without using the Anaconda LD_LIBRARY_PATH and probably, to avoid the Matlab libhdf5_hl.so.9 error, you can do the following:

• Go to the libraries directory:

  cd /usr/lib/x86_64-linux-gnu
  

• Link the system version of HDF5 from 7 to 9:

  sudo ln -s libhdf5.so.7 libhdf5.so.9
  sudo ln -s libhdf5_hl.so.7 libhdf5_hl.so.9
  

• Update the "Dynamic Linker":

  sudo ldconfig 
  

I wrote a guide on installing almost everything in Ubuntu 14.04 and it seems working, it might be useful: https://github.com/tiangolo/caffe/blob/ubuntu-tutorial-b/docs/install_apt2.md

[Armand]

Thanks it works!

[flyv...@gmail.com]

Sorry to resurrect this thread but I am having a similar issue that is not being resolved by the method you have written, although I believe that the resolution will be similar. When running

make runtest -j $(($(nproc) + 1))

I get the following output:

Enter code here...user@cuda:~/caffe_BVLC/caffe-master$ make runtest -j $(($(nproc) + 1))
.build_release/tools/caffe
.build_release/tools/caffe: error while loading shared libraries: libhdf5_hl.so.10: cannot open shared object file: No such file or directory
make: *** [runtest] Error 127

I am using Ubuntu 14.04 and followed the instructions on your online guide. My libhdf5_hl.so.10 file is located at:

user@cuda:/usr/lib/x86_64-linux-gnu$ locate libhdf5_hl.so
/home/tom/anaconda3/lib/libhdf5_hl.so
/home/tom/anaconda3/lib/libhdf5_hl.so.10
/home/tom/anaconda3/lib/libhdf5_hl.so.10.0.2
/home/tom/anaconda3/pkgs/hdf5-1.8.16-0/lib/libhdf5_hl.so
/home/tom/anaconda3/pkgs/hdf5-1.8.16-0/lib/libhdf5_hl.so.10
/home/tom/anaconda3/pkgs/hdf5-1.8.16-0/lib/libhdf5_hl.so.10.0.2
/usr/lib/x86_64-linux-gnu/libhdf5_hl.so
/usr/lib/x86_64-linux-gnu/libhdf5_hl.so.7
/usr/lib/x86_64-linux-gnu/libhdf5_hl.so.7.0.0
/usr/lib/x86_64-linux-gnu/libhdf5_hl.so.9

Any thoughts? Help would be greatly appreciated.

[Sebastián Ramírez]

I haven't used a local installation of Caffe in a while, I have been using Docker and nvidia-docker to run GPU containers. And right now I have Ubuntu 16.04, so I can't debug it very easily right now.

Nevertheless, here's what I see: the guide assumes it's needed to create a link from version 7 to 9. By the error you posted, it seems right now the needed version is 10.

So I think you could try one of the solutions other people posted above, like adding a file in "/etc/ld.so.conf.d/caffeLibs.conf" with "/home/tom/anaconda3/lib": https://groups.google.com/d/msg/caffe-users/tjRiYa8-VjI/_fyjj78pBtYJ

You may try creating a link from your Anaconda in home to the global directory (although it seems weird creating a link from a user home to the global environment) with something like:

sudo ln -s /home/tom/anaconda3/lib/libhdf5.so.10 /usr/lib/x86_64-linux-gnu/libhdf5.so.10

sudo ln -s /home/tom/anaconda3/lib/libhdf5_hl.so.10 /usr/lib/x86_64-linux-gnu/libhdf5_hl.so.10

Or you may want to try it in Ubuntu 16.04 (the latest LTS version), because the version 10 of libhdf5 seems to be available in Ubuntu 16.04 but not in Ubuntu 14.04:

http://packages.ubuntu.com/search?keywords=libhdf5-10

I think that creating a link from version 9 to version 10 would probably break, I think there might be compatibility issues, but you might try that if you are desperate.

[flyv...@gmail.com]

Thank you very much for the suggestions, Sebastián. Unfortunately as you predicted they did not work. I believe that my only option at this point is to wipe out my Ubuntu 14.04 install and begin anew with a new platform. I obviously don't know everything about Caffe, but it does seem to me that based on this issue there there may be reason to amend the official Caffe install page for Ubuntu to state that 14.04 is not a good platform to use for installing Caffe. But that's a separate issue.

If I may ask one more question: what do you recommend as the current most stable platform to use as a base for running Caffe? In other words, I am looking for the platform that will give me the greatest chance of successfully compiling and using Caffe with the least hassle. I don't mind tweaking things and hunting down answers, but I have spent almost two weeks now trying to get this to run, on both a Fedora 21 system and an Ubuntu 14.04 system. Would you recommend me copying your setup of Ubuntu 16.04 + Docker? My only concern with that approach is that Ubuntu 16.04 doesn't seem to be officially supported by NVIDIA for the latest version of CUDA, but I assume there are work-arounds for that.

Thanks again.

[Sebastián Ramírez]

I'll be completely honest, for my investigation and projects I'm now using Keras (http://keras.io/) instead of Caffe. It's been a lot easier to me to generate dynamic networks created in code instead of needing configuration files (although Keras can save and load network topologies and parameters in configuration files too).

I spent quite some time making Caffe work for me but since I tried Keras I haven't used Caffe again actually. Caffe is great, but for my projects Keras fitted better.

These are the main important things about Keras for me:

* A trained network can be saved to files (as in Caffe): the topology and the trained parameters.

* Networks are created in very simple Python code (or loaded from files), so they can be created dynamically. That allows me to do hyperparameter optimization (for example with Hyperopt: http://hyperopt.github.io/hyperopt/) a lot easier, creating new networks with different parameters and sizes from code, not from files that I would have to somehow create in code.

* I don't have to compile, understand, write and read ProtocolBuffers or its files, or try to understand the files in the source code in Caffe, as the docs aren't always completely up to date.

* I can easily create RNN (Recurrent Neural Networks) (for example for time series or text).

* It can run on top of Theano or TensorFlow, taking the advantage of a Nvidia CUDA graphics card (as with Caffe). ...and you could brag around with the fad of TensorFlow telling that you are using it hehe ;) .

* Theano seems to perform quite well, so there's not much performance lost: https://github.com/soumith/convnet-benchmarks

* Another Python option not based in Theano that performs very well too is Nervana, but it requires a recent graphics card architecture (from Kepler and onwards) and I needed to be able to run in a Fermi (older) architecture (and I'm not sure about the other points with Nervana).

I had that question about nvidia-docker in Ubuntu 16.04 before upgrading, but I confirmed it works before doing the upgrade. Check the GitHub issue and conversation here: https://github.com/NVIDIA/nvidia-docker/issues/81

Here's how I see it (if I was you):

* You will be probably better starting with an Ubuntu 16.04. Any new software and new versions will be available for it, for the next years. I would put my bets on Ubuntu 16.04.

* Have you used Docker or are you interested in Docker? If "yes" to any of those, go with NVIDIA-Docker (and learn Docker if you haven't, it's completely worthwhile). I love Docker and I'm using it everywhere, it solves a whole lot of problems for me, if I could install Docker also in my Android phone I would, and it will be useful for you in a lot of other cases in the future. And there are Docker images with Caffe already.

* If you want to just deploy a Caffe model from the model zoo (for example, an already trained big network for image recognition), go with Caffe, you would just have to make it work and you are done.

* If you aren't very "expert" in Linux (taken from the private email, not from this thread), I think that installing Keras and making it work would be a lot easier than Caffe, with all the C++ needs, protocol buffers, libraries, paths, variables, etc. The installation instructions in the docs worked for me quite straightforward and I had an installation in one day compared to one or more weeks installing Caffe (which ended up in that long guide I made).

* If you want to build, customize and experiment with neural networks from scratch, and would prefer Python than C++, I would go with Keras.

* If you want to try RNNs, Keras.

* If you want hyperparameter optimization with Hyperopt, Keras.

* If you want to fiddle with the network, save intermediate results (for example to plot learning curves), etc. Keras. Caffe can train with Python and you theoretically could get those intermediate results with Python, but training with Python is a lot slower than training with the binaries (C++) and you end up losing the C++ "advantage" (at least that happened the last time I tried). And the C++ advantage is not that notorious compared to Theano for example, if you look at the benchmarks.

But again, if you *just* want to deploy a pre-trained model "as a service" to classify images or something like that, the models from the model zoo are great and they already made the hard work for you.

If you end up using Caffe and make it work for you in Ubuntu 16.04 making some changes to that guide I made, I would greatly appreciate a GitHub pull request with those changes you make (or even more comments here), so that others can take advantage of you new knowlegdge.

And to finish, a quick personal tip for development that you might find useful. I'm not very good at memorizing and knowing all the code, functions, modules, methods, parameters, etc... I use lots of "autocompletion". And I like to be able to run parts of my code to see what to do next and to check intermediate values, etc. I really like how IPython notebooks have autocompletion and running of parts of the code (letting me see intermediate values, etc). But IPython notebook isn't really a "code editor". So right now I'm in love with Atom (https://atom.io) with the plug-in Hydrogen (https://atom.io/packages/hydrogen). If the description of my needs fits you too, you may want to try that out. Specially while you are exploring and coding with new Python modules and APIs. And Hydrogen (in Atom) now supports running Python from inside a Docker container (I added that), and it has helped me a lot. That might help you explore the APIs of the modules you end up using.

I'm sorry for not being able to help more, but I hope some of those ideas help you find your solution.

Best regards.